Categories
c# Uncategorized

Making Users Re-Enter their Password: ASP.NET Core & IdentityServer4

It’s often good security behaviour to require users to re-enter their password when they want to change some secure property of their account, like generate personal access tokens, or change their Multi-factor Authentication (MFA) settings.

You may have seen the Github ‘sudo’ mode, which asks you to re-enter your password when you try to change something sensitive.

Sudo Mode Dialog
The Github sudo mode prompt.

Most of the time a user’s session is long-lived, so when they want to do something sensitive, it’s best to check they still are who they say.

I’ve been working on the implementation of IdentityServer4 at Enclave for the past week or so, and had this requirement to require password confirmation before users can modify their MFA settings.

I thought I’d write up how I did this for posterity, because it took a little figuring out.

The Layout

In our application, we have two components, both running on ASP.NET Core 3.1

  • The Accounts app that holds all the user data; this is where Identity Server runs; we use ASP.NET Core Identity to do the actual user management.
  • The Portal app that holds the UI. This is a straightforward MVC app right now, no JS or SPA to worry about.

To make changes to a user’s account settings, the Profile Controller in the Portal app makes API calls to the Accounts app.

The Portal calls APIs in the Accounts app

All the API calls to the Accounts app are already secured using the Access Token from when the user logged in; we have an ASP.NET Core Policy in place for our additional API (as per the IdentityServer docs) to protect it.

The Goal

The desired outcome here is that specific sensitive API endpoints within the Accounts app require the calling user to have undergone a second verification, where they must have re-entered their password recently in order to use the API.

What we want to do is:

  • Allow the Portal app to request a ‘step-up’ access token from the Accounts app.
  • Limit the step-up access token to a short lifetime (say 15 minutes), with no refresh tokens.
  • Call a sensitive API on the Accounts App, and have the Accounts App validate the step-up token.

Issuing the Step-Up Token

First up, we need to generate a suitable access token when asked. I’m going to add a new controller, StepUpApiController, in the Accounts app.

This controller is going to have a single endpoint, which requires a regular access token before you can call it.

We’re going to use the provided IdentityServerTools class, that we can inject into our controller, to do the actual token generation.

Without further ado, let’s look at the code for the controller:

[Route("api/stepup")]
[ApiController]
[Authorize(ApiScopePolicy.WriteUser)]
public class StepUpApiController : ControllerBase
{
    private static readonly TimeSpan ValidPeriod = TimeSpan.FromMinutes(15);

    private readonly UserManager<ApplicationUser> _userManager;
    private readonly IdentityServerTools _idTools;

    public StepUpApiController(UserManager<ApplicationUser> userManager,
                               IdentityServerTools idTools)
    {
        _userManager = userManager;
        _idTools = idTools;
    }

    [HttpPost]
    public async Task<StepUpApiResponse> StepUp(StepUpApiModel model)
    {
        var user = await _userManager.GetUserAsync(User);

        // Verify the provided password.
        if (await _userManager.CheckPasswordAsync(user, model.Password))
        {
            var clientId = User.FindFirstValue(JwtClaimTypes.ClientId);

            var claims = new Claim[]
            {
                new Claim(JwtClaimTypes.Subject, User.FindFirstValue(JwtClaimTypes.Subject)),
            };

            // Create a token that:
            //  - Is associated to the User's client.
            //  - Is only valid for our configured period (15 minutes)
            //  - Has a single scope, indicating that the token can only be used for stepping up.
            //  - Has the same subject as the user.
            var token = await _idTools.IssueClientJwtAsync(
                clientId,
                (int)ValidPeriod.TotalSeconds,
                new[] { "account-stepup" },
                additionalClaims: claims);

            return new StepUpApiResponse { Token = token, ValidUntil = DateTime.UtcNow.Add(ValidPeriod) };
        }

        Response.StatusCode = StatusCodes.Status401Unauthorized;

        return null;
    }
}

A couple of important points here:

  • In order to even access this API, the normal access token being passed in the requested must conform to our own WriteUser scope policy, which requires a particular scope be in the access token to get to this API.
  • This generated access token is really basic; it has a single scope, “account-stepup”, and only a single additional claim containing the subject.
  • We associate the step-up token to the same client ID as the normal access token, so only the requesting client can use that token.
  • We explicitly state a relatively short lifetime on the token (15 minutes here).

Sending the Token

This is the easy bit; once you have the token, you can store it somewhere in the client, and send it in a subsequent request.

Before sending the step-up token, you’ll want to check the expiry on it, and if you need a new one, then prompt the user for their credentials and start the process again.

For any request to the sensitive API, we need to include both the normal access token from the user’s session, plus the new step-up token.

I set this up when I create the HttpClient:

private async Task<HttpClient> GetClient(string? stepUpToken = null)
{
    var client = new HttpClient();

    // Set the base address to the URL of our Accounts app.
    client.BaseAddress = _accountUrl;

    // Get the regular user access token in the session and add that as the normal
    // Authorization Bearer token.
    // _contextAccessor is an instance of IHttpContextAccessor.
    var accessToken = await _contextAccessor.HttpContext.GetUserAccessTokenAsync();
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

    if (stepUpToken is object)
    {
        // We have a step-up token; include it as an additional header (without the Bearer indicator).
        client.DefaultRequestHeaders.Add("X-Authorization-StepUp", stepUpToken);
    }

    return client;
}

That X-Authorization-StepUp header is where we’re going to look when checking for the token in the Accounts app.

Validating the Step-Up Token

To validate a provided step-up token in the Accounts app, I’m going to define a custom ASP.NET Core Policy that requires the API call to provide a step-up token.

If there are terms in here that don’t seem immediately obvious, check out the docs on Policy-based authorization in ASP.NET Core. It’s a complex topic, but the docs do a pretty good job of breaking it down.

Let’s take a look at an API call endpoint that requires step-up:

[ApiController]
[Route("api/user")]
[Authorize(ApiScopePolicy.WriteUser)]
public class UserApiController : Controller
{
    [HttpPost("totp-enable")]
    [Authorize("require-stepup")]
    public async Task<IActionResult> EnableTotp(TotpApiEnableModel model)
    {
        // ... do stuff ...
    }
}

That Authorize attribute I placed on the action method specifies that we want to enforce a require-stepup policy on this action. Authorize attributes are additive, so a request to EnableTotp requires both our normal WriteUser policy and our step-up policy.

Defining our Policy

To define our require-stepup policy, lets jump over to our Startup class; specifically, in ConfigureServices, where we set up Authorization using the AddAuthorization method:

services.AddAuthorization(options =>
{
    // Other policies omitted...

    options.AddPolicy("require-stepup", policy =>
    { 
        policy.AddAuthenticationSchemes("local-api-scheme");
        policy.RequireAuthenticatedUser();
        
        // Add a new requirement to the policy (for step-up).
        policy.AddRequirements(new StepUpRequirement());
    });
});

The ‘local-api-scheme’ is the built-in scheme provided by IdentityServer for protecting local API calls.

That requirement class, StepUpRequirement is just a simple marker class for indicating to the policy that we need step-up. It’s also how we wire up a handler to check that requirement:

public class StepUpRequirement : IAuthorizationRequirement
{
}

Defining our Authorization Handler

We now need an Authorization Handler that lets us check incoming requests meet our new step-up requirement.

So, let’s create one:

public class StepUpAuthorisationHandler : AuthorizationHandler<StepUpRequirement>
{
    private const string StepUpTokenHeader = "X-Authorization-StepUp";

    private readonly IHttpContextAccessor _httpContextAccessor;
    private readonly ITokenValidator _tokenValidator;

    public StepUpAuthorisationHandler(
        IHttpContextAccessor httpContextAccessor,
        ITokenValidator tokenValidator)
    {
        _httpContextAccessor = httpContextAccessor;
        _tokenValidator = tokenValidator;
    }

    /// <summary>
    /// Called by the framework when we need to check a request.
    /// </summary>
    protected override async Task HandleRequirementAsync(
        AuthorizationHandlerContext context,
        StepUpRequirement requirement)
    {
        // Only interested in authenticated users.
        if (!context.User.IsAuthenticated())
        {
            return;
        }

        var httpContext = _httpContextAccessor.HttpContext;

        // Look for our special request header.
        if (httpContext.Request.Headers.TryGetValue(StepUpTokenHeader, out var stepUpHeader))
        {
            var headerValue = stepUpHeader.FirstOrDefault();

            if (!string.IsNullOrEmpty(headerValue))
            {
                // Call our method to check the token.
                var validated = await ValidateStepUp(context.User, headerValue);

                // Token was valid, so succeed.
                // We don't explicitly have to fail, because that is the default.
                if (validated)
                {
                    context.Succeed(requirement);
                }
            }
        }
    }

    private async Task<bool> ValidateStepUp(ClaimsPrincipal user, string header)
    {
        // Use the normal token validator to check the access token is valid, and contains our
        // special expected scope.
        var validated = await _tokenValidator.ValidateAccessTokenAsync(header, "account-stepup");

        if (validated.IsError)
        {
            // Bad token.
            return false;
        }

        // Validate that the step-up token is for the same client as the access token.
        var clientIdClaim = validated.Claims.FirstOrDefault(x => x.Type == JwtClaimTypes.ClientId);

        if (clientIdClaim is null || clientIdClaim.Value != user.FindFirstValue(JwtClaimTypes.ClientId))
        {
            return false;
        }

        // Confirm a subject is supplied.
        var subjectClaim = validated.Claims.FirstOrDefault(x => x.Type == JwtClaimTypes.Subject);

        if (subjectClaim is null)
        {
            return false;
        }

        // Confirm that the subject of the stepup and the current user are the same.
        return subjectClaim.Value == user.FindFirstValue(JwtClaimTypes.Subject);
    }
}

Again, let’s take a look at the important bits of the class:

  • The handler derives from AuthorizationHandler<StepUpRequirement>, indicating to ASP.NET that we are a handler for our custom requirement.
  • We stop early if there is no authenticated user; that’s because the step-up token is only valid for a user who is already logged in.
  • We inject and use IdentityServer’s ITokenValidator interface to let us validate the token using ValidateAccessTokenAsync; we specify the scope we require.
  • We check that the client ID of the step-up token is the same as the regular access token used to authenticate with the API.
  • We check that the subjects match (i.e. this step-up token is for the same user).

The final hurdle is to register our authorization handler in our Startup class:

services.AddSingleton<IAuthorizationHandler, StepUpAuthorisationHandler>();

Wrapping Up

There you go, we’ve now got a secondary access token being issued to indicate step-up has been done, and we’ve got a custom authorization handler to check our new token.

Categories
arduino c#

Displaying Real-time Sensor Data in the Browser with SignalR and ChartJS

In my previous posts on Modding My Rowing Machine, I wired up an Arduino to my rowing machine, and streamed the speed sensor data to an ASP.NET core application.

In this post, I’m going to show you how to take sensor and counter data, push it to a browser as it arrives, and display it in a real-time chart.

If you want to skip ahead, I’ve uploaded all the code for the Arduino and ASP.NET components to a github repo at https://github.com/alistairjevans/rower-mod.

I’m using Visual Studio 2019 with the ASP.NET Core 3.0 Preview for all the server-side components, but the latest stable release of ASP.NET Core will work just fine, I’m not using any of the new features.

Pushing Data to the Browser

So, you will probably have heard of SignalR, the ASP.NET technology that can be used to push data to the browser from the server, and generally establish a closer relationship between the two.

I’m going to use it send data to the browser whenever new sensor data arrives, and also to let the browser request that the count be reset.

The overall component layout looks like this:

Setting up SignalR

This bit is pretty easy; first up, head over to the Startup.cs file in your ASP.NET app project, and in the ConfigureServices method, add SignalR:

public void ConfigureServices(IServiceCollection services)
{
// Define a writer that saves my data to disk
var folderPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData),
"rower");
services.AddSingleton<ISampleWriter>(svc => new SampleWriter(folderPath, "samples"));
// Keep my machine state as a singleton
services.AddSingleton<IMachineState, MachineState>();
services.AddControllersWithViews()
.AddNewtonsoftJson();
services.AddRazorPages();
// Add signalr services
services.AddSignalR();
}
view raw Startup.cs hosted with ❤ by GitHub

Next, create a SignalR Hub. This is effectively the endpoint your clients will connect to, and will contain any methods a client needs to invoke on the server.

public class FeedHub : Hub
{
private readonly IMachineState machineState;
private readonly ISampleWriter sampleWriter;
public FeedHub(IMachineState machineState, ISampleWriter sampleWriter)
{
this.machineState = machineState;
this.sampleWriter = sampleWriter;
}
public void ResetCount()
{
// Reset the state, and start a new data file
machineState.ZeroCount();
sampleWriter.StartNewFile();
}
}
view raw FeedHub.cs hosted with ❤ by GitHub

SignalR Hubs are just classes that derive from the Hub class. I’ve got just the one method in mine at the moment, for resetting my counter.

Before that Hub will work, you need to register it in your Startup class’ Configure method:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
//
// Omitted standard content for brevity...
//
app.UseSignalR(cfg => cfg.MapHub<FeedHub>("/feed"));
}
view raw startup.cs hosted with ❤ by GitHub

You’re also going to want to add the necessary SignalR javascript to your project. I did it using the “Manage Client-Side Libraries” feature in Visual Studio; you can find my entire libman.json file (which defines which libraries I’m using) on my github repo

Sending Data to the Client

In the MVC Controller where the data arrives from the Arduino, I’m going to push the sensor data to all clients connected to the hub.

The way you access the clients of a hub from outside the hub (i.e. an MVC Controller) is by resolving an IHubContext<THubType>, and then accessing the Clients property.

public class DataController : Controller
{
private readonly IMachineState machineState;
private readonly ISampleWriter sampleWriter;
private readonly IHubContext<FeedHub> feedHub;
public DataController(IMachineState machineState, ISampleWriter sampleWriter, IHubContext<FeedHub> feedHub)
{
this.machineState = machineState;
this.sampleWriter = sampleWriter;
this.feedHub = feedHub;
}
[HttpPost]
public async Task<ActionResult> ProvideReading(uint milliseconds, double speed, int count)
{
// Update our machine state.
machineState.UpdateMachineState(milliseconds, speed, count);
// Write the sample to file (our sample writer) and update all clients
// Wait for them both to finish.
await Task.WhenAll(
sampleWriter.ProvideSample(machineState.LastSample, machineState.Speed, machineState.Count),
feedHub.Clients.All.SendAsync("newData",
machineState.LastSample.ToString("o"),
machineState.Speed,
machineState.Count)
);
return StatusCode(200);
}
}
view raw DataController.cs hosted with ❤ by GitHub

Pro tip:
Got multiple IO operations to do in a single request, that don’t depend on each other? Don’t just await one, then await the other; use Task.WhenAll, and the operations will run in parallel.

In my example above I’m writing to a file and to SignalR clients at the same time, and only continuing when both are done.

Browser

Ok, so we’ve got the set-up to push data to the browser, but no HTML just yet. I don’t actually need any MVC Controller functionality, so I’m just going to create a Razor Page, which still gives me a Razor template, but without having to write the controller behind it.

If I put an ‘Index.cshtml’ file under a new ‘Pages’ folder in my project, and put the following content in it, that becomes the landing page of my app:

@page
<html>
<head>
</head>
<body>
<div class="container">
<div class="lblSpeed text lbl">Speed:</div>
<div class="valSpeed text" id="currentSpeed"><!-- speed goes here --></div>
<div class="lblCount text lbl">Count:</div>
<div class="valCount text" id="currentCount"><!-- stroke count goes here --></div>
<div class="btnContainer">
<button id="reset">Reset Count</button>
</div>
<div class="chartContainer">
<!-- I'm going to render my chart in this canvas -->
<canvas id="chartCanvas"></canvas>
</div>
</div>
<script src="~/lib/signalr/dist/browser/signalr.js"></script>
<script src="~/js/site.js"></script>
</body>
</html>
view raw Index.cshtml hosted with ❤ by GitHub

In my site.js file, I’m just going to open a connection to the SignalR hub and attach a callback for data being given to me:

"use strict";
// Define my connection (note the /feed address to specify the hub)
var connection = new signalR.HubConnectionBuilder().withUrl("/feed").build();
// Get the elements I need
var speedValue = document.getElementById("currentSpeed");
var countValue = document.getElementById("currentCount");
var resetButton = document.getElementById("reset");
window.onload = function () {
// Start the SignalR connection
connection.start().then(function () {
console.log("Connected");
}).catch(function (err) {
return console.error(err.toString());
});
resetButton.addEventListener("click", function () {
// When someone clicks the reset button, this
// will call the ResetCount method in my FeedHub.
connection.invoke("ResetCount");
});
};
// This callback is going to fire every time I get new data.
connection.on("newData", function (time, speed, count) {
speedValue.innerText = speed;
countValue.innerText = count;
});
view raw site.js hosted with ❤ by GitHub

That’s actually all we need to get data flowing down to the browser, and displaying the current speed and counter values!

I want something a little more visual though….

Displaying the Chart

I’m going to use the ChartJS library to render a chart, plus a handy plugin for ChartJS that helps with streaming live data and rendering it, the chartjs-plugin-streaming plugin.

First off, add the two libraries to your project (and your HTML file), plus MomentJS, which ChartJS requires to function.

Next, let’s set up our chart, by defining it’s configuration and attaching it to the 2d context of the canvas object:

window.onload = function () {
var ctx = document.getElementById('chartCanvas').getContext('2d');
window.myChart = new Chart(ctx, {
type: 'line',
data: {
datasets: [{
label: 'Speed',
data: []
}]
},
options: {
scales: {
xAxes: [{
type: 'realtime',
delay: 0,
// 20 seconds of data
duration: 20000
}],
yAxes: [{
ticks: {
suggestedMin: 0,
suggestedMax: 50
}
}]
}
}
});
// The other signalr setup is still here...
}
view raw site.js hosted with ❤ by GitHub

Finally, let’s make our chart display new sensor data as it arrives:

connection.on("newData", function (time, speed, count) {
// This subtract causes the data to be placed
// in the centre of the chart as it arrives,
// which I personally think looks better...
var dateValue = moment(time).subtract(5, 'seconds');
speedValue.innerText = speed;
countValue.innerText = count;
// append the new data to the existing chart data
myChart.data.datasets[0].data.push({
x: dateValue,
y: speed
});
// update chart datasets keeping the current animation
myChart.update({
preservation: true
});
});
view raw site.js hosted with ❤ by GitHub

With all that together, let’s see what we get!

Awesome, a real-time graph of my rowing!

As an aside, I used the excellent tool by @sarah_edo to generate a CSS grid really quickly, so thanks for that! You can find it at https://cssgrid-generator.netlify.com/

You can check out the entire solution, including all the code for the Arduino and the ASP.NET app, on the github repo at https://github.com/alistairjevans/rower-mod.

Next up for the rowing machine project, I want to put some form of gamification, achievements or progress tracking into the app, but not sure exactly how it will look yet.

Categories
Uncategorized

Implementing a shared resource lock in your load-balanced application with MongoDB in C#

In your load-balanced cluster of nice, performant application servers, you may occasionally find that there is an outside resource that requires access be synchronised between each server in the cluster (ideally not too often, because it’s a pain, but there you go).

We encountered this recently when dealing with multiple clients trying to access and update a reporting component we interface with via our application. Each client issued HTTP API requests to the application that would:

  1. Read a resource from the reporting system (by the ID)
  2. Do something
  3. Update the resource

The problem is that no two tasks should be allowed to get past step 1 at the same time for a given named resource. This is starting to look pretty familiar, right?

We see this in a single-server multi-threaded model sometimes (although not too often hopefully; I find synchronous locks are generally bad for performance if used too liberally in web requests).

lock(someSharedResource)
{
// Do some work using the resource
// and maybe update it.
}

The problem with the above code is that it only locks a resource in the current process; a different HTTP request, routed to a different server by the load balancer, would happily acquire it’s own lock.

Establishing a distributed lock

What we need now is to lock the resource across our entire cluster, not just on the one server.

We use MongoDB for general shared state between servers in our cluster, so I looked into how to use MongoDB to also synchronise access to our resource between the application servers.

Luckily, it turns out that by using existing MongoDB functionality, this is pretty straightforward to create a short-lived resource lock.

Note
I’ve written this solution in C#, using the official MongoDB C# client, but there’s no reason this wouldn’t apply to a different MongoDB client implementation in any language.

Want to jump to the end?
https://github.com/alistairjevans/mongodb-locks

Create the Collection

First up, I want to create a new MongoDB collection to hold my locks. I’ll create a model and then get an instance of that collection.

public class LockModel
{
// We only need an ID right now
public string Id { get; set; }
}
view raw LockModel.cs hosted with ❤ by GitHub
public class LockProvider
{
private readonly IMongoCollection<LockModel> collection;
public LockProvider(string mongodbConnString)
{
// Create a lock collection
var client = new MongoClient(mongodbConnString);
var database = client.GetDatabase("mydb");
// Get our collection
collection = database.GetCollection<LockModel>("resourceLocks");
}
}
view raw LockProvider.cs hosted with ❤ by GitHub

That’s pretty basic stuff, you’d have to do that generally to access any MongoDB collection.

Next we’re going to add the function used to acquire a lock, AcquireLock. This method is responsible for the ‘spin’ or retry on the lock, waiting to acquire it.

public class LockProvider
{
private readonly IMongoCollection<LockModel> collection;
public LockProvider(string mongodbConnString) {} // Collapsed
public async Task<IDisposable> AcquireLock(string resourceId)
{
// Determine the id of the lock
var lockId = $"lock_{resourceId}";
var distributedLock = new DistributedLock(collection, lockId);
var startLockAcquireTime = DateTime.Now;
// Try and acquire the lock
while (!await distributedLock.AttemptGetLock())
{
// If we failed to acquire the lock, wait a moment.
await Task.Delay(100);
// Only try to acquire the lock for 10 seconds
if ((DateTime.Now - startLockAcquireTime).TotalSeconds > 10)
{
throw new ApplicationException($"Could not acquire lock for {resourceId} within the timeout.");
}
}
// This will only return if we have the lock.
return distributedLock;
}
}
view raw LockProvider.cs hosted with ❤ by GitHub

The AcquireLock method:

  1. Creates a ‘lock id’ from the resource id.
  2. Creates a ‘DistributedLock’ object, which is where the locking mechanism happens (more on this in a moment).
  3. Attempts to get the lock in a while loop.
  4. Waits up to a 10 second timeout to acquire the lock, attempting again every 100ms.
  5. Returns the DistributedLock once the lock is acquired (but only as an IDisposable).

Next let’s look at what is going on inside the DistributedLock class.

DistributedLock

The DistributedLock class is responsible for the actual MongoDB commands, and attempting the lock.

public class DistributedLock : IDisposable
{
private readonly IMongoCollection<LockModel> collection;
private readonly string lockId;
public DistributedLock(IMongoCollection<LockModel> collection, string lockId)
{
this.collection = collection;
this.lockId = lockId;
}
public async Task<bool> AttemptGetLock()
{
var response = await collection.FindOneAndUpdateAsync<LockModel>(
// Find a record with the lock ID
x => x.Id == lockId,
// If our 'upsert' creates a document, set the ID to the lock ID
Builders<LockModel>.Update.SetOnInsert(x => x.Id, lockId),
new FindOneAndUpdateOptions<LockModel>
{
// If the record doesn't exist, create it.
IsUpsert = true,
// Specifies that the result we want is the record BEFORE it
// was created (this is important).
ReturnDocument = ReturnDocument.Before
});
// If the result of the FindOneAndUpdateAsync is null, then that means there was no record
// before we ran our statement, so we now have the lock.
// If the result is not null, then it means a document for the lock already existed, so someone else has the lock.
if (response == null)
{
return true;
}
return false;
}
public void Dispose()
{
// Delete the document with the specified lock ID, effectively releasing the lock.
collection.DeleteOne(x => x.Id == lockId);
}
}
view raw DistributedLock.cs hosted with ❤ by GitHub

Let’s break down what happens here. The AttemptGetLock method issues a FindOneAndUpdate MongoDB command that does the following:

  1. Looks for a record with an ID the same as the provided lock ID.
  2. If it finds it, it returns it without doing anything (because our update is only a SetOnInsert, not a Set).
  3. If it doesn’t find it, it creates a new document (because IsUpsert is true), with the expected ID.

We’ve set the ReturnDocument option to ‘Before’, because that means the result of the FindOneAndUpdateAsync is null if there was no existing lock document. If there was no existing document, there will be one now, and we have acquired the lock!

When you dispose of the lock, we just delete the lock document from the collection, and hey presto, the lock has been released, and the next thread to try to get the lock will do so.

Using It

Because the AcquireLock method returns an IDisposable (via a Task), you can just use a ‘using’ statement in a similar manner to how you would use the ‘lock’ statement.

using (await LockProvider.AcquireLock(id))
{
// Do some work
}

Do you see that await inside the using definition? Right, do not forget to use it. If you do forget, all hell will break loose, because Task<IDisposable> also implements IDisposable! So you’ll end up instantly entering the using block, and then disposing of the task afterwards at some point while the lock attempts are happening. This causes many bad things.

Make sense? Good. Unfortunately, we’re not quite done…

Handing Concurrent Lock Attempts

So, while the above version of DistributedLock works pretty well most of the time, at some point it will inevitably throw an exception when handling multiple concurrent lock attempts:

Duplicate keys…

Why does this happen? Well, the upsert functionality in findAndModify is not technically atomic in the way you might expect; the find and the insert are different operations internally to MongoDB, so two lock attempts might both attempt to insert a record.

When that happens, the default ID index on the MongoDB collection will throw an E11000 duplicate key error.

This is actually OK; one of the threads that attempted to acquire a lock will get it (the first one to complete the insert), and the second one will get an exception, so we just need to amend our code to say that the thread with the exception failed to get the lock.

try
{
var response = await collection.FindOneAndUpdateAsync<LockModel>(
// Collapsed
);
// If the result of the FindOneAndUpdateAsync is null, then that means there was no record
// before we ran our statement, so we now have the lock.
// If the result is not null, then it means a document for the lock already existed, so someone else has the lock.
if (response == null)
{
return true;
}
return false;
}
catch (MongoCommandException ex)
{
// 11000 == MongoDB Duplicate Key error
if (ex.Code == 11000)
{
// Two threads have tried to acquire a lock at the exact same moment on the same key,
// which will cause a duplicate key exception in MongoDB.
// So this thread failed to acquire the lock.
return false;
}
throw;
}
view raw DistributedLock.cs hosted with ❤ by GitHub

Handling a Crash

The last problem we have to solve is what happens if one of the application servers crashes part-way through a piece of work?

If a thread has a lock, but the server crashes or is otherwise disconnected from MongoDB, it can’t release the resource, meaning no-one else will ever be able to take a lock on the resource.

We need to put in some safeguard against this that allows the lock to eventually be released even if the application isn’t able to do it correctly.

To do this, we can use one of my favourite MongoDB features, TTL Indexes, which allows MongoDB to ‘age out’ records automatically, based on a column that contains an ‘expiry’ time.

Let’s update the original LockModel with an expiry property, and add a TTL index to our collection.

public class LockModel
{
public string Id { get; set; }
/// <summary>
/// I'm going to set this to the moment in time when the lock should be cleared.
/// </summary>
public DateTime ExpireAt { get; set; }
}
view raw LockModel.cs hosted with ❤ by GitHub
// In our LockProvider constructor...
// Specify a TTL index on the ExpiryPoint field.
collection.Indexes.CreateOne(new CreateIndexModel<LockModel>(
Builders<LockModel>.IndexKeys.Ascending(l => l.ExpireAt),
new CreateIndexOptions
{
ExpireAfter = TimeSpan.Zero
}
));
view raw LockProvider.cs hosted with ❤ by GitHub

In the above index creation instruction, I’m specifying an ExpireAfter of TimeSpan.Zero, meaning that as soon as the DateTime specified in ExpireAt of the lock document passes, the document will be deleted.

Finally, we’ll update the MongoDB FindOneAndUpdate instruction in DistributedLock to set the ExpireAt property to the current time plus 1 minute.

var response = await collection.FindOneAndUpdateAsync<LockModel>(
// Find a record with the lock ID
x => x.Id == lockId,
// If our 'upsert' creates a document, set the ID to the lock ID
Builders<LockModel>.Update
.SetOnInsert(x => x.Id, lockId)
.SetOnInsert(x => x.ExpireAt, DateTime.UtcNow.AddMinutes(1)),
new FindOneAndUpdateOptions<LockModel>
{
// If the record doesn't exist, create it.
IsUpsert = true,
// Specifies that the result we want is the record BEFORE it
// was created (this is important).
ReturnDocument = ReturnDocument.Before
});
view raw DistributedLock.cs hosted with ❤ by GitHub

Now, if a lock isn’t released after 1 minute, it will automatically be cleaned up by MongoDB, and another thread will be able to acquire the lock.

Notes on TTL Indexes and Timeouts

  • The TTL index is not a fast way of releasing these locks; the accuracy on it is low, because by default the MongoDB thread that checks for expired documents only runs once every 60 seconds.
    We’re using it as a safety net for an edge case, rather than a predictable mechanism. If you need a faster recovery than that 60 second window, then you may need to look for an alternative solution.
  • You may notice that I’m using DateTime.UtcNow for the ExpireAt value, rather than DateTime.Now; this is because I have had a variety of problems storing C# DateTimes with a timezone in MongoDB in a reliable way, so I tend to prefer storing UTC values whenever possible (especially when it’s not a user-entered value).

Sample Project

I’ve created a github repo with an ASP.NET Core project at https://github.com/alistairjevans/mongodb-locks that has a complete implementation of the above, with an example API controller that outputs timing information for the lock acquisition.