Displaying Real-time Sensor Data in the Browser with SignalR and ChartJS

In my previous posts on Modding My Rowing Machine, I wired up an Arduino to my rowing machine, and streamed the speed sensor data to an ASP.NET core application.

In this post, I’m going to show you how to take sensor and counter data, push it to a browser as it arrives, and display it in a real-time chart.

If you want to skip ahead, I’ve uploaded all the code for the Arduino and ASP.NET components to a github repo at https://github.com/alistairjevans/rower-mod.

I’m using Visual Studio 2019 with the ASP.NET Core 3.0 Preview for all the server-side components, but the latest stable release of ASP.NET Core will work just fine, I’m not using any of the new features.

Pushing Data to the Browser

So, you will probably have heard of SignalR, the ASP.NET technology that can be used to push data to the browser from the server, and generally establish a closer relationship between the two.

I’m going to use it send data to the browser whenever new sensor data arrives, and also to let the browser request that the count be reset.

The overall component layout looks like this:

Setting up SignalR

This bit is pretty easy; first up, head over to the Startup.cs file in your ASP.NET app project, and in the ConfigureServices method, add SignalR:

Next, create a SignalR Hub. This is effectively the endpoint your clients will connect to, and will contain any methods a client needs to invoke on the server.

SignalR Hubs are just classes that derive from the Hub class. I’ve got just the one method in mine at the moment, for resetting my counter.

Before that Hub will work, you need to register it in your Startup class’ Configure method:

You’re also going to want to add the necessary SignalR javascript to your project. I did it using the “Manage Client-Side Libraries” feature in Visual Studio; you can find my entire libman.json file (which defines which libraries I’m using) on my github repo

Sending Data to the Client

In the MVC Controller where the data arrives from the Arduino, I’m going to push the sensor data to all clients connected to the hub.

The way you access the clients of a hub from outside the hub (i.e. an MVC Controller) is by resolving an IHubContext<THubType>, and then accessing the Clients property.

Pro tip:
Got multiple IO operations to do in a single request, that don’t depend on each other? Don’t just await one, then await the other; use Task.WhenAll, and the operations will run in parallel.

In my example above I’m writing to a file and to SignalR clients at the same time, and only continuing when both are done.

Browser

Ok, so we’ve got the set-up to push data to the browser, but no HTML just yet. I don’t actually need any MVC Controller functionality, so I’m just going to create a Razor Page, which still gives me a Razor template, but without having to write the controller behind it.

If I put an ‘Index.cshtml’ file under a new ‘Pages’ folder in my project, and put the following content in it, that becomes the landing page of my app:

In my site.js file, I’m just going to open a connection to the SignalR hub and attach a callback for data being given to me:

That’s actually all we need to get data flowing down to the browser, and displaying the current speed and counter values!

I want something a little more visual though….

Displaying the Chart

I’m going to use the ChartJS library to render a chart, plus a handy plugin for ChartJS that helps with streaming live data and rendering it, the chartjs-plugin-streaming plugin.

First off, add the two libraries to your project (and your HTML file), plus MomentJS, which ChartJS requires to function.

Next, let’s set up our chart, by defining it’s configuration and attaching it to the 2d context of the canvas object:

Finally, let’s make our chart display new sensor data as it arrives:

With all that together, let’s see what we get!

Awesome, a real-time graph of my rowing!

As an aside, I used the excellent tool by @sarah_edo to generate a CSS grid really quickly, so thanks for that! You can find it at https://cssgrid-generator.netlify.com/

You can check out the entire solution, including all the code for the Arduino and the ASP.NET app, on the github repo at https://github.com/alistairjevans/rower-mod.

Next up for the rowing machine project, I want to put some form of gamification, achievements or progress tracking into the app, but not sure exactly how it will look yet.

Implementing a shared resource lock in your load-balanced application with MongoDB in C#

In your load-balanced cluster of nice, performant application servers, you may occasionally find that there is an outside resource that requires access be synchronised between each server in the cluster (ideally not too often, because it’s a pain, but there you go).

We encountered this recently when dealing with multiple clients trying to access and update a reporting component we interface with via our application. Each client issued HTTP API requests to the application that would:

  1. Read a resource from the reporting system (by the ID)
  2. Do something
  3. Update the resource

The problem is that no two tasks should be allowed to get past step 1 at the same time for a given named resource. This is starting to look pretty familiar, right?

We see this in a single-server multi-threaded model sometimes (although not too often hopefully; I find synchronous locks are generally bad for performance if used too liberally in web requests).

The problem with the above code is that it only locks a resource in the current process; a different HTTP request, routed to a different server by the load balancer, would happily acquire it’s own lock.

Establishing a distributed lock

What we need now is to lock the resource across our entire cluster, not just on the one server.

We use MongoDB for general shared state between servers in our cluster, so I looked into how to use MongoDB to also synchronise access to our resource between the application servers.

Luckily, it turns out that by using existing MongoDB functionality, this is pretty straightforward to create a short-lived resource lock.

Note
I’ve written this solution in C#, using the official MongoDB C# client, but there’s no reason this wouldn’t apply to a different MongoDB client implementation in any language.

Want to jump to the end?
https://github.com/alistairjevans/mongodb-locks

Create the Collection

First up, I want to create a new MongoDB collection to hold my locks. I’ll create a model and then get an instance of that collection.

That’s pretty basic stuff, you’d have to do that generally to access any MongoDB collection.

Next we’re going to add the function used to acquire a lock, AcquireLock. This method is responsible for the ‘spin’ or retry on the lock, waiting to acquire it.

The AcquireLock method:

  1. Creates a ‘lock id’ from the resource id.
  2. Creates a ‘DistributedLock’ object, which is where the locking mechanism happens (more on this in a moment).
  3. Attempts to get the lock in a while loop.
  4. Waits up to a 10 second timeout to acquire the lock, attempting again every 100ms.
  5. Returns the DistributedLock once the lock is acquired (but only as an IDisposable).

Next let’s look at what is going on inside the DistributedLock class.

DistributedLock

The DistributedLock class is responsible for the actual MongoDB commands, and attempting the lock.

Let’s break down what happens here. The AttemptGetLock method issues a FindOneAndUpdate MongoDB command that does the following:

  1. Looks for a record with an ID the same as the provided lock ID.
  2. If it finds it, it returns it without doing anything (because our update is only a SetOnInsert, not a Set).
  3. If it doesn’t find it, it creates a new document (because IsUpsert is true), with the expected ID.

We’ve set the ReturnDocument option to ‘Before’, because that means the result of the FindOneAndUpdateAsync is null if there was no existing lock document. If there was no existing document, there will be one now, and we have acquired the lock!

When you dispose of the lock, we just delete the lock document from the collection, and hey presto, the lock has been released, and the next thread to try to get the lock will do so.

Using It

Because the AcquireLock method returns an IDisposable (via a Task), you can just use a ‘using’ statement in a similar manner to how you would use the ‘lock’ statement.

Do you see that await inside the using definition? Right, do not forget to use it. If you do forget, all hell will break loose, because Task<IDisposable> also implements IDisposable! So you’ll end up instantly entering the using block, and then disposing of the task afterwards at some point while the lock attempts are happening. This causes many bad things.

Make sense? Good. Unfortunately, we’re not quite done…

Handing Concurrent Lock Attempts

So, while the above version of DistributedLock works pretty well most of the time, at some point it will inevitably throw an exception when handling multiple concurrent lock attempts:

Duplicate keys…

Why does this happen? Well, the upsert functionality in findAndModify is not technically atomic in the way you might expect; the find and the insert are different operations internally to MongoDB, so two lock attempts might both attempt to insert a record.

When that happens, the default ID index on the MongoDB collection will throw an E11000 duplicate key error.

This is actually OK; one of the threads that attempted to acquire a lock will get it (the first one to complete the insert), and the second one will get an exception, so we just need to amend our code to say that the thread with the exception failed to get the lock.

Handling a Crash

The last problem we have to solve is what happens if one of the application servers crashes part-way through a piece of work?

If a thread has a lock, but the server crashes or is otherwise disconnected from MongoDB, it can’t release the resource, meaning no-one else will ever be able to take a lock on the resource.

We need to put in some safeguard against this that allows the lock to eventually be released even if the application isn’t able to do it correctly.

To do this, we can use one of my favourite MongoDB features, TTL Indexes, which allows MongoDB to ‘age out’ records automatically, based on a column that contains an ‘expiry’ time.

Let’s update the original LockModel with an expiry property, and add a TTL index to our collection.

In the above index creation instruction, I’m specifying an ExpireAfter of TimeSpan.Zero, meaning that as soon as the DateTime specified in ExpireAt of the lock document passes, the document will be deleted.

Finally, we’ll update the MongoDB FindOneAndUpdate instruction in DistributedLock to set the ExpireAt property to the current time plus 1 minute.

Now, if a lock isn’t released after 1 minute, it will automatically be cleaned up by MongoDB, and another thread will be able to acquire the lock.

Notes on TTL Indexes and Timeouts

  • The TTL index is not a fast way of releasing these locks; the accuracy on it is low, because by default the MongoDB thread that checks for expired documents only runs once every 60 seconds.
    We’re using it as a safety net for an edge case, rather than a predictable mechanism. If you need a faster recovery than that 60 second window, then you may need to look for an alternative solution.
  • You may notice that I’m using DateTime.UtcNow for the ExpireAt value, rather than DateTime.Now; this is because I have had a variety of problems storing C# DateTimes with a timezone in MongoDB in a reliable way, so I tend to prefer storing UTC values whenever possible (especially when it’s not a user-entered value).

Sample Project

I’ve created a github repo with an ASP.NET Core project at https://github.com/alistairjevans/mongodb-locks that has a complete implementation of the above, with an example API controller that outputs timing information for the lock acquisition.