Categories
arduino c# networks

Streaming real-time sensor data to an ASP.NET Core app from an Arduino

In my previous posts on Modding my Rowing Machine, I got started with an Arduino, and started collecting speed sensor data. The goal of this post is to connect to the WiFi network and upload sensor data to a server application I’ve got running on my laptop in as close to real-time as I can make it.

Connecting to WiFi

My Arduino Uno WiFi Rev 2 board has got a built-in WiFi module; it was considerably easier than I expected to get everything connected.

I first needed to install the necessary library to support the board, the WiFiNINA library:

Then you can just include the necessary header file and connect to the network:

#include <WiFiNINA.h>
#define NETSSID "MYNETWORK"
#define NETPASS "SECRETPASSWORD"
WiFiClient client;
void setup()
{
Serial.begin(9600);
// Start connecting...
WiFi.begin(NETSSID, NETPASS);
// Give it a moment...
delay(5000);
if(WiFi.status() == WL_CONNECTED)
{
Serial.println("Connected!");
}
}
view raw app.ino hosted with ❤ by GitHub

To be honest, that code probably isn’t going to cut it, because WiFi networks don’t work that nicely. You need a retry mechanism with timeouts to keep trying to connect. Let’s take a look at the full example:

#include <WiFiNINA.h>
#define NETSSID "MYNETWORK"
#define NETPASS "SECRETPASSWORD"
#define TOTAL_WAIT_TIME 60000 // 1 minute
#define ATTEMPT_TIME 5000 // 5 seconds between attempts
WiFiClient client;
void setup()
{
Serial.begin(9600);
unsigned long startTime = millis();
unsigned long lastAttemptTime = 0;
int wifiStatus;
// attempt to connect to Wifi network in a loop,
// until we connect.
while (wifiStatus != WL_CONNECTED)
{
unsigned long currentTime = millis();
if(currentTime - startTime > TOTAL_WAIT_TIME)
{
// Exceeded the total timeout for trying to connect, so stop.
Serial.println("Failed to connect");
while(true);
}
else if(currentTime - lastAttemptTime > ATTEMPT_TIME)
{
// Exceeded our attempt delay, initiate again.
Serial.println("Attempting Wifi Connection");
lastAttemptTime = currentTime;
wifiStatus = WiFi.begin(NETSSID, NETPASS);
}
else
{
// wait 500ms before we check the WiFi status.
delay(500);
}
}
Serial.println("Connected!");
}
view raw app.ino hosted with ❤ by GitHub

The Server

To receive the data from the Arduino, I created a light-weight ASP.NET Core 3.0 web application with a single controller endpoint to handle incoming data, taking a timestamp and the speed:

public class DataController : Controller
{
private readonly SampleWriter sampleWriter;
public DataController(SampleWriter sampleWriter)
{
this.sampleWriter = sampleWriter;
}
[HttpPost]
public async Task<ActionResult> ProvideReading(uint milliseconds, double speed)
{
// sampleWriter is just a singleton dependency with an open file stream,
// writing each record to a CSV file as it arrives.
await sampleWriter.ProvideSample(milliseconds, speed);
return StatusCode(200);
}
}
view raw DataController.cs hosted with ❤ by GitHub

Then, in my Arduino, I put the following code in a method to send data to my application:

#define SERVER "MYSERVER"
#define SERVERPORT 5000
WiFiClient client;
void sendData(unsigned long timestamp, double speed)
{
// Host and port
if(client.connect(SERVER, SERVERPORT))
{
char body[64];
// Clear the array to zeroes.
memset(body, 0, 64);
// Arduino sprintf does not support floats or doubles.
sprintf(body, "milliseconds=%lu&speed=", timestamp);
// Use the dtostrf to append the speed.
dtostrf(speed, 2, 3, &body[strlen(body)]);
int bodyLength = strlen(body);
// Specify the endpoint
client.println("POST /data/providereading HTTP/1.1");
// Write Host: SERVER:SERVERPORT
client.print("Host: ");
client.print(SERVER);
client.print(":");
client.println(SERVERPORT);
// Close the connection after the request
client.println("Connection: close");
// Write the amount of body data
client.print("Content-Length: ");
client.println(bodyLength);
client.println("Content-Type: application/x-www-form-urlencoded");
client.println();
client.print(body);
// Wait for the response
delay(100);
// Read the response (but we don't care what is in it)
while(client.read() != -1);
}
}
view raw app.ino hosted with ❤ by GitHub

I just want to briefly mention one part of the above code, where I’m preparing body data to send.

// Arduino sprintf does not support floats or doubles.
sprintf(body, "milliseconds=%lu&speed=", timestamp);
// Use the dtostrf to append the speed.
dtostrf(speed, 2, 3, &body[strlen(body)]);
view raw app.ino hosted with ❤ by GitHub

The Arduino libraries do not support the %f specifier (for a float) in the sprintf method, so I can’t just add the speed as an argument there. Instead, you have to use the dtostrf method to insert a double into the string, specifying the number of decimal points you want.

Also, if you specify %d (int) instead of %lu (unsigned long) for the timestamp, the sprintf method treats the value as a signed int and you get very strange numbers being sent through for the timestamp.

Once that was uploaded, I started getting requests through!

Performance

We now have HTTP requests from the Arduino to our ASP.NET Core app. But I’m not thrilled with the amount of time it takes to execute a single request.

If we take a look at the WireShark trace (I love WireShark), you can see that each request from start to finish is taking in the order of 100ms!

This is loads, and I can’t have my Arduino sitting there for that long.

ASP.NET Core Performance

You can see in the above trace that the web app handling the request is taking 20ms to return the response, which is a lot. I know that ASP.NET Core can do better than that.

Turns out this problem was actually due to the fact I had console logging switched on. Due to the synchronisation that takes place when writing to the console, it can add a lot of time to requests to print all that information-level data.

Once I turned the logging down from Information to Warning in my appsettings.json file, it got way better.

That’s better!

That actually gives us sub-millisecond response times from the server, which is awesome.

TCP Handshake Overhead

Annoyingly, each request is still taking up to 100ms from start of connection to the end. How come?

If you look at those WireShark traces, we spend a lot of time in the TCP handshaking process. Opening a TCP connection does generally come with lots of network overhead, and that call to client.connect(SERVER, SERVERPORT) in my code blocks until the TCP connection is open; I don’t want to sit there waiting for that every time I want to send a sample.

The simple solution to this is to make the connection stay open between samples, so we can just repeatedly sent data on the same connection, only needing to do the handshake once.

Let’s rework our previous sendData code on the Arduino to keep the connection open:

void sendData(unsigned long timestamp, double speed)
{
char body[64];
int success = 1;
// If we're not connected, open the connection.
if(!client.connected())
{
success = client.connect(SERVER, SERVERPORT);
}
if(success)
{
// Empty the buffer
// (I still don't really care about the response)
while(client.read() != -1);
memset(body, 0, 64);
sprintf(body, "milliseconds=%lu&speed=", timestamp);
dtostrf(speed, 2, 3, &body[strlen(body)]);
int bodyLength = strlen(body);
client.println("POST /data/providereading HTTP/1.1");
client.print("Host: ");
client.print(SERVER);
client.print(":");
client.println(SERVERPORT);
// This tells the server we want to leave the
// connection open.
client.println("Connection: keep-alive");
client.print("Content-Length: ");
client.println(bodyLength);
client.println("Content-Type: application/x-www-form-urlencoded");
client.println();
client.print(body);
}
}
view raw app.ino hosted with ❤ by GitHub

In this version, we ask the server to leave the connection open after the request, and only open the connection if it is closed. I’m also not blocking waiting for a response.

This gives us way better behaviour, and we’re now down to about 40ms total:

There’s one more thing that I don’t love about this though…

TCP Packet Fragmentation

So, what’s left to look at?

TCP segments

I’ve got a packet preceding each of my POST requests, that seems to hold things up by around 40ms. What’s going on here? Let’s look at the content of that packet:

Wireshark data view

What I can tell from this is that rather than wait for my HTTP request data, the Arduino is not buffering for long enough, and is just sending what it has after the first println call containing POST /data/providereading HTTP/1.1. This packet fragmentation slows everything up because the Arduino has to wait for an ACK from the server before it continues.

I just wanted to point out that it looks like the software in the Arduino libraries isn’t responsible for the fragmentation; it looks all the TCP behaviour is handled by the hardware WiFi module, that’s what is splitting my packets.

To stop this packet fragmentation, let’s adjust the sending code to prepare the entire request and send it all at once:

void sendData(unsigned long timestamp, double speed)
{
int success = 1;
char request[256];
char body[64];
if(!client.connected())
{
success = client.connect(server, 5000);
}
if(success)
{
// Empty the buffer
// (I still don't really care about the response)
while(client.read() != -1);
// Clear the request data
memset(request, 0, 256);
// Clear the body data
memset(body, 0, 64);
sprintf(body, "milliseconds=%lu&speed=", timestamp);
dtostrf(speed, 2, 3, &body[strlen(body)]);
char* currentPos = request;
// I'm using sprintf for the fixed length strings here
// to make it easier to read.
currentPos += sprintf(currentPos, "POST /data/providereading HTTP/1.1\r\n");
currentPos += sprintf(currentPos, "Host: %s:%d\r\n", server, 5000);
currentPos += sprintf(currentPos, "Connection: keep-alive\r\n");
currentPos += sprintf(currentPos, "Content-Length: %d\r\n", strlen(body));
currentPos += sprintf(currentPos, "Content-Type: application/x-www-form-urlencoded\r\n");
currentPos += sprintf(currentPos, "\r\n");
strcpy(currentPos, body);
// Send the entire request
client.print(request);
// Force the wifi module to send the packet now
// rather than buffering any more data.
client.flush();
}
}
view raw app.ino hosted with ❤ by GitHub

Once uploaded, let’s look at the new WireShark trace:

No TCP Fragmentation

There we go! Sub-millisecond responses from the server, and precisely hitting my desired 50ms window between each sample send.

There’s still ACKs going on obviously, but they aren’t blocking packet issuing, which is the important thing.

Summary

It’s always good to look at the WireShark trace for your requests to see if you’re getting the performance you want, and don’t dismiss the overhead of opening a new TCP connection each time!

Next Steps

Next up in the ‘Modding my Rowing Machine’ series, I’ll be taking this speed data and generating a real-time graph in my browser, that updates continuously! Stay tuned…

One reply on “Streaming real-time sensor data to an ASP.NET Core app from an Arduino”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s