Categories
autofac c#

Autofac 6.0 Released

I’m super excited today, along with the rest of the Autofac team, to be able to announce the release of Autofac 6.0!

This version has got some major new features in it, and general improvements throughout the library, including an overhaul of the Autofac internals to use a customisable Resolve Pipeline when resolving services, built-in diagnostic support, support for the composite pattern, and more!

I’d like to thank everyone on the Autofac team for the load of effort that has gone into this release, I’m pretty thrilled to be able to unleash it into the world.

There’s a couple of breaking changes you should be aware of, and then I’ll go through an overview of some of the great new features at your disposal!

Breaking Changes

Despite the pretty big internal changes, the number of breaking changes are pretty low; we’ve managed to avoid any real behavioural changes.

You can see the complete set of breaking code changes between 5.x and 6.0 in our documentation. I’ll list some of the more pertinent ones here.

Framework Version Targeting Changes

Starting with Autofac 6.0, we now only target netstandard2.0 and netstandard2.1; we have removed the explicit target for net461.

The impact to you is that, while Autofac will still work on .NET Framework 4.6.1 as it did before, we strongly encourage you to upgrade to .NET Framework 4.7.2 or higher, as per the .NET Standard Documentation, to avoid any of the known dependency issues when using .NET Standard packages in .NET Framework 4.6.1.

Custom Registration Sources

If you have implemented a custom registration source you will need to update the IRegistrationSource.RegistrationsFor method.

// 5.x
IEnumerable<IComponentRegistration> RegistrationsFor(Service service, Func<Service, IEnumerable<IComponentRegistration>> registrationAccessor);

// 6.x
IEnumerable<IComponentRegistration> RegistrationsFor(Service service, Func<Service, IEnumerable<ServiceRegistration>> registrationAccessor);

The registrationAccessor parameter is a callback that, given a service, will return the set of registrations available for that service.

In 6.x, the return type of this callback was changed from IEnumerable to IEnumerable.

A ServiceRegistration encapsulates the registration (via the Registration property of the type), but also exposes the resolve pipeline Autofac needs in order to resolve a registration.

Custom Constructor Selectors

If you have implemented a custom IConstructorSelector to pass to the UsingConstructor registration method, you will need to update your implementation to use BoundConstructor instead of ConstructorParameterBinding.

The new BoundConstructor type exposes similar properties (including the TargetConstructor):

// v5.x
ConstructorParameterBinding SelectConstructorBinding(ConstructorParameterBinding[] constructorBindings, IEnumerable<Parameter> parameters);

// v6.x
BoundConstructor SelectConstructorBinding(BoundConstructor[] constructorBindings, IEnumerable<Parameter> parameters);

New Features/Improvements

There are a tonne of new features in Autofac 6.0; I’ll hit some of the highlights here.

Pipelines

The internals of Autofac have been through a major overhaul, so that the work of actually resolving an instance of a registration is implemented as a pipeline, consisting of middleware that handles each part of the process.

The existing ways you configure Autofac haven’t changed, but we have added some powerful new extensibility points you can use for advanced scenarios.

For example, you can add pipeline middleware to all resolves of a service, that runs before any built-in Autofac code:

var builder = new ContainerBuilder();

// Run some middleware at the very start of the pipeline, before any core Autofac behaviour.
builder.RegisterServiceMiddleware<IMyService>(PipelinePhase.ResolveRequestStart, (context, next) =>
{
    Console.WriteLine("Requesting Service: {0}", context.Service);

    // Continue the pipeline.
    next(context);
});

Anyone familiar with ASP.NET Core middleware may notice some similarities here! We have a context, and a next method to call to continue the pipeline.

You can check out our detailed docs on pipelines for a complete run down on how these work, and how to use them.

A lot of the following new features are only possible because of the pipeline change; it gave us the flexibility to do new and interesting things!

Support for the Composite Pattern

For some time we’ve been working towards adding built-in support for the Composite Pattern, going back to 2016.

Well, it’s finally here, and gives you the new RegisterComposite method on the ContainerBuilder!

Here’s an example from our documentation, where we have multiple log sinks that we want to wrap in a CompositeLogSink:

var builder = new ContainerBuilder();

// Here are our normal implementations.
builder.RegisterType<FileLogSink>().As<ILogSink>();
builder.RegisterType<DbLogSink>().As<ILogSink>();

// We're going to register a class to act as a Composite wrapper for ILogSink
builder.RegisterComposite<CompositeLogSink, ILogSink>();

var container = builder.Build();

// This will return an instance of `CompositeLogSink`.
var logSink = container.Resolve<ILogSink>();

logSink.WriteLog("log message");

// ...

// Here's our composite class; it's just a regular class that injects a
// collection of the same service.
public class CompositeLogSink : ILogSink
{
    private readonly IEnumerable<ILogSink> _implementations;

    public CompositeLogSink(IEnumerable<ILogSink> implementations)
    {
        // implementations will contain all the 'actual' registrations.
        _implementations = implementations;
    }

    public void WriteLog(string log)
    {
        foreach (var sink in _implementations)
        {
            sink.WriteLog(log);
        }
    }
}

Thanks to @johneking for his input and feedback on the design of the composites implementation.

There’s more guidance around how to use composites (including how to register open-generic composites, and use relationships like Lazy and Func) in our documentation on composites.

Diagnostic Tracing

One thing that has always been a bit challenging with Autofac (and Dependency Injection in general really), is figuring out why something isn’t working, and particularly which one of your services in your really complex object graph is causing your problem!

Happily, in Autofac 6.0, we have added built-in support for the .NET DiagnosticSource class, and we generate diagnostic events while we are resolving a service.

The easiest way to get started with our diagnostics is using the Autofac DefaultDiagnosticTracer, which will generate a tree-like view of each resolve, with dependencies, showing you exactly where things go wrong.

var builder = new ContainerBuilder();

// A depends on B1 and B2, but B2 is going to fail.
builder.RegisterType<A>();
builder.RegisterType<B1>();
builder.Register<B2>(ctx => throw new InvalidOperationException("No thanks."));

var container = builder.Build();

// Let's add a tracer.
var tracer = new DefaultDiagnosticTracer();
tracer.OperationCompleted += (sender, args) =>
{
    // TraceContent contains the output.
    Trace.WriteLine(args.TraceContent);
};

container.SubscribeToDiagnostics(tracer);

// Resolve A - will fail.
container.Resolve<A>();

When that Resolve<A>() call completes, our tracer’s event handler will fire, and TraceContent contains your verbose trace:

Resolve Operation Starting
{
  Resolve Request Starting
  {
    Service: AutofacDotGraph.A
    Component: AutofacDotGraph.A

    Pipeline:
    -> CircularDependencyDetectorMiddleware
      -> ScopeSelectionMiddleware
        -> SharingMiddleware
          -> RegistrationPipelineInvokeMiddleware
            -> ActivatorErrorHandlingMiddleware
              -> DisposalTrackingMiddleware
                -> A (ReflectionActivator)
                  Resolve Request Starting
                  {
                    Service: AutofacDotGraph.B1
                    Component: AutofacDotGraph.B1

                    Pipeline:
                    -> CircularDependencyDetectorMiddleware
                      -> ScopeSelectionMiddleware
                        -> SharingMiddleware
                          -> RegistrationPipelineInvokeMiddleware
                            -> ActivatorErrorHandlingMiddleware
                              -> DisposalTrackingMiddleware
                                -> B1 (ReflectionActivator)
                                <- B1 (ReflectionActivator)
                              <- DisposalTrackingMiddleware
                            <- ActivatorErrorHandlingMiddleware
                          <- RegistrationPipelineInvokeMiddleware
                        <- SharingMiddleware
                      <- ScopeSelectionMiddleware
                    <- CircularDependencyDetectorMiddleware
                  }
                  Resolve Request Succeeded; result instance was AutofacDotGraph.B1
                  Resolve Request Starting
                  {
                    Service: AutofacDotGraph.B2
                    Component: λ:AutofacDotGraph.B2

                    Pipeline:
                    -> CircularDependencyDetectorMiddleware
                      -> ScopeSelectionMiddleware
                        -> SharingMiddleware
                          -> RegistrationPipelineInvokeMiddleware
                            -> ActivatorErrorHandlingMiddleware
                              -> DisposalTrackingMiddleware
                                -> λ:AutofacDotGraph.B2
                                X- λ:AutofacDotGraph.B2
                              X- DisposalTrackingMiddleware
                            X- ActivatorErrorHandlingMiddleware
                          X- RegistrationPipelineInvokeMiddleware
                        X- SharingMiddleware
                      X- ScopeSelectionMiddleware
                    X- CircularDependencyDetectorMiddleware
                  }
                  Resolve Request FAILED
                    System.InvalidOperationException: No thanks.
                       at AutofacExamples.<>c.<ErrorExample>b__0_0(IComponentContext ctx) in D:\Experiments\Autofac\Examples.cs:line 24
                       at Autofac.RegistrationExtensions.<>c__DisplayClass39_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
                       at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
                       at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
                       ...
                X- A (ReflectionActivator)
              X- DisposalTrackingMiddleware
            X- ActivatorErrorHandlingMiddleware
          X- RegistrationPipelineInvokeMiddleware
        X- SharingMiddleware
      X- ScopeSelectionMiddleware
    X- CircularDependencyDetectorMiddleware
  }
  Resolve Request FAILED: Nested Resolve Failed
}
Operation FAILED

There’s a lot there, but you can see the start and end of the request for each of the child dependencies, including content telling you exactly which registration failed and every pipeline middleware visited during the operation.

We’re hoping this will help people investigate problems in their container, and make it easier to support you!

We’ve got some detailed documentation on diagnostics, including how to set up your own tracers, go check it out for more info.

Visualising your Services

Building on top of the diagnostics support I just mentioned, we’ve also added support for outputting graphs (in DOT format) representing your resolve operation, which can then be rendered to an image, using the Graphviz tools (or anything that can render the DOT format).

This feature is available in the new NuGet package, Autofac.Diagnostics.DotGraph.

var builder = new ContainerBuilder();

// Here's my complicated(ish) dependency graph.
builder.RegisterType<A>();
builder.RegisterType<B1>();
builder.RegisterType<B2>();
builder.RegisterType<C1>();
builder.RegisterType<C2>().SingleInstance();

var container = builder.Build();

// Using the new DOT tracer here.
var tracer = new DotDiagnosticTracer();
tracer.OperationCompleted += (sender, args) =>
{
    // Writing to file in-line may not be ideal, this is just an example.
    File.WriteAllText("graphContent.dot", args.TraceContent);
};

container.SubscribeToDiagnostics(tracer);

container.Resolve<A>();

Once I convert this to a visual graph (there’s a useful VSCode Extension that will quickly preview the graph for you), I get this:

If you’ve got a big dependency graph, hopefully this will help you understand the chain of dependencies more readily!

There’s more information on the DOT Graph support in our documentation.

Pooled Instances

A new Autofac package, Autofac.Pooling, is now available that provides the functionality to maintain a pool of object instances within your Autofac container.

The idea is that, for certain resources (like connections to external components), rather than get a new instance for every lifetime scope, which is disposed at the end of the scope, you can retrieve from a container-shared pool of these objects, and return to the pool at the end of the scope.

You can do this by configuring a registration with PooledInstancePerLifetimeScope or PooledInstancePerMatchingLifetimeScope methods:

var builder = new ContainerBuilder();

// Configure my pooled registration.
builder.RegisterType<MyCustomConnection>()
        .As<ICustomConnection>()
        .PooledInstancePerLifetimeScope();

var container = builder.Build();

using (var scope = container.BeginLifetimeScope())
{
    // Creates a new instance of MyCustomConnection
    var instance = scope.Resolve<ICustomConnection>();

    instance.DoSomething();
}

// When the scope ends, the instance of MyCustomConnection
// is returned to the pool, rather than being disposed.

using (var scope2 = container.BeginLifetimeScope())
{
    // Does **not** create a new instance, but instead gets the
    // previous instance from the pool.
    var instance = scope.Resolve<ICustomConnection>();

    instance.DoSomething();
}

// Instance gets returned back to the pool again at the
// end of the lifetime scope.

You can resolve these pooled services like any normal service, but you’ll be getting an instance from the pool when you do!

Check out the documentation on pooled instances for details on how to control pool capacity, implement custom behaviour when instances are retrieved/returned to the pool, and even how to implement custom pool policies to do interesting things like throttle your application based on the capacity of the pool!

Generic Delegate Registrations

Autofac has had the concept of open generic registrations for some time, where you can specify an open-generic type to provide an open-generic service.

var builder = new ContainerBuilder();
// Register a generic that will provide closed types of IService<>
builder.RegisterGeneric(typeof(Implementation<>)).As(typeof(IService<>));

In Autofac 6.0, we’ve added the ability to register a delegate to provide the generic, instead of a type, for advanced scenarios where you need to make custom decisions about the resulting closed type.

var builder = new ContainerBuilder();

builder.RegisterGeneric((ctxt, types, parameters) =>
{
    // Make decisions about what closed type to use.
    if (types.Contains(typeof(string)))
    {
        return new StringSpecializedImplementation();
    }

    return Activator.CreateInstance(typeof(GeneralImplementation<>).MakeGenericType(types));
}).As(typeof(IService<>));

Concurrency Performance Improvements

There’s been a lot of work into performance with this release of Autofac; particularly around performance in highly-concurrent scenarios, like web servers.

We’ve removed a load of locking from the core of Autofac, to the point that once a service has been resolved once from a lifetime scope, subsequent resolves of that service are lock-free.

In some highly-concurrent scenarios, we’ve seen a 4x reduction in the time it takes to resolve objects through Autofac!

Thanks @alsami for the work on automating our benchmark execution, @twsouthwick for work on caching generated delegate types, and @DamirAinullin for varied performance tweaks.

Other Changes

  • Support async handlers for OnPreparing, OnActivating, OnActivated and OnRelease (PR#1172).
  • Circular Dependency depth changes to allow extremely deep dependency graphs that have no circular references (PR#1148).
  • ContainerBuilder is now sealed (Issue#1120).
  • Explicitly injected properties can now be declared using an expression (PR#1123, thanks @mashbrno).

Still Todo

We’re working hard to get all of the ~25 integration packages pushed to NuGet as quickly as we can, so please bear with us while we get these sorted.

Some of this is sitting in branches ready to go, other things need to be done now that we have this core package out there.

If your favorite integration isn’t ready yet, we’re doing our best. Rather than filing "When will this be ready?" issues, consider pull requests with the required updates.

Thank You!

I’d like to personally thank all the contributors who contributed to the 6.0 release since we shipped 5.0:

Hopefully the Github Contributors page hasn’t let me down, I wouldn’t want to miss anyone!

Categories
c# Uncategorized

Making Users Re-Enter their Password: ASP.NET Core & IdentityServer4

It’s often good security behaviour to require users to re-enter their password when they want to change some secure property of their account, like generate personal access tokens, or change their Multi-factor Authentication (MFA) settings.

You may have seen the Github ‘sudo’ mode, which asks you to re-enter your password when you try to change something sensitive.

Sudo Mode Dialog
The Github sudo mode prompt.

Most of the time a user’s session is long-lived, so when they want to do something sensitive, it’s best to check they still are who they say.

I’ve been working on the implementation of IdentityServer4 at Enclave for the past week or so, and had this requirement to require password confirmation before users can modify their MFA settings.

I thought I’d write up how I did this for posterity, because it took a little figuring out.

The Layout

In our application, we have two components, both running on ASP.NET Core 3.1

  • The Accounts app that holds all the user data; this is where Identity Server runs; we use ASP.NET Core Identity to do the actual user management.
  • The Portal app that holds the UI. This is a straightforward MVC app right now, no JS or SPA to worry about.

To make changes to a user’s account settings, the Profile Controller in the Portal app makes API calls to the Accounts app.

The Portal calls APIs in the Accounts app

All the API calls to the Accounts app are already secured using the Access Token from when the user logged in; we have an ASP.NET Core Policy in place for our additional API (as per the IdentityServer docs) to protect it.

The Goal

The desired outcome here is that specific sensitive API endpoints within the Accounts app require the calling user to have undergone a second verification, where they must have re-entered their password recently in order to use the API.

What we want to do is:

  • Allow the Portal app to request a ‘step-up’ access token from the Accounts app.
  • Limit the step-up access token to a short lifetime (say 15 minutes), with no refresh tokens.
  • Call a sensitive API on the Accounts App, and have the Accounts App validate the step-up token.

Issuing the Step-Up Token

First up, we need to generate a suitable access token when asked. I’m going to add a new controller, StepUpApiController, in the Accounts app.

This controller is going to have a single endpoint, which requires a regular access token before you can call it.

We’re going to use the provided IdentityServerTools class, that we can inject into our controller, to do the actual token generation.

Without further ado, let’s look at the code for the controller:

[Route("api/stepup")]
[ApiController]
[Authorize(ApiScopePolicy.WriteUser)]
public class StepUpApiController : ControllerBase
{
    private static readonly TimeSpan ValidPeriod = TimeSpan.FromMinutes(15);

    private readonly UserManager<ApplicationUser> _userManager;
    private readonly IdentityServerTools _idTools;

    public StepUpApiController(UserManager<ApplicationUser> userManager,
                               IdentityServerTools idTools)
    {
        _userManager = userManager;
        _idTools = idTools;
    }

    [HttpPost]
    public async Task<StepUpApiResponse> StepUp(StepUpApiModel model)
    {
        var user = await _userManager.GetUserAsync(User);

        // Verify the provided password.
        if (await _userManager.CheckPasswordAsync(user, model.Password))
        {
            var clientId = User.FindFirstValue(JwtClaimTypes.ClientId);

            var claims = new Claim[]
            {
                new Claim(JwtClaimTypes.Subject, User.FindFirstValue(JwtClaimTypes.Subject)),
            };

            // Create a token that:
            //  - Is associated to the User's client.
            //  - Is only valid for our configured period (15 minutes)
            //  - Has a single scope, indicating that the token can only be used for stepping up.
            //  - Has the same subject as the user.
            var token = await _idTools.IssueClientJwtAsync(
                clientId,
                (int)ValidPeriod.TotalSeconds,
                new[] { "account-stepup" },
                additionalClaims: claims);

            return new StepUpApiResponse { Token = token, ValidUntil = DateTime.UtcNow.Add(ValidPeriod) };
        }

        Response.StatusCode = StatusCodes.Status401Unauthorized;

        return null;
    }
}

A couple of important points here:

  • In order to even access this API, the normal access token being passed in the requested must conform to our own WriteUser scope policy, which requires a particular scope be in the access token to get to this API.
  • This generated access token is really basic; it has a single scope, “account-stepup”, and only a single additional claim containing the subject.
  • We associate the step-up token to the same client ID as the normal access token, so only the requesting client can use that token.
  • We explicitly state a relatively short lifetime on the token (15 minutes here).

Sending the Token

This is the easy bit; once you have the token, you can store it somewhere in the client, and send it in a subsequent request.

Before sending the step-up token, you’ll want to check the expiry on it, and if you need a new one, then prompt the user for their credentials and start the process again.

For any request to the sensitive API, we need to include both the normal access token from the user’s session, plus the new step-up token.

I set this up when I create the HttpClient:

private async Task<HttpClient> GetClient(string? stepUpToken = null)
{
    var client = new HttpClient();

    // Set the base address to the URL of our Accounts app.
    client.BaseAddress = _accountUrl;

    // Get the regular user access token in the session and add that as the normal
    // Authorization Bearer token.
    // _contextAccessor is an instance of IHttpContextAccessor.
    var accessToken = await _contextAccessor.HttpContext.GetUserAccessTokenAsync();
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

    if (stepUpToken is object)
    {
        // We have a step-up token; include it as an additional header (without the Bearer indicator).
        client.DefaultRequestHeaders.Add("X-Authorization-StepUp", stepUpToken);
    }

    return client;
}

That X-Authorization-StepUp header is where we’re going to look when checking for the token in the Accounts app.

Validating the Step-Up Token

To validate a provided step-up token in the Accounts app, I’m going to define a custom ASP.NET Core Policy that requires the API call to provide a step-up token.

If there are terms in here that don’t seem immediately obvious, check out the docs on Policy-based authorization in ASP.NET Core. It’s a complex topic, but the docs do a pretty good job of breaking it down.

Let’s take a look at an API call endpoint that requires step-up:

[ApiController]
[Route("api/user")]
[Authorize(ApiScopePolicy.WriteUser)]
public class UserApiController : Controller
{
    [HttpPost("totp-enable")]
    [Authorize("require-stepup")]
    public async Task<IActionResult> EnableTotp(TotpApiEnableModel model)
    {
        // ... do stuff ...
    }
}

That Authorize attribute I placed on the action method specifies that we want to enforce a require-stepup policy on this action. Authorize attributes are additive, so a request to EnableTotp requires both our normal WriteUser policy and our step-up policy.

Defining our Policy

To define our require-stepup policy, lets jump over to our Startup class; specifically, in ConfigureServices, where we set up Authorization using the AddAuthorization method:

services.AddAuthorization(options =>
{
    // Other policies omitted...

    options.AddPolicy("require-stepup", policy =>
    { 
        policy.AddAuthenticationSchemes("local-api-scheme");
        policy.RequireAuthenticatedUser();
        
        // Add a new requirement to the policy (for step-up).
        policy.AddRequirements(new StepUpRequirement());
    });
});

The ‘local-api-scheme’ is the built-in scheme provided by IdentityServer for protecting local API calls.

That requirement class, StepUpRequirement is just a simple marker class for indicating to the policy that we need step-up. It’s also how we wire up a handler to check that requirement:

public class StepUpRequirement : IAuthorizationRequirement
{
}

Defining our Authorization Handler

We now need an Authorization Handler that lets us check incoming requests meet our new step-up requirement.

So, let’s create one:

public class StepUpAuthorisationHandler : AuthorizationHandler<StepUpRequirement>
{
    private const string StepUpTokenHeader = "X-Authorization-StepUp";

    private readonly IHttpContextAccessor _httpContextAccessor;
    private readonly ITokenValidator _tokenValidator;

    public StepUpAuthorisationHandler(
        IHttpContextAccessor httpContextAccessor,
        ITokenValidator tokenValidator)
    {
        _httpContextAccessor = httpContextAccessor;
        _tokenValidator = tokenValidator;
    }

    /// <summary>
    /// Called by the framework when we need to check a request.
    /// </summary>
    protected override async Task HandleRequirementAsync(
        AuthorizationHandlerContext context,
        StepUpRequirement requirement)
    {
        // Only interested in authenticated users.
        if (!context.User.IsAuthenticated())
        {
            return;
        }

        var httpContext = _httpContextAccessor.HttpContext;

        // Look for our special request header.
        if (httpContext.Request.Headers.TryGetValue(StepUpTokenHeader, out var stepUpHeader))
        {
            var headerValue = stepUpHeader.FirstOrDefault();

            if (!string.IsNullOrEmpty(headerValue))
            {
                // Call our method to check the token.
                var validated = await ValidateStepUp(context.User, headerValue);

                // Token was valid, so succeed.
                // We don't explicitly have to fail, because that is the default.
                if (validated)
                {
                    context.Succeed(requirement);
                }
            }
        }
    }

    private async Task<bool> ValidateStepUp(ClaimsPrincipal user, string header)
    {
        // Use the normal token validator to check the access token is valid, and contains our
        // special expected scope.
        var validated = await _tokenValidator.ValidateAccessTokenAsync(header, "account-stepup");

        if (validated.IsError)
        {
            // Bad token.
            return false;
        }

        // Validate that the step-up token is for the same client as the access token.
        var clientIdClaim = validated.Claims.FirstOrDefault(x => x.Type == JwtClaimTypes.ClientId);

        if (clientIdClaim is null || clientIdClaim.Value != user.FindFirstValue(JwtClaimTypes.ClientId))
        {
            return false;
        }

        // Confirm a subject is supplied.
        var subjectClaim = validated.Claims.FirstOrDefault(x => x.Type == JwtClaimTypes.Subject);

        if (subjectClaim is null)
        {
            return false;
        }

        // Confirm that the subject of the stepup and the current user are the same.
        return subjectClaim.Value == user.FindFirstValue(JwtClaimTypes.Subject);
    }
}

Again, let’s take a look at the important bits of the class:

  • The handler derives from AuthorizationHandler<StepUpRequirement>, indicating to ASP.NET that we are a handler for our custom requirement.
  • We stop early if there is no authenticated user; that’s because the step-up token is only valid for a user who is already logged in.
  • We inject and use IdentityServer’s ITokenValidator interface to let us validate the token using ValidateAccessTokenAsync; we specify the scope we require.
  • We check that the client ID of the step-up token is the same as the regular access token used to authenticate with the API.
  • We check that the subjects match (i.e. this step-up token is for the same user).

The final hurdle is to register our authorization handler in our Startup class:

services.AddSingleton<IAuthorizationHandler, StepUpAuthorisationHandler>();

Wrapping Up

There you go, we’ve now got a secondary access token being issued to indicate step-up has been done, and we’ve got a custom authorization handler to check our new token.

Categories
Uncategorized

Loading Plugins/Extensions at Run Time from NuGet in .NET Core : Part 1 – NuGet

This post is the first in a short series, writing up my efforts creating an plugin/extension system working in .NET Core that:

  • Loads extension packages from NuGet, with all their dependencies (this post).
  • Loads the extensions into my .NET Core Process.
  • Allows the loaded extensions to be unloaded.

Background

As a bit of context, I’m currently building an open-source BDD testing platform that goes beyond Gherkin, AutoStep, which is built entirely in C#, on top of .NET Core 3.1.

In AutoStep, I need to be able to load in extensions that provide additional functionality for AutoStep. For example, extensions might provide:

  • Bindings for some UI platform or library
  • Custom Report Formats
  • Integration with some external Test Management System

In terms of what’s in them, AutoStep extensions are going to consist of things like:

  • .NET DLLs
  • AutoStep Test Files
  • Dependencies on various NuGet packages (Selenium.WebDriver anyone?).

All of the above items fit pretty well within the scope of NuGet packages, and I don’t want to build my own extension packaging, hosting, versioning and so on, so I’m going to say that each extension can be represented by a NuGet package.

AutoStep does not require the .NET Core SDK to build or run any tests, so I can’t just create a csproj, chuck PackageReferences in and be done with it.

I need to bake the idea of extensions into the platform itself.


If you want to jump ahead, you can check out the GitHub repository for AutoStep.Extensions, which provides the NuGet package used to load extensions into our VS Code Language Server and our commmand-line runner.

Loading Extensions from NuGet

Microsoft supplies the NuGet Client SDK, to work with both NuGet packages and source repositories; specifically the NuGet.Protocol and NuGet.Resolver packages.

The documentation on how to actually use the NuGet Client libraries is a bit sparse, so I’m permanently indebted to Martin Bjorkstrom for writing a blog post on it that I used as a pretty detailed guide to get me started.

Loading our extension packages from NuGet involves three phases:

  1. Determine the best version of an extension package to install, given a version range (and normal NuGet rules).
    For example, if the version of the extension requested is 1.4.0, and there is a 1.4.5 version available, we want that one.
  2. Get the list of all NuGet package dependencies (recursively) for each extension.
  3. Download and Extract your packages.

Choosing the Extension Version

This is (relatively) the easy bit. First up, we’ll create some of the context objects we need to get started:

/// <summary>
/// Represents the configuration for a single extension to install.
/// </summary>
public class ExtensionConfiguration
{
    public string Package { get; set; }
    public string Version { get; set; }
    public bool PreRelease { get; set; }
}

public async Task LoadExtensions()
{
    // Define a source provider, with the main NuGet feed, plus my own feed.
    var sourceProvider = new PackageSourceProvider(NullSettings.Instance, new[]
    {
        new PackageSource("https://api.nuget.org/v3/index.json"),
        new PackageSource("https://f.feedz.io/autostep/ci/nuget/index.json")
    });

    // Establish the source repository provider; the available providers come from our custom settings.
    var sourceRepositoryProvider = new SourceRepositoryProvider(sourceProvider, Repository.Provider.GetCoreV3());

    // Get the list of repositories.
    var repositories = sourceRepositoryProvider.GetRepositories();

    // Disposable source cache.
    using var sourceCacheContext = new SourceCacheContext();

    // You should use an actual logger here, this is a NuGet ILogger instance.
    var logger = new NullLogger();

    // My extension configuration:
    var extensions = new[]
    { 
        new ExtensionConfiguration
        {
            Package = "AutoStep.Web",
            PreRelease = true // Allow pre-release versions.
        }
    };
}

Next, let’s write a method to actually get the desired package identity to install. The GetPackageIdentity method goes through each repository, and either:

  • Picks the latest available version if no version range has been configured or,
  • If a version range has been specified, uses the provided NuGet VersionRange class to find the best match given the set of all versions.
private async Task<PackageIdentity> GetPackageIdentity(
          ExtensionConfiguration extConfig, SourceCacheContext cache, ILogger nugetLogger,
          IEnumerable<SourceRepository> repositories, CancellationToken cancelToken)
{
    // Go through each repository.
    // If a repository contains only pre-release packages (e.g. AutoStep CI), and 
    // the configuration doesn't permit pre-release versions,
    // the search will look at other ones (e.g. NuGet).
    foreach (var sourceRepository in repositories)
    {
        // Get a 'resource' from the repository.
        var findPackageResource = await sourceRepository.GetResourceAsync<FindPackageByIdResource>();

        // Get the list of all available versions of the package in the repository.
        var allVersions = await findPackageResource.GetAllVersionsAsync(extConfig.Package, cache, nugetLogger, cancelToken);

        NuGetVersion selected;

        // Have we specified a version range?
        if (extConfig.Version != null)
        {
            if (!VersionRange.TryParse(extConfig.Version, out var range))
            {
                throw new InvalidOperationException("Invalid version range provided.");
            }

            // Find the best package version match for the range.
            // Consider pre-release versions, but only if the extension is configured to use them.
            var bestVersion = range.FindBestMatch(allVersions.Where(v => extConfig.PreRelease || !v.IsPrerelease));

            selected = bestVersion;
        }
        else
        {
            // No version; choose the latest, allow pre-release if configured.
            selected = allVersions.LastOrDefault(v => v.IsPrerelease == extConfig.PreRelease);
        }

        if (selected is object)
        {
            return new PackageIdentity(extConfig.Package, selected);
        }
    }

    return null;
}

Let’s plug that code into our previous code, so we’re now getting the identity:

// ...

// My extension configuration:
var extensions = new[] { new ExtensionConfiguration
{
    Package = "AutoStep.Web",
    PreRelease = true // Allow pre-release versions.
}};

foreach (var ext in extensions)
{
    var packageIdentity = await GetPackageIdentity(ext, sourceCacheContext, logger, repositories, CancellationToken.None);

    if (packageIdentity is null)
    {
        throw new InvalidOperationException($"Cannot find package {ext.Package}.");
    }
}

With this we get a package identity of AutoStep.Web.1.0.0-develop.20 (the latest pre-release version at the time).

Get the List of Package Dependencies

This is where things get interesting. We need to get the complete set of all dependencies, across all the extensions, that we need to install in order to use the extension package.

First off, let’s look at an initial, very naive solution, which just does a straight-forward recurse through the entire dependency graph.

private async Task GetPackageDependencies(PackageIdentity package, SourceCacheContext cacheContext, 
                                          NuGetFramework framework, ILogger logger, 
                                          IEnumerable<SourceRepository> repositories,
                                          ISet<SourcePackageDependencyInfo> availablePackages, 
                                          CancellationToken cancelToken)
{
    // Don't recurse over a package we've already seen.
    if (availablePackages.Contains(package))
    {
        return;
    }

    foreach (var sourceRepository in repositories)
    {
        // Get the dependency info for the package.
        var dependencyInfoResource = await sourceRepository.GetResourceAsync<DependencyInfoResource>();
        var dependencyInfo = await dependencyInfoResource.ResolvePackage(
            package,
            framework,
            cacheContext,
            logger,
            cancelToken);

        // No info for the package in this repository.
        if (dependencyInfo == null)
        {
            continue;
        }

        // Add to the list of all packages.
        availablePackages.Add(dependencyInfo);

        // Recurse through each package.
        foreach (var dependency in dependencyInfo.Dependencies)
        {
            await GetPackageDependencies(
                new PackageIdentity(dependency.Id, dependency.VersionRange.MinVersion),
                cacheContext,
                framework,
                logger,
                repositories,
                availablePackages,
                cancelToken);
        }

        break;
    }
}

That does indeed create the complete graph of all libraries required by that extension, the problem is that it has 104 packages in it!

Long package list

I’ve got the AutoStep.Web package at the top there, but I’ve also got
System.Runtime, which I definitely don’t want.

All the extensions are going to reference the AutoStep.Extensions.Abstractions package (because that’s where we define our interfaces for extensions), but we don’t want to download it ourselves!

Besides the fact that we don’t need to download these shared packages, if we load in the AutoStep.Extensions.Abstractions assembly from the extension’s dependencies, it will not be compatible with the version referenced by the host process.

The actual requirement for our behaviour here is:

All packages provided by the host process should be excluded from the set of dependencies to install.

Filtering the Dependencies

At runtime, how do we know what the set of installed packages are for a .NET Core Application? Luckily, there happens to be an existing file containing this information, the {AssemblyName}.deps.json file that gets copied to your output directory.

You probably haven’t had to worry about it much, but if you look in your application’s output directory, you’ll find it.

It contains the complete package reference graph for your application, and looks a little something like this:

{
  "runtimeTarget": {
    "name": ".NETCoreApp,Version=v3.1",
    "signature": ""
  },
  "compilationOptions": {},
  "targets": {
    ".NETCoreApp,Version=v3.1": {
      "NugetConsole/1.0.0": {
        "dependencies": {
          "Microsoft.Extensions.DependencyModel": "3.1.3",
          "NuGet.Protocol": "5.5.1",
          "NuGet.Resolver": "5.5.1"
        },
        "runtime": {
          "NugetConsole.dll": {}
        }
      },
      "Microsoft.CSharp/4.0.1": {
        "dependencies": {
          "System.Collections": "4.3.0",
          "System.Diagnostics.Debug": "4.3.0",
          "System.Dynamic.Runtime": "4.3.0",
          "System.Globalization": "4.3.0",
          "System.Linq": "4.3.0",
          "System.Linq.Expressions": "4.3.0",
          "System.ObjectModel": "4.3.0",

// ...a lot more content...

Handily, we don’t have to parse this ourselves. If you add the Microsoft.Extensions.DependencyModel package to your project, you can directly access this content using DependencyContext.Default, which gives you a DependencyContext you can interrogate.

Let’s define a method that takes this DependencyContext and a PackageDependency, and checks whether it is provided by the host:

private bool DependencySuppliedByHost(DependencyContext hostDependencies, PackageDependency dep)
{
    // See if a runtime library with the same ID as the package is available in the host's runtime libraries.
    var runtimeLib = hostDependencies.RuntimeLibraries.FirstOrDefault(r => r.Name == dep.Id);

    if (runtimeLib is object)
    {
        // What version of the library is the host using?
        var parsedLibVersion = NuGetVersion.Parse(runtimeLib.Version);

        if (parsedLibVersion.IsPrerelease)
        {
            // Always use pre-release versions from the host, otherwise it becomes
            // a nightmare to develop across multiple active versions.
            return true;
        }
        else
        {
            // Does the host version satisfy the version range of the requested package?
            // If so, we can provide it; otherwise, we cannot.
            return dep.VersionRange.Satisfies(parsedLibVersion);
        }
    }

    return false;
}

Then, let’s plug that in to our existing GetPackageDependencies method:

private async Task GetPackageDependencies(PackageIdentity package, SourceCacheContext cacheContext, NuGetFramework framework, 
                                          ILogger logger, IEnumerable<SourceRepository> repositories, DependencyContext hostDependencies,
                                          ISet<SourcePackageDependencyInfo> availablePackages, CancellationToken cancelToken)
{
    // Don't recurse over a package we've already seen.
    if (availablePackages.Contains(package))
    {
        return;
    }

    foreach (var sourceRepository in repositories)
    {
        // Get the dependency info for the package.
        var dependencyInfoResource = await sourceRepository.GetResourceAsync<DependencyInfoResource>();
        var dependencyInfo = await dependencyInfoResource.ResolvePackage(
            package,
            framework,
            cacheContext,
            logger,
            cancelToken);

        // No info for the package in this repository.
        if (dependencyInfo == null)
        {
            continue;
        }


        // Filter the dependency info.
        // Don't bring in any dependencies that are provided by the host.
        var actualSourceDep = new SourcePackageDependencyInfo(
            dependencyInfo.Id,
            dependencyInfo.Version,
            dependencyInfo.Dependencies.Where(dep => !DependencySuppliedByHost(hostDependencies, dep)),
            dependencyInfo.Listed,
            dependencyInfo.Source);

        availablePackages.Add(actualSourceDep);

        // Recurse through each package.
        foreach (var dependency in actualSourceDep.Dependencies)
        {
            await GetPackageDependencies(
                new PackageIdentity(dependency.Id, dependency.VersionRange.MinVersion),
                cacheContext,
                framework,
                logger,
                repositories,
                hostDependencies,
                availablePackages,
                cancelToken);
        }

        break;
    }
}

This cuts down on the set of packages significantly, but it’s still pulling down some runtime-provided packages I don’t want:

AutoStep.Web : 1.0.0-develop.20       // correct
Selenium.Chrome.WebDriver : 79.0.0    // correct
Selenium.WebDriver : 3.141.0          // correct
Newtonsoft.Json : 10.0.3              // correct
Microsoft.CSharp : 4.3.0              // Ah. This is a runtime package...
System.ComponentModel.TypeConverter : 4.3.0 
System.Collections.NonGeneric : 4.3.0       
System.Collections.Specialized : 4.3.0
System.ComponentModel : 4.3.0
System.ComponentModel.Primitives : 4.3.0
System.Runtime.Serialization.Primitives : 4.3.0
System.Runtime.Serialization.Formatters : 4.3.0
System.Xml.XmlDocument : 4.3.0

So, something is still not right. What’s causing these packages to be present?

Well, simply put, my program doesn’t use System.ComponentModel, so it isn’t in the list of my dependencies. But it is provided by the host, because it’s part of the distributed .NET Runtime.

Ignoring Runtime-Provided Packages

We want to filter out runtime-provided packages completely, but how do we know which ones to exclude? We can’t just filter out any System.* packages, because there are a number of System.* packages that aren’t shipped with the runtime (e.g. System.Text.Json).

As far as I can tell, it’s more or less impossible to determine the full set at run time dynamically.

After some considerable searching however, I found a complete listing of all runtime-provided packages in an MSBuild task in the dotnet SDK, called PackageConflictOverrides, which tells the build system which packages don’t need to be restored! Yay!

This allowed me to define the following static lookup class (excerpt only). You can find a full version here.

/// <summary>
/// Contains a pre-determined list of NuGet packages that are provided by the run-time, and
/// therefore should not be restored from an extensions dependency list.
/// </summary>
internal static class RuntimeProvidedPackages
{
    /// <summary>
    /// Checks whether the set of known runtime packages contains the given package ID.
    /// </summary>
    /// <param name="packageId">The package ID.</param>
    /// <returns>True if the package is provided by the framework, otherwise false.</returns>
    public static bool IsPackageProvidedByRuntime(string packageId)
    {
        return ProvidedPackages.Contains(packageId);
    }

    /// <summary>
    /// This list comes from the package overrides for the .NET SDK,
    /// at https://github.com/dotnet/sdk/blob/v3.1.201/src/Tasks/Common/targets/Microsoft.NET.DefaultPackageConflictOverrides.targets.
    /// If the executing binaries ever change to a newer version, this project must update as well, and refresh this list.
    /// </summary>
    private static readonly ISet<string> ProvidedPackages = new HashSet<string>
    {
        "Microsoft.CSharp",
        "Microsoft.Win32.Primitives",
        "Microsoft.Win32.Registry",
        "runtime.debian.8-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.fedora.23-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.opensuse.42.1-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple",
        "runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.rhel.7-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.16.04-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.16.10-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "System.AppContext",
        "System.Buffers",
        "System.Collections",
        "System.Collections.Concurrent",
        // Removed a load for brevity....
        "System.Xml.ReaderWriter",
        "System.Xml.XDocument",
        "System.Xml.XmlDocument",
        "System.Xml.XmlSerializer",
        "System.Xml.XPath",
        "System.Xml.XPath.XDocument",
    };
}

Ok, so let’s update our DependencySuppliedByHost method to use this look-up:

private bool DependencySuppliedByHost(DependencyContext hostDependencies, PackageDependency dep)
{
    // Check our look-up list.
    if(RuntimeProvidedPackages.IsPackageProvidedByRuntime(dep.Id))
    {
        return true;
    }

    // See if a runtime library with the same ID as the package is available in the host's runtime libraries.
    var runtimeLib = hostDependencies.RuntimeLibraries.FirstOrDefault(r => r.Name == dep.Id);

    if (runtimeLib is object)
    {
        // What version of the library is the host using?
        var parsedLibVersion = NuGetVersion.Parse(runtimeLib.Version);

        if (parsedLibVersion.IsPrerelease)
        {
            // Always use pre-release versions from the host, otherwise it becomes
            // a nightmare to develop across multiple active versions.
            return true;
        }
        else
        {
            // Does the host version satisfy the version range of the requested package?
            // If so, we can provide it; otherwise, we cannot.
            return dep.VersionRange.Satisfies(parsedLibVersion);
        }
    }

    return false;
}

Now, when we run our code, we get precisely the set of packages we want!

AutoStep.Web : 1.0.0-develop.20
Selenium.Chrome.WebDriver : 79.0.0
Selenium.WebDriver : 3.141.0
Newtonsoft.Json : 10.0.3

Downloading and Extracting

At the moment, our list of dependencies ‘might’ contain duplicates. For example,
two different extensions might reference two different versions of NewtonSoft.Json.
We need to pick one to install that will be compatible with both.

To do this, we need to use the supplied PackageResolver class to constrain the set of packages
to only the ones we want to actually download and install, in a new GetPackagesToInstall method:

private IEnumerable<SourcePackageDependencyInfo> GetPackagesToInstall(SourceRepositoryProvider sourceRepositoryProvider, 
                                                                      ILogger logger, IEnumerable<ExtensionConfiguration> extensions, 
                                                                      HashSet<SourcePackageDependencyInfo> allPackages)
{
    // Create a package resolver context.
    var resolverContext = new PackageResolverContext(
            DependencyBehavior.Lowest,
            extensions.Select(x => x.Package),
            Enumerable.Empty<string>(),
            Enumerable.Empty<PackageReference>(),
            Enumerable.Empty<PackageIdentity>(),
            allPackages,
            sourceRepositoryProvider.GetRepositories().Select(s => s.PackageSource),
            logger);

    var resolver = new PackageResolver();

    // Work out the actual set of packages to install.
    var packagesToInstall = resolver.Resolve(resolverContext, CancellationToken.None)
                                    .Select(p => allPackages.Single(x => PackageIdentityComparer.Default.Equals(x, p)));
    return packagesToInstall;
}

Once we have that list, we can pass it to another new method that actually downloads and extracts the packages for us, InstallPackages.

private async Task InstallPackages(SourceCacheContext sourceCacheContext, ILogger logger, 
                                    IEnumerable<SourcePackageDependencyInfo> packagesToInstall, string rootPackagesDirectory, 
                                    ISettings nugetSettings, CancellationToken cancellationToken)
{
    var packagePathResolver = new PackagePathResolver(rootPackagesDirectory, true);
    var packageExtractionContext = new PackageExtractionContext(
        PackageSaveMode.Defaultv3,
        XmlDocFileSaveMode.Skip,
        ClientPolicyContext.GetClientPolicy(nugetSettings, logger),
        logger);

    foreach (var package in packagesToInstall)
    {
        var downloadResource = await package.Source.GetResourceAsync<DownloadResource>(cancellationToken);

        // Download the package (might come from the shared package cache).
        var downloadResult = await downloadResource.GetDownloadResourceResultAsync(
            package,
            new PackageDownloadContext(sourceCacheContext),
            SettingsUtility.GetGlobalPackagesFolder(nugetSettings),
            logger,
            cancellationToken);

        // Extract the package into the target directory.
        await PackageExtractor.ExtractPackageAsync(
            downloadResult.PackageSource,
            downloadResult.PackageStream,
            packagePathResolver,
            packageExtractionContext,
            cancellationToken);
    }
}

Let’s go ahead and plug those extra methods into our main calling method:

public async Task LoadExtensions()
{
    // Define a source provider, with nuget, plus my own feed.
    var sourceProvider = new PackageSourceProvider(NullSettings.Instance, new[]
    {
        new PackageSource("https://api.nuget.org/v3/index.json"),
        new PackageSource("https://f.feedz.io/autostep/ci/nuget/index.json")
    });

    // Establish the source repository provider; the available providers come from our custom settings.
    var sourceRepositoryProvider = new SourceRepositoryProvider(sourceProvider, Repository.Provider.GetCoreV3());

    // Get the list of repositories.
    var repositories = sourceRepositoryProvider.GetRepositories();

    // Disposable source cache.
    using var sourceCacheContext = new SourceCacheContext();

    // You should use an actual logger here, this is a NuGet ILogger instance.
    var logger = new NullLogger();

    // My extension configuration:
    var extensions = new[]
    {
        new ExtensionConfiguration
        {
            Package = "AutoStep.Web",
            PreRelease = true // Allow pre-release versions.
        }
    };

    // Replace this with a proper cancellation token.
    var cancellationToken = CancellationToken.None;

    // The framework we're using.
    var targetFramework = NuGetFramework.ParseFolder("netcoreapp3.1");
    var allPackages = new HashSet<SourcePackageDependencyInfo>();

    var dependencyContext = DependencyContext.Default;

    foreach (var ext in extensions)
    {
        var packageIdentity = await GetPackageIdentity(ext, sourceCacheContext, logger, repositories, cancellationToken);

        if (packageIdentity is null)
        {
            throw new InvalidOperationException($"Cannot find package {ext.Package}.");
        }

        await GetPackageDependencies(packageIdentity, sourceCacheContext, targetFramework, logger, repositories, dependencyContext, allPackages, cancellationToken);
    }

    var packagesToInstall = GetPackagesToInstall(sourceRepositoryProvider, logger, extensions, allPackages);

    // Where do we want to install our packages?
    // For now we'll pop them in the .extensions folder.
    var packageDirectory = Path.Combine(Environment.CurrentDirectory, ".extensions");
    var nugetSettings = Settings.LoadDefaultSettings(packageDirectory);

    await InstallPackages(sourceCacheContext, logger, packagesToInstall, packageDirectory, nugetSettings, cancellationToken);
}

With all these changes, here’s what the ./extensions folder looks like when we run this:

> ls ./extensions
AutoStep.Web.1.0.0-develop.20
Newtonsoft.Json.10.0.3
Selenium.Chrome.WebDriver.79.0.0
Selenium.WebDriver.3.141.0

All the packages we need are now on disk!


Wrapping Up

At the end of this post, we now have a mechanism for loading packages and a filtered set of dependencies from NuGet.

In the next post, we will load those packages into a custom AssemblyLoadContext and use them in our application.

You can find the complete set of code from this post in this gist.

The ‘production’ code this fed into is in the GitHub repository for AutoStep.Extensions, which provides the NuGet package used to load extensions into our VS Code Language Server and our commmand-line runner.