Categories
Uncategorized

Loading Plugins/Extensions at Run Time from NuGet in .NET Core : Part 1 – NuGet

This post is the first in a short series, writing up my efforts creating an plugin/extension system working in .NET Core that:

  • Loads extension packages from NuGet, with all their dependencies (this post).
  • Loads the extensions into my .NET Core Process.
  • Allows the loaded extensions to be unloaded.

Background

As a bit of context, I’m currently building an open-source BDD testing platform that goes beyond Gherkin, AutoStep, which is built entirely in C#, on top of .NET Core 3.1.

In AutoStep, I need to be able to load in extensions that provide additional functionality for AutoStep. For example, extensions might provide:

  • Bindings for some UI platform or library
  • Custom Report Formats
  • Integration with some external Test Management System

In terms of what’s in them, AutoStep extensions are going to consist of things like:

  • .NET DLLs
  • AutoStep Test Files
  • Dependencies on various NuGet packages (Selenium.WebDriver anyone?).

All of the above items fit pretty well within the scope of NuGet packages, and I don’t want to build my own extension packaging, hosting, versioning and so on, so I’m going to say that each extension can be represented by a NuGet package.

AutoStep does not require the .NET Core SDK to build or run any tests, so I can’t just create a csproj, chuck PackageReferences in and be done with it.

I need to bake the idea of extensions into the platform itself.


If you want to jump ahead, you can check out the GitHub repository for AutoStep.Extensions, which provides the NuGet package used to load extensions into our VS Code Language Server and our commmand-line runner.

Loading Extensions from NuGet

Microsoft supplies the NuGet Client SDK, to work with both NuGet packages and source repositories; specifically the NuGet.Protocol and NuGet.Resolver packages.

The documentation on how to actually use the NuGet Client libraries is a bit sparse, so I’m permanently indebted to Martin Bjorkstrom for writing a blog post on it that I used as a pretty detailed guide to get me started.

Loading our extension packages from NuGet involves three phases:

  1. Determine the best version of an extension package to install, given a version range (and normal NuGet rules).
    For example, if the version of the extension requested is 1.4.0, and there is a 1.4.5 version available, we want that one.
  2. Get the list of all NuGet package dependencies (recursively) for each extension.
  3. Download and Extract your packages.

Choosing the Extension Version

This is (relatively) the easy bit. First up, we’ll create some of the context objects we need to get started:

/// <summary>
/// Represents the configuration for a single extension to install.
/// </summary>
public class ExtensionConfiguration
{
    public string Package { get; set; }
    public string Version { get; set; }
    public bool PreRelease { get; set; }
}

public async Task LoadExtensions()
{
    // Define a source provider, with the main NuGet feed, plus my own feed.
    var sourceProvider = new PackageSourceProvider(NullSettings.Instance, new[]
    {
        new PackageSource("https://api.nuget.org/v3/index.json"),
        new PackageSource("https://f.feedz.io/autostep/ci/nuget/index.json")
    });

    // Establish the source repository provider; the available providers come from our custom settings.
    var sourceRepositoryProvider = new SourceRepositoryProvider(sourceProvider, Repository.Provider.GetCoreV3());

    // Get the list of repositories.
    var repositories = sourceRepositoryProvider.GetRepositories();

    // Disposable source cache.
    using var sourceCacheContext = new SourceCacheContext();

    // You should use an actual logger here, this is a NuGet ILogger instance.
    var logger = new NullLogger();

    // My extension configuration:
    var extensions = new[]
    { 
        new ExtensionConfiguration
        {
            Package = "AutoStep.Web",
            PreRelease = true // Allow pre-release versions.
        }
    };
}

Next, let’s write a method to actually get the desired package identity to install. The GetPackageIdentity method goes through each repository, and either:

  • Picks the latest available version if no version range has been configured or,
  • If a version range has been specified, uses the provided NuGet VersionRange class to find the best match given the set of all versions.
private async Task<PackageIdentity> GetPackageIdentity(
          ExtensionConfiguration extConfig, SourceCacheContext cache, ILogger nugetLogger,
          IEnumerable<SourceRepository> repositories, CancellationToken cancelToken)
{
    // Go through each repository.
    // If a repository contains only pre-release packages (e.g. AutoStep CI), and 
    // the configuration doesn't permit pre-release versions,
    // the search will look at other ones (e.g. NuGet).
    foreach (var sourceRepository in repositories)
    {
        // Get a 'resource' from the repository.
        var findPackageResource = await sourceRepository.GetResourceAsync<FindPackageByIdResource>();

        // Get the list of all available versions of the package in the repository.
        var allVersions = await findPackageResource.GetAllVersionsAsync(extConfig.Package, cache, nugetLogger, cancelToken);

        NuGetVersion selected;

        // Have we specified a version range?
        if (extConfig.Version != null)
        {
            if (!VersionRange.TryParse(extConfig.Version, out var range))
            {
                throw new InvalidOperationException("Invalid version range provided.");
            }

            // Find the best package version match for the range.
            // Consider pre-release versions, but only if the extension is configured to use them.
            var bestVersion = range.FindBestMatch(allVersions.Where(v => extConfig.PreRelease || !v.IsPrerelease));

            selected = bestVersion;
        }
        else
        {
            // No version; choose the latest, allow pre-release if configured.
            selected = allVersions.LastOrDefault(v => v.IsPrerelease == extConfig.PreRelease);
        }

        if (selected is object)
        {
            return new PackageIdentity(extConfig.Package, selected);
        }
    }

    return null;
}

Let’s plug that code into our previous code, so we’re now getting the identity:

// ...

// My extension configuration:
var extensions = new[] { new ExtensionConfiguration
{
    Package = "AutoStep.Web",
    PreRelease = true // Allow pre-release versions.
}};

foreach (var ext in extensions)
{
    var packageIdentity = await GetPackageIdentity(ext, sourceCacheContext, logger, repositories, CancellationToken.None);

    if (packageIdentity is null)
    {
        throw new InvalidOperationException($"Cannot find package {ext.Package}.");
    }
}

With this we get a package identity of AutoStep.Web.1.0.0-develop.20 (the latest pre-release version at the time).

Get the List of Package Dependencies

This is where things get interesting. We need to get the complete set of all dependencies, across all the extensions, that we need to install in order to use the extension package.

First off, let’s look at an initial, very naive solution, which just does a straight-forward recurse through the entire dependency graph.

private async Task GetPackageDependencies(PackageIdentity package, SourceCacheContext cacheContext, 
                                          NuGetFramework framework, ILogger logger, 
                                          IEnumerable<SourceRepository> repositories,
                                          ISet<SourcePackageDependencyInfo> availablePackages, 
                                          CancellationToken cancelToken)
{
    // Don't recurse over a package we've already seen.
    if (availablePackages.Contains(package))
    {
        return;
    }

    foreach (var sourceRepository in repositories)
    {
        // Get the dependency info for the package.
        var dependencyInfoResource = await sourceRepository.GetResourceAsync<DependencyInfoResource>();
        var dependencyInfo = await dependencyInfoResource.ResolvePackage(
            package,
            framework,
            cacheContext,
            logger,
            cancelToken);

        // No info for the package in this repository.
        if (dependencyInfo == null)
        {
            continue;
        }

        // Add to the list of all packages.
        availablePackages.Add(dependencyInfo);

        // Recurse through each package.
        foreach (var dependency in dependencyInfo.Dependencies)
        {
            await GetPackageDependencies(
                new PackageIdentity(dependency.Id, dependency.VersionRange.MinVersion),
                cacheContext,
                framework,
                logger,
                repositories,
                availablePackages,
                cancelToken);
        }

        break;
    }
}

That does indeed create the complete graph of all libraries required by that extension, the problem is that it has 104 packages in it!

Long package list

I’ve got the AutoStep.Web package at the top there, but I’ve also got
System.Runtime, which I definitely don’t want.

All the extensions are going to reference the AutoStep.Extensions.Abstractions package (because that’s where we define our interfaces for extensions), but we don’t want to download it ourselves!

Besides the fact that we don’t need to download these shared packages, if we load in the AutoStep.Extensions.Abstractions assembly from the extension’s dependencies, it will not be compatible with the version referenced by the host process.

The actual requirement for our behaviour here is:

All packages provided by the host process should be excluded from the set of dependencies to install.

Filtering the Dependencies

At runtime, how do we know what the set of installed packages are for a .NET Core Application? Luckily, there happens to be an existing file containing this information, the {AssemblyName}.deps.json file that gets copied to your output directory.

You probably haven’t had to worry about it much, but if you look in your application’s output directory, you’ll find it.

It contains the complete package reference graph for your application, and looks a little something like this:

{
  "runtimeTarget": {
    "name": ".NETCoreApp,Version=v3.1",
    "signature": ""
  },
  "compilationOptions": {},
  "targets": {
    ".NETCoreApp,Version=v3.1": {
      "NugetConsole/1.0.0": {
        "dependencies": {
          "Microsoft.Extensions.DependencyModel": "3.1.3",
          "NuGet.Protocol": "5.5.1",
          "NuGet.Resolver": "5.5.1"
        },
        "runtime": {
          "NugetConsole.dll": {}
        }
      },
      "Microsoft.CSharp/4.0.1": {
        "dependencies": {
          "System.Collections": "4.3.0",
          "System.Diagnostics.Debug": "4.3.0",
          "System.Dynamic.Runtime": "4.3.0",
          "System.Globalization": "4.3.0",
          "System.Linq": "4.3.0",
          "System.Linq.Expressions": "4.3.0",
          "System.ObjectModel": "4.3.0",

// ...a lot more content...

Handily, we don’t have to parse this ourselves. If you add the Microsoft.Extensions.DependencyModel package to your project, you can directly access this content using DependencyContext.Default, which gives you a DependencyContext you can interrogate.

Let’s define a method that takes this DependencyContext and a PackageDependency, and checks whether it is provided by the host:

private bool DependencySuppliedByHost(DependencyContext hostDependencies, PackageDependency dep)
{
    // See if a runtime library with the same ID as the package is available in the host's runtime libraries.
    var runtimeLib = hostDependencies.RuntimeLibraries.FirstOrDefault(r => r.Name == dep.Id);

    if (runtimeLib is object)
    {
        // What version of the library is the host using?
        var parsedLibVersion = NuGetVersion.Parse(runtimeLib.Version);

        if (parsedLibVersion.IsPrerelease)
        {
            // Always use pre-release versions from the host, otherwise it becomes
            // a nightmare to develop across multiple active versions.
            return true;
        }
        else
        {
            // Does the host version satisfy the version range of the requested package?
            // If so, we can provide it; otherwise, we cannot.
            return dep.VersionRange.Satisfies(parsedLibVersion);
        }
    }

    return false;
}

Then, let’s plug that in to our existing GetPackageDependencies method:

private async Task GetPackageDependencies(PackageIdentity package, SourceCacheContext cacheContext, NuGetFramework framework, 
                                          ILogger logger, IEnumerable<SourceRepository> repositories, DependencyContext hostDependencies,
                                          ISet<SourcePackageDependencyInfo> availablePackages, CancellationToken cancelToken)
{
    // Don't recurse over a package we've already seen.
    if (availablePackages.Contains(package))
    {
        return;
    }

    foreach (var sourceRepository in repositories)
    {
        // Get the dependency info for the package.
        var dependencyInfoResource = await sourceRepository.GetResourceAsync<DependencyInfoResource>();
        var dependencyInfo = await dependencyInfoResource.ResolvePackage(
            package,
            framework,
            cacheContext,
            logger,
            cancelToken);

        // No info for the package in this repository.
        if (dependencyInfo == null)
        {
            continue;
        }


        // Filter the dependency info.
        // Don't bring in any dependencies that are provided by the host.
        var actualSourceDep = new SourcePackageDependencyInfo(
            dependencyInfo.Id,
            dependencyInfo.Version,
            dependencyInfo.Dependencies.Where(dep => !DependencySuppliedByHost(hostDependencies, dep)),
            dependencyInfo.Listed,
            dependencyInfo.Source);

        availablePackages.Add(actualSourceDep);

        // Recurse through each package.
        foreach (var dependency in actualSourceDep.Dependencies)
        {
            await GetPackageDependencies(
                new PackageIdentity(dependency.Id, dependency.VersionRange.MinVersion),
                cacheContext,
                framework,
                logger,
                repositories,
                hostDependencies,
                availablePackages,
                cancelToken);
        }

        break;
    }
}

This cuts down on the set of packages significantly, but it’s still pulling down some runtime-provided packages I don’t want:

AutoStep.Web : 1.0.0-develop.20       // correct
Selenium.Chrome.WebDriver : 79.0.0    // correct
Selenium.WebDriver : 3.141.0          // correct
Newtonsoft.Json : 10.0.3              // correct
Microsoft.CSharp : 4.3.0              // Ah. This is a runtime package...
System.ComponentModel.TypeConverter : 4.3.0 
System.Collections.NonGeneric : 4.3.0       
System.Collections.Specialized : 4.3.0
System.ComponentModel : 4.3.0
System.ComponentModel.Primitives : 4.3.0
System.Runtime.Serialization.Primitives : 4.3.0
System.Runtime.Serialization.Formatters : 4.3.0
System.Xml.XmlDocument : 4.3.0

So, something is still not right. What’s causing these packages to be present?

Well, simply put, my program doesn’t use System.ComponentModel, so it isn’t in the list of my dependencies. But it is provided by the host, because it’s part of the distributed .NET Runtime.

Ignoring Runtime-Provided Packages

We want to filter out runtime-provided packages completely, but how do we know which ones to exclude? We can’t just filter out any System.* packages, because there are a number of System.* packages that aren’t shipped with the runtime (e.g. System.Text.Json).

As far as I can tell, it’s more or less impossible to determine the full set at run time dynamically.

After some considerable searching however, I found a complete listing of all runtime-provided packages in an MSBuild task in the dotnet SDK, called PackageConflictOverrides, which tells the build system which packages don’t need to be restored! Yay!

This allowed me to define the following static lookup class (excerpt only). You can find a full version here.

/// <summary>
/// Contains a pre-determined list of NuGet packages that are provided by the run-time, and
/// therefore should not be restored from an extensions dependency list.
/// </summary>
internal static class RuntimeProvidedPackages
{
    /// <summary>
    /// Checks whether the set of known runtime packages contains the given package ID.
    /// </summary>
    /// <param name="packageId">The package ID.</param>
    /// <returns>True if the package is provided by the framework, otherwise false.</returns>
    public static bool IsPackageProvidedByRuntime(string packageId)
    {
        return ProvidedPackages.Contains(packageId);
    }

    /// <summary>
    /// This list comes from the package overrides for the .NET SDK,
    /// at https://github.com/dotnet/sdk/blob/v3.1.201/src/Tasks/Common/targets/Microsoft.NET.DefaultPackageConflictOverrides.targets.
    /// If the executing binaries ever change to a newer version, this project must update as well, and refresh this list.
    /// </summary>
    private static readonly ISet<string> ProvidedPackages = new HashSet<string>
    {
        "Microsoft.CSharp",
        "Microsoft.Win32.Primitives",
        "Microsoft.Win32.Registry",
        "runtime.debian.8-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.fedora.23-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.opensuse.42.1-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple",
        "runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.rhel.7-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.16.04-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "runtime.ubuntu.16.10-x64.runtime.native.System.Security.Cryptography.OpenSsl",
        "System.AppContext",
        "System.Buffers",
        "System.Collections",
        "System.Collections.Concurrent",
        // Removed a load for brevity....
        "System.Xml.ReaderWriter",
        "System.Xml.XDocument",
        "System.Xml.XmlDocument",
        "System.Xml.XmlSerializer",
        "System.Xml.XPath",
        "System.Xml.XPath.XDocument",
    };
}

Ok, so let’s update our DependencySuppliedByHost method to use this look-up:

private bool DependencySuppliedByHost(DependencyContext hostDependencies, PackageDependency dep)
{
    // Check our look-up list.
    if(RuntimeProvidedPackages.IsPackageProvidedByRuntime(dep.Id))
    {
        return true;
    }

    // See if a runtime library with the same ID as the package is available in the host's runtime libraries.
    var runtimeLib = hostDependencies.RuntimeLibraries.FirstOrDefault(r => r.Name == dep.Id);

    if (runtimeLib is object)
    {
        // What version of the library is the host using?
        var parsedLibVersion = NuGetVersion.Parse(runtimeLib.Version);

        if (parsedLibVersion.IsPrerelease)
        {
            // Always use pre-release versions from the host, otherwise it becomes
            // a nightmare to develop across multiple active versions.
            return true;
        }
        else
        {
            // Does the host version satisfy the version range of the requested package?
            // If so, we can provide it; otherwise, we cannot.
            return dep.VersionRange.Satisfies(parsedLibVersion);
        }
    }

    return false;
}

Now, when we run our code, we get precisely the set of packages we want!

AutoStep.Web : 1.0.0-develop.20
Selenium.Chrome.WebDriver : 79.0.0
Selenium.WebDriver : 3.141.0
Newtonsoft.Json : 10.0.3

Downloading and Extracting

At the moment, our list of dependencies ‘might’ contain duplicates. For example,
two different extensions might reference two different versions of NewtonSoft.Json.
We need to pick one to install that will be compatible with both.

To do this, we need to use the supplied PackageResolver class to constrain the set of packages
to only the ones we want to actually download and install, in a new GetPackagesToInstall method:

private IEnumerable<SourcePackageDependencyInfo> GetPackagesToInstall(SourceRepositoryProvider sourceRepositoryProvider, 
                                                                      ILogger logger, IEnumerable<ExtensionConfiguration> extensions, 
                                                                      HashSet<SourcePackageDependencyInfo> allPackages)
{
    // Create a package resolver context.
    var resolverContext = new PackageResolverContext(
            DependencyBehavior.Lowest,
            extensions.Select(x => x.Package),
            Enumerable.Empty<string>(),
            Enumerable.Empty<PackageReference>(),
            Enumerable.Empty<PackageIdentity>(),
            allPackages,
            sourceRepositoryProvider.GetRepositories().Select(s => s.PackageSource),
            logger);

    var resolver = new PackageResolver();

    // Work out the actual set of packages to install.
    var packagesToInstall = resolver.Resolve(resolverContext, CancellationToken.None)
                                    .Select(p => allPackages.Single(x => PackageIdentityComparer.Default.Equals(x, p)));
    return packagesToInstall;
}

Once we have that list, we can pass it to another new method that actually downloads and extracts the packages for us, InstallPackages.

private async Task InstallPackages(SourceCacheContext sourceCacheContext, ILogger logger, 
                                    IEnumerable<SourcePackageDependencyInfo> packagesToInstall, string rootPackagesDirectory, 
                                    ISettings nugetSettings, CancellationToken cancellationToken)
{
    var packagePathResolver = new PackagePathResolver(rootPackagesDirectory, true);
    var packageExtractionContext = new PackageExtractionContext(
        PackageSaveMode.Defaultv3,
        XmlDocFileSaveMode.Skip,
        ClientPolicyContext.GetClientPolicy(nugetSettings, logger),
        logger);

    foreach (var package in packagesToInstall)
    {
        var downloadResource = await package.Source.GetResourceAsync<DownloadResource>(cancellationToken);

        // Download the package (might come from the shared package cache).
        var downloadResult = await downloadResource.GetDownloadResourceResultAsync(
            package,
            new PackageDownloadContext(sourceCacheContext),
            SettingsUtility.GetGlobalPackagesFolder(nugetSettings),
            logger,
            cancellationToken);

        // Extract the package into the target directory.
        await PackageExtractor.ExtractPackageAsync(
            downloadResult.PackageSource,
            downloadResult.PackageStream,
            packagePathResolver,
            packageExtractionContext,
            cancellationToken);
    }
}

Let’s go ahead and plug those extra methods into our main calling method:

public async Task LoadExtensions()
{
    // Define a source provider, with nuget, plus my own feed.
    var sourceProvider = new PackageSourceProvider(NullSettings.Instance, new[]
    {
        new PackageSource("https://api.nuget.org/v3/index.json"),
        new PackageSource("https://f.feedz.io/autostep/ci/nuget/index.json")
    });

    // Establish the source repository provider; the available providers come from our custom settings.
    var sourceRepositoryProvider = new SourceRepositoryProvider(sourceProvider, Repository.Provider.GetCoreV3());

    // Get the list of repositories.
    var repositories = sourceRepositoryProvider.GetRepositories();

    // Disposable source cache.
    using var sourceCacheContext = new SourceCacheContext();

    // You should use an actual logger here, this is a NuGet ILogger instance.
    var logger = new NullLogger();

    // My extension configuration:
    var extensions = new[]
    {
        new ExtensionConfiguration
        {
            Package = "AutoStep.Web",
            PreRelease = true // Allow pre-release versions.
        }
    };

    // Replace this with a proper cancellation token.
    var cancellationToken = CancellationToken.None;

    // The framework we're using.
    var targetFramework = NuGetFramework.ParseFolder("netcoreapp3.1");
    var allPackages = new HashSet<SourcePackageDependencyInfo>();

    var dependencyContext = DependencyContext.Default;

    foreach (var ext in extensions)
    {
        var packageIdentity = await GetPackageIdentity(ext, sourceCacheContext, logger, repositories, cancellationToken);

        if (packageIdentity is null)
        {
            throw new InvalidOperationException($"Cannot find package {ext.Package}.");
        }

        await GetPackageDependencies(packageIdentity, sourceCacheContext, targetFramework, logger, repositories, dependencyContext, allPackages, cancellationToken);
    }

    var packagesToInstall = GetPackagesToInstall(sourceRepositoryProvider, logger, extensions, allPackages);

    // Where do we want to install our packages?
    // For now we'll pop them in the .extensions folder.
    var packageDirectory = Path.Combine(Environment.CurrentDirectory, ".extensions");
    var nugetSettings = Settings.LoadDefaultSettings(packageDirectory);

    await InstallPackages(sourceCacheContext, logger, packagesToInstall, packageDirectory, nugetSettings, cancellationToken);
}

With all these changes, here’s what the ./extensions folder looks like when we run this:

> ls ./extensions
AutoStep.Web.1.0.0-develop.20
Newtonsoft.Json.10.0.3
Selenium.Chrome.WebDriver.79.0.0
Selenium.WebDriver.3.141.0

All the packages we need are now on disk!


Wrapping Up

At the end of this post, we now have a mechanism for loading packages and a filtered set of dependencies from NuGet.

In the next post, we will load those packages into a custom AssemblyLoadContext and use them in our application.

You can find the complete set of code from this post in this gist.

The ‘production’ code this fed into is in the GitHub repository for AutoStep.Extensions, which provides the NuGet package used to load extensions into our VS Code Language Server and our commmand-line runner.