Improving Azure Key Vault Performance in ASP.NET Core By Up To 10x

tldr;

If you’re using the Azure.Extensions.AspNetCore.Configuration.Secrets package to plugin Azure Key Vault into IConfiguration, this integration uses a number of different authenticaton options to authenticate to Azure Key Vault. By excluding the ones you aren’t using, you can drastically improve your startup time from ~15 seconds to ~1.5 seconds in my experience, depending on the ones you exclude.

I have listed all the options below, but commented them out, because which ones you exclude will depend on your situation.

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
// Which ones you exclude will depend on your situation, listing them all here:
// ExcludeEnvironmentCredential = true,
// ExcludeInteractiveBrowserCredential = true,
// ExcludeAzurePowerShellCredential = true,
// ExcludeSharedTokenCacheCredential = true,
// ExcludeVisualStudioCodeCredential = true,
// ExcludeVisualStudioCredential = true,
// ExcludeAzureCliCredential = true,
// ExcludeManagedIdentityCredential = true,
}));
view raw Program.cs hosted with ❤ by GitHub

What is Azure Key Vault?

Azure Key Vault is a secret management store hosted in Azure for storing sensitive information such as secrets that are needed for your application to function. These secrets can be things such as database connection strings, API keys, passwords, and other sensitive information that you do not want falling into the wrong hands. Azure Key Vault solves numerous problems with secret management – including secure storage (i.e. not in source control), access control, audit logging, versioning, and more.

Azure Key Vault can store things like secrets, certificates, and keys. In this blog post, we’re going to focus on secrets.

Azure Key Vault + ASP.NET Core = ♥

Azure Key Vault offers a tight integration with ASP.NET Core by way of a Configuration Provider that plugs into the IConfiguration system in ASP.NET Core. You can find more details here, but in a nut shell the Azure.Extensions.AspNetCore.Configuration.Secrets package offers a single method called AddAzureKeyVault that allows you to specify your key vault name and how to authenticate.

This allows the secrets to all be loaded a single time for the lifetime of your application.

Loading secrets one time has numerous benefits:

  1. Keeping costs low. At the time of this writing you pay $0.03 USD per 10,000 retrievals, so paying that cost once per secret per application start would be much less than once per secret per request for any application with reasonable traffic.
  2. Performance on a per request basis is improved. You are not round-tripping to Azure Key Vault on each request. Instead you are paying that roundtrip cost once at application start. Of course, this comes with a tradeoff that if you update your secrets, you will have to restart your application. If this limitation is hard for your application to accept, you can set a ReloadInterval property to reload secrets on a specified TimeSpan.

Below I’m connecting to a Azure Key Vault using the new DefaultAzureIdentity which is a managed identity that works for Azure resources (such as Azure App Service) as well as local development, provided you’re logged into the Azure CLI via az login.

var builder = WebApplication.CreateBuilder(args);
// other configuration omitted
builder.Configuration.AddAzureKeyVault(
new Uri("https://yourazurekeyvaultnamegoeshere.vault.azure.net/"),
new DefaultAzureCredential());
// other configuration omitted
view raw Program.cs hosted with ❤ by GitHub

This will result in all of your secrets being available via IConfiguration or IOptions in your ASP.NET Core application. The name of the secret in Azure Key Vault will match the name of the secret exposed in IConfiguration. If you have nested values (i.e. Parent:Child), those will be represented in Azure Key Vault with double dashes such as Parent--Child.

The Problem – by default it can take 15 seconds or longer

The problem is, this can take ~15 seconds or longer in my experience (keep in mind, your performance may vary depending on a number of factors – including the number of secrets in your Azure Key Vault, your geographic location relative to the Azure Key Vault, etc.). Not only does this affect your application’s startup, but it can also slow down your tests if you’re using Azure Key Vault in combination with things like WebApplicationFactory.

The Solution – Exclude Authentication Credential You’re Not Using

The DefaultAzureCredential will try many different authentication credentials to try and work out which one to use. You can view the source code here, but each one has different impacts on the performance of your application.

The order of authentication credentials it attempts is as follows:

  1. Environment variables
  2. Managed Identity
  3. Shared Token
  4. Visual Studio
  5. Visual Studio Code
  6. Azure CLI
  7. Azure Powershell
  8. Interactive Browser

You will have to figure out which make sense to exclude for your scenarios. For my scenario, I authenticate locally with the CLI via az login, but when deployed to an Azure App Service, I leverage the Managed Identity (which is essentially the user your application is running as). That means I can ignore the rest as implemented in the code below:

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
ExcludeEnvironmentCredential = true,
ExcludeInteractiveBrowserCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeSharedTokenCacheCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeVisualStudioCredential = true,
// The following two I'm explicitly setting to false but they could be omitted because false is the default
ExcludeAzureCliCredential = false,
ExcludeManagedIdentityCredential = false,
}));
view raw Program.cs hosted with ❤ by GitHub

Before making this change, most of the time I was getting ~15 seconds for an Azure Key Vault with 10 secrets. After making this change it dropped it to ~6 seconds.

Optimizing even further by detecting the ASP.NET Core Environment

However, I really only use the Azure CLI locally and the Managed Identity out in Azure App Service. On most applications, I usually create a custom environment called “Local” for my local development, which means I can use that to tell Azure KeyVault to use the CLI locally and Managed Identity anywhere else:

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
ExcludeEnvironmentCredential = true,
ExcludeInteractiveBrowserCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeSharedTokenCacheCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeVisualStudioCredential = true,
ExcludeAzureCliCredential = !builder.Environment.IsEnvironment("Local"),
ExcludeManagedIdentityCredential = builder.Environment.IsEnvironment("Local"),
}));
view raw Program.cs hosted with ❤ by GitHub

This drops the performance of my local development to just 1.5 seconds. This is a savings of ~13.5 seconds of what I was consistently seeing.

Don’t Forget To Benchmark Your Scenario

Again, your mileage may vary depending on which options you exclude, how many secrets you have, your geographic location relative to the Azure Key Vault, and lots of other factors.

As with anything related to performance, make sure to benchmark before and after to ensure that you’re gaining performance, because your exact scenario may behave differently than my scenarios.

Why not user dotnet user-secrets for local development?

I hear this question a lot – why not use dotnet user-secrets for local development? I think that dotnet user-secrets is not very team friendly in an active codebase. Here’s why:

  1. Any time anyone adds a new secret or changes a value for an existing one, they have to let the whole team know the key and value to add to the local user secret store
  2. Any time we onboard someone, we have to usually copy the user secrets from one machine to another

Using an Azure Key Vault specific for local development, both problems are solved. Anytime someone adds a new secret or changes a value to an existing one to Azure Key Vault, the whole team gets it. Also onboarding is as simple as adding someone to an Azure Active Directory Group that has access to that Azure Key Vault and you’re done.

Hope this helps!

4 thoughts on “Improving Azure Key Vault Performance in ASP.NET Core By Up To 10x

  1. I don’t want to be that guy, but if you aren’t using the chained fallback functionality of the default Azure identity, just use the explicit implementations you need.

    TokenCredential credentials = ManagedIdentityCredential();
    If(builder.Environment.IsEnvironment(“Local”)) {
    credentials = new AzureCliCredential();
    }

    Cuts out tons of code, and is exactly what default Azure credential is doing when you limit it to one source. It’s just a chained token credential with defaults on fallback under the covers.
    https://github.com/Azure/azure-sdk-for-net/blob/e542d6b5208be52dc27005ccfa030dc8f247b755/sdk/identity/Azure.Identity/src/DefaultAzureCredential.cs#L169

    See also the roll your own chain ChainedTokenCredential which you can use to chain any number of token credentials together to sit your needs.
    https://github.com/Azure/azure-sdk-for-net/blob/e542d6b5208be52dc27005ccfa030dc8f247b755/sdk/identity/Azure.Identity/src/ChainedTokenCredential.cs

    • Definitely for this example your solution makes sense and you’re not “that guy” at all! Part of the point of this post is to illuminate the performance characteristics of the DefaultAzureCredential that is getting pushed so hard across docs, tutorials, etc.

      Appreciate the post Tim.

Leave a Reply