Improving Azure Key Vault Performance in ASP.NET Core By Up To 10x

tldr;

If you’re using the Azure.Extensions.AspNetCore.Configuration.Secrets package to plugin Azure Key Vault into IConfiguration, this integration uses a number of different authenticaton options to authenticate to Azure Key Vault. By excluding the ones you aren’t using, you can drastically improve your startup time from ~15 seconds to ~1.5 seconds in my experience, depending on the ones you exclude.

I have listed all the options below, but commented them out, because which ones you exclude will depend on your situation.

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
// Which ones you exclude will depend on your situation, listing them all here:
// ExcludeEnvironmentCredential = true,
// ExcludeInteractiveBrowserCredential = true,
// ExcludeAzurePowerShellCredential = true,
// ExcludeSharedTokenCacheCredential = true,
// ExcludeVisualStudioCodeCredential = true,
// ExcludeVisualStudioCredential = true,
// ExcludeAzureCliCredential = true,
// ExcludeManagedIdentityCredential = true,
}));
view raw Program.cs hosted with ❤ by GitHub

What is Azure Key Vault?

Azure Key Vault is a secret management store hosted in Azure for storing sensitive information such as secrets that are needed for your application to function. These secrets can be things such as database connection strings, API keys, passwords, and other sensitive information that you do not want falling into the wrong hands. Azure Key Vault solves numerous problems with secret management – including secure storage (i.e. not in source control), access control, audit logging, versioning, and more.

Azure Key Vault can store things like secrets, certificates, and keys. In this blog post, we’re going to focus on secrets.

Azure Key Vault + ASP.NET Core = ♥

Azure Key Vault offers a tight integration with ASP.NET Core by way of a Configuration Provider that plugs into the IConfiguration system in ASP.NET Core. You can find more details here, but in a nut shell the Azure.Extensions.AspNetCore.Configuration.Secrets package offers a single method called AddAzureKeyVault that allows you to specify your key vault name and how to authenticate.

This allows the secrets to all be loaded a single time for the lifetime of your application.

Loading secrets one time has numerous benefits:

  1. Keeping costs low. At the time of this writing you pay $0.03 USD per 10,000 retrievals, so paying that cost once per secret per application start would be much less than once per secret per request for any application with reasonable traffic.
  2. Performance on a per request basis is improved. You are not round-tripping to Azure Key Vault on each request. Instead you are paying that roundtrip cost once at application start. Of course, this comes with a tradeoff that if you update your secrets, you will have to restart your application. If this limitation is hard for your application to accept, you can set a ReloadInterval property to reload secrets on a specified TimeSpan.

Below I’m connecting to a Azure Key Vault using the new DefaultAzureIdentity which is a managed identity that works for Azure resources (such as Azure App Service) as well as local development, provided you’re logged into the Azure CLI via az login.

var builder = WebApplication.CreateBuilder(args);
// other configuration omitted
builder.Configuration.AddAzureKeyVault(
new Uri("https://yourazurekeyvaultnamegoeshere.vault.azure.net/"),
new DefaultAzureCredential());
// other configuration omitted
view raw Program.cs hosted with ❤ by GitHub

This will result in all of your secrets being available via IConfiguration or IOptions in your ASP.NET Core application. The name of the secret in Azure Key Vault will match the name of the secret exposed in IConfiguration. If you have nested values (i.e. Parent:Child), those will be represented in Azure Key Vault with double dashes such as Parent--Child.

The Problem – by default it can take 15 seconds or longer

The problem is, this can take ~15 seconds or longer in my experience (keep in mind, your performance may vary depending on a number of factors – including the number of secrets in your Azure Key Vault, your geographic location relative to the Azure Key Vault, etc.). Not only does this affect your application’s startup, but it can also slow down your tests if you’re using Azure Key Vault in combination with things like WebApplicationFactory.

The Solution – Exclude Authentication Credential You’re Not Using

The DefaultAzureCredential will try many different authentication credentials to try and work out which one to use. You can view the source code here, but each one has different impacts on the performance of your application.

The order of authentication credentials it attempts is as follows:

  1. Environment variables
  2. Managed Identity
  3. Shared Token
  4. Visual Studio
  5. Visual Studio Code
  6. Azure CLI
  7. Azure Powershell
  8. Interactive Browser

You will have to figure out which make sense to exclude for your scenarios. For my scenario, I authenticate locally with the CLI via az login, but when deployed to an Azure App Service, I leverage the Managed Identity (which is essentially the user your application is running as). That means I can ignore the rest as implemented in the code below:

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
ExcludeEnvironmentCredential = true,
ExcludeInteractiveBrowserCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeSharedTokenCacheCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeVisualStudioCredential = true,
// The following two I'm explicitly setting to false but they could be omitted because false is the default
ExcludeAzureCliCredential = false,
ExcludeManagedIdentityCredential = false,
}));
view raw Program.cs hosted with ❤ by GitHub

Before making this change, most of the time I was getting ~15 seconds for an Azure Key Vault with 10 secrets. After making this change it dropped it to ~6 seconds.

Optimizing even further by detecting the ASP.NET Core Environment

However, I really only use the Azure CLI locally and the Managed Identity out in Azure App Service. On most applications, I usually create a custom environment called “Local” for my local development, which means I can use that to tell Azure KeyVault to use the CLI locally and Managed Identity anywhere else:

builder.Configuration.AddAzureKeyVault(
new Uri($"https://azurekeyvaultperformance-{builder.Environment.EnvironmentName}.vault.azure.net/"),
new DefaultAzureCredential(new DefaultAzureCredentialOptions
{
ExcludeEnvironmentCredential = true,
ExcludeInteractiveBrowserCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeSharedTokenCacheCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeVisualStudioCredential = true,
ExcludeAzureCliCredential = !builder.Environment.IsEnvironment("Local"),
ExcludeManagedIdentityCredential = builder.Environment.IsEnvironment("Local"),
}));
view raw Program.cs hosted with ❤ by GitHub

This drops the performance of my local development to just 1.5 seconds. This is a savings of ~13.5 seconds of what I was consistently seeing.

Don’t Forget To Benchmark Your Scenario

Again, your mileage may vary depending on which options you exclude, how many secrets you have, your geographic location relative to the Azure Key Vault, and lots of other factors.

As with anything related to performance, make sure to benchmark before and after to ensure that you’re gaining performance, because your exact scenario may behave differently than my scenarios.

Why not user dotnet user-secrets for local development?

I hear this question a lot – why not use dotnet user-secrets for local development? I think that dotnet user-secrets is not very team friendly in an active codebase. Here’s why:

  1. Any time anyone adds a new secret or changes a value for an existing one, they have to let the whole team know the key and value to add to the local user secret store
  2. Any time we onboard someone, we have to usually copy the user secrets from one machine to another

Using an Azure Key Vault specific for local development, both problems are solved. Anytime someone adds a new secret or changes a value to an existing one to Azure Key Vault, the whole team gets it. Also onboarding is as simple as adding someone to an Azure Active Directory Group that has access to that Azure Key Vault and you’re done.

Hope this helps!

Using Husky Git Hooks and Lint-Staged With Nested Folders

What is husky?

Husky is a JavaScript package that allows you to run some code during various parts of your git workflow. Husky leverages git hooks to allow you to hook into various git events such as pre-commit and pre-push.

 

What is lint-staged?

Lint-staged allows you to run code against your staged git files to automate the tedious part of your workflows, such as formatting with Prettier and/or linting with ESLint.

 

Using husky with lint-staged

You can use husky to trigger lint-staged during the pre-commit hook, so that your coding standards are enforced right as you commit. The major benefit of this is it enforces your coding standards without someone needing to configure and install certain extensions in their IDE to enforce them on save or remembering to do anything. Your code gets fixed before it ever leaves your machine, so you don’t have to wait for your CI to inform you that you forgot to run the formatter or linter.

 

The Problem: husky expects your package.json at the root

The problem is, husky expects your package.json to be at the root of your project. That’s a fine assumption to make a lot of times, but sometimes we might be in more of a monorepo situation, or where a single repo contains both the Server and Client in separate folders.

 

The Solution

I’m going to use the example from the default React and ASP.NET Core template when you run dotnet new react. The default folder structure has the React code nested under a /ClientApp folder with the corresponding package.json. The folder structure looks like this:

Let’s dive into the steps. You can view the completed repo here.

  1. cd into the directory with your package.json 
    • In my case: ./ClientApp

  1. Install husky and lint-staged:
    • npm i husky lint-staged -D

  1. Add a “prepare” npm script to your package.json with the following contents (note: the initial cd goes to the repo root and the husky install goes from the repo root down to your directory with the package.json):
{
  "scripts": {
      // other scripts omitted 
      "prepare": "cd ../ && husky install ./ClientApp/.husky"
   }
}
  1. Run npm install

  1. FYI – a .husky folder will appear in the same path as your package.json, in my case under ./ClientApp

  1. Create a pre-commit file with no file extension under the .husky folder with the following contents (note: the ./ClientApp is the path to your package.json relative to the root of your repository):
#!/bin/sh
. "$(dirname "$0")/_/husky.sh"
cd ./ClientApp && npx lint-staged 
  1. Make sure the pre-commit file is executable via chmod:
chmod +x ./.husky/pre-commit
  1. Add a .lintstagedrc file under your ./ClientApp folder (or wherever your package.json lives) with the following contents:
{
  "*.{js,ts,tsx,scss,css,md}": ["eslint", "prettier --write"]
} 

As you can probably guess, the above says look for any changes to any of the file extensions defined on the left, and run the commands in the array on the right if you find any changes to those files.

And that’s it! Now if you try to make a commit, you will see that eslint and prettier will run and fix themselves as you would expect. See an example of it in action below: