My current solution is to use an environment variable with the access token.

Get the key and store in environment variable (after you did an az login and set the correct subscription):

$Env:ACCESS_TOKEN=(az account get-access-token  --resource=https://vault.azure.net | ConvertFrom-Json).accessToken

The we add that environment variable in Visual Studio:

Change the code to:

                config.AddEnvironmentVariables();                KeyVaultClient keyVaultClient;                var accessToken = Environment.GetEnvironmentVariable("ACCESS_TOKEN");                if (accessToken != null)                {                    keyVaultClient = new KeyVaultClient(                        async (string a, string r, string s) => accessToken);                }                else                {                    var azureServiceTokenProvider = new AzureServiceTokenProvider();                    keyVaultClient = new KeyVaultClient(                       new KeyVaultClient.AuthenticationCallback(                           azureServiceTokenProvider.KeyVaultTokenCallback));                }                config.AddAzureKeyVault(                    $"https://{builtConfig["KeyVaultName"]}.vault.azure.net/",                    keyVaultClient,                    new DefaultKeyVaultSecretManager());

Solution (not for production use)

A possible Solution to your Problem is to generate a Service Principal (SP) and grant this Service Principal access to the key vault (via RBAC or IAM). Microsoft Documentation on creating a SP

Using the credentials of the SP as client-id and client-secret (Random example) you can then log into the vault and retrieve the secrets.

Concerns

• with this approach, you will introduce secrets into the code (propably the exact reason why you use the key vault). I suppose the local docker image is for development use only. Therefore I would recommend creating a Keyvault just for development (and access it using SP) while using a separate Kevault for Production where one of the established, secret-less authentication schemes is used.
• You must make sure that the key vault allows access from outside the azure cloud (see the access policies on portal.azure.com)

Although there's some time since you make this question, another option, suitable for production environments, would be using an x509 certificate.

Microsoft has this article explaining how to do this. You can use self-signed certificates or any other valid SSL certificate. That depends on your needs.