Azure Data Factory – Get and store a Key Vault secret as a variable

I have occasionally found that I need to use keys, passwords, or other sensitive strings as part of my ADF Pipelines. While it is completely possible to store this data in parameters against the Pipeline, it is not the best option for security of the data. Also, if you need to use the same keys or passwords in multiple places it can be a pain to find and update all references across your Pipelines when you need to roll the key/password.

Instead, I prefer to store the keys, passwords or other sensitive strings in an Azure Key Vault and retrieve the latest values when the Pipeline is run. This also makes it quite easy to have parameters on the Pipeline to get different credentials, i.e. I could have an “InDebug” parameter and an If activity to get different credentials base on how the parameter is set.

I use this method in conjunction with a REST API call using a Web Activity when the inbuilt Linked servers/Datasets don’t meet my needs.

Table of Contents

Adding your secret to an Azure Key Vault

I won’t cover how to create an Azure Key Vault or how to populate it with your secrets in this guide. It is widely covered elsewhere and if you are reading this post I am assuming you already have this covered. If not, theses links may help

https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal

https://docs.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal

Required permissions

Before your Azure Data Factory will be able to list and get secrets from your Azure Key Vault you will need to set the permissions correctly, else you will get a message that you do not have secrets list permission.

Look at my other post on the subject

Required information

Before continuing you will need a few bits of information in relation to your Key Vault and your secret.

  • The name of your Key Vault
  • The name of your secret
  • Optionally the version of the key you want to get if to the latest version

The easiest method to get this information is to view your secret in the Key Vault and copy the “Secret Identifier”

This will be in the format {vaultBaseUrl}/secrets/{secret-name}/{secret-version}

Getting Key Vault data from an ADF Pipeline

To get the Key Vault we will be using the Azure Key Vault REST API authenticated using the managed identity of the Data Factory. This will require us to have a Web Activity to call and get data from the Azure Key Vault REST API and a Set Variable activity to store the returned data. The Set Variable activity is optional, you could use the returned data directly in other activities. But storing in in a variable it cleaner and easier to work with.

  • Create a new Pipeline or modify an existing.
  • Add a new variable to your Pipeline to hold the returned KeyVault data, in this example I will use “client_secret”
  • Add a “Web Activity” and “Set variable” activity to your Pipeline, name them as required and link them as below.

Web Activity “Get KeyVault Secret”

Settings => URL

This will need to be in the format {vaultBaseUrl}/secrets/{secret-name}/{secret-version}?api-version=7.3. Where /{secret-version} is optional, if omitted the latest version of the certificate is returned.

For example, if my Key Vault Name = MyKeyVault and my Secrets Name = MySecret and I only need the latest version, my URL would be

https://MyKeyVault.vault.azure.net/secrets/MySecret?api-version=7.3

At the time or writing the latest API version is 7.3, look here for the latest information https://docs.microsoft.com/en-us/rest/api/keyvault/secrets/get-secret/get-secret

Settings => Method

Will be GET

Settings => Authentication

Will be System Assigned Managed Identity

Settings => Resource

Will be https://vault.azure.net

Set Variable Activity “Store Secret”

Variables => Name

Select the variable you what to store the secret in

Variables => Value

Add the below dynamic content where “Get KeyVault Secret” is the name of you Web Activity calling the Key Vault API

@activity(‘Get KeyVault Secret’).output.value

You may also like

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.