I have occasionally found that I need to use keys, passwords, or other sensitive strings as part of my ADF Pipelines. While it is completely possible to store this data in parameters against the Pipeline, it is not the best option for security of the data. Also, if you need to use the same keys or passwords in multiple places it can be a pain to find and update all references across your Pipelines when you need to roll the key/password.
Instead, I prefer to store the keys, passwords or other sensitive strings in an Azure Key Vault and retrieve the latest values when the Pipeline is run. This also makes it quite easy to have parameters on the Pipeline to get different credentials, i.e. I could have an “InDebug” parameter and an If activity to get different credentials base on how the parameter is set.
I use this method in conjunction with a REST API call using a Web Activity when the inbuilt Linked servers/Datasets don’t meet my needs.
Table of Contents
- Adding your secret to an Azure Key Vault
- Required permissions
- Required information
- Getting Key Vault data from an ADF Pipeline
- You may also like
Adding your secret to an Azure Key Vault
I won’t cover how to create an Azure Key Vault or how to populate it with your secrets in this guide. It is widely covered elsewhere and if you are reading this post I am assuming you already have this covered. If not, theses links may help
https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal
https://docs.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal
Required permissions
Before your Azure Data Factory will be able to list and get secrets from your Azure Key Vault you will need to set the permissions correctly, else you will get a message that you do not have secrets list permission.
Look at my other post on the subject
Required information
Before continuing you will need a few bits of information in relation to your Key Vault and your secret.
- The name of your Key Vault
- The name of your secret
- Optionally the version of the key you want to get if to the latest version
The easiest method to get this information is to view your secret in the Key Vault and copy the “Secret Identifier”
This will be in the format {vaultBaseUrl}/secrets/{secret-name}/{secret-version}
Getting Key Vault data from an ADF Pipeline
To get the Key Vault we will be using the Azure Key Vault REST API authenticated using the managed identity of the Data Factory. This will require us to have a Web Activity to call and get data from the Azure Key Vault REST API and a Set Variable activity to store the returned data. The Set Variable activity is optional, you could use the returned data directly in other activities. But storing in in a variable it cleaner and easier to work with.
- Create a new Pipeline or modify an existing.
- Add a new variable to your Pipeline to hold the returned KeyVault data, in this example I will use “client_secret”
- Add a “Web Activity” and “Set variable” activity to your Pipeline, name them as required and link them as below.
Web Activity “Get KeyVault Secret”
Settings => URL
This will need to be in the format {vaultBaseUrl}/secrets/{secret-name}/{secret-version}?api-version=7.3. Where /{secret-version} is optional, if omitted the latest version of the certificate is returned.
For example, if my Key Vault Name = MyKeyVault and my Secrets Name = MySecret and I only need the latest version, my URL would be
https://MyKeyVault.vault.azure.net/secrets/MySecret?api-version=7.3
At the time or writing the latest API version is 7.3, look here for the latest information https://docs.microsoft.com/en-us/rest/api/keyvault/secrets/get-secret/get-secret
Settings => Method
Will be GET
Settings => Authentication
Will be System Assigned Managed Identity
Settings => Resource
Will be https://vault.azure.net
Set Variable Activity “Store Secret”
Variables => Name
Select the variable you what to store the secret in
Variables => Value
Add the below dynamic content where “Get KeyVault Secret” is the name of you Web Activity calling the Key Vault API
@activity(‘Get KeyVault Secret’).output.value
