Azure Data Factory – How to Save Web Activity to Blob Storage

In this post, I will cover how you can save the output from an Azure Data Factory (ADF) or Azure Synapse Pipeline Web Activity as a Blob file in an Azure Storage Account. You could use the same method to save the output from other activities.

I find I use this method every now and then, predominantly as a method of easily passing data returned from a REST API between a parent Pipeline and a Data Flow. Or between different Pipelines without the need for anything like a database. I can just save the data as a JSON blob file in a Storage Account to be picked up elsewhere.

Storage Account

First, you will need a Storage Account, if you don’t already have created that a look at this Microsoft guide https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal

I use a storage account in the same resource group as my Data Factory to make It easy to track the costs as a whole for Data Factory, but whatever works for you.

Blob Container

If you already have a container in your Storage Account you would like to use you can skip this step.

Within your chosen Storage Account create a Container to store your blob file(s).

Within you Storage Account
Click Containers under Data Storage
Click the option to add a new container
Give your container a name, make a note we will need this later
Setting the access level as private is fine.
Click Create

Shared Access Token

To allow Azure Data Factory to have write permission to your chosen container in your Storage Account you will need to create a shared access token.

Within you Storage Account
Click Containers under Data Storage
Click your container
Click Shared access tokens under Settings
Grant at least the permissions Read, Add, Create and Write.
Set an appropriate expiry date.
Click Generate SAS token and URL
Take a copy the Blob SAS URL at the bottom

Saving to Blob

In this example, I have a Web Activity that is performing a REST API call and I want to save the output from this activity to a blob.

Saving to a blob can be done with another Web activity, so you end up with a layout like the below. You have a Web activity collecting your data and a second that will pick up the output from the first and perform an HTTP PUT to blob storage

I am only going to cover how to set up the “Save Output to Blob” web activity in this example.

Here is how it is done;

Create a new Web activity, name it as required “Save Output to Blob” in my example and link it to your source activity (as above).
Open the properties for your newly created Web Activity and select the Settings tab
Set the URL to be to be your previously copied Blob SAS URL. Adding in the desired blob file name after the container name in the URL.

For example:

If your URL ware;

https://mystorageaccount.blob.core.windows.net/mycontainer?sp=racwdli&st=2021-11-28T23:08:09Z&se=2099-11-29T07:08:09Z&spr=https&sv=2020-08-04&sr=c&sig=xyz

You would amend it to be;

https://mystorageaccount.blob.core.windows.net/mycontainer/blobfilename.json?sp=racwdli&st=2021-11-28T23:08:09Z&se=2099-11-29T07:08:09Z&spr=https&sv=2020-08-04&sr=c&sig=xyz

Set the Method to be PUT
Add a header named “x-ms-blob-type” with a value of “BlockBlob“
Set the Body to by a dynamic contact with the value @activity(‘REST API Call’).output replacing “REST API Call” with the name of you source activity