Azure

Create A Copy Of Azure Data Factory Using Azure ARM Templates

Introduction

In day-to-day operations, we must have faced requirements to backup and restore or copy an Azure data factory from existing to new ones. In today’s demo we will see how can we backup and restore the Azure data factory using ARM templates export/import option in azure data factory studio.

Steps

I will create a simple pipeline that will copy few rows of data from my azure blob storage into an Azure SQL database. For this, I am going to create a blob storage and add a file into it which we are going to use to upload records into the Azure SQL database. I have created a database and with it a table called financial which could hold the data that we are going to import from blob storage.

You could see from the above picture that currently there are no records in the table.

The pipeline has completed successfully. Now check and we can see the database table and the records that have been copied from blob to database table.

Backup Azure data factory

As stated earlier, we are going to utilize the ARM template Export/Import method to backup and restore the Azure Data Factory. Go to the ‘Manage’ tab on the left side pane and select ‘ARM template’ under source control. The idea is to export the ADF and restore it into a new ADF pipeline.

Backup Azure data factory

Once you click Export ARM template you will have a zip file downloaded into your folder which you have to unzip to see the highlighted JSON file is available. With this step we have completed the backup process. Now we have to use this file to restore it into a new ADF.

Backup Azure data factory

Restore Azure data factory

Before starting with the restore process we have to create a new data factory from azure portal. I have named the new data factory as adfdemo100222. Once created open the same in data factory studio and click on ‘manage’ tab from the left side pane, followed by the options highlighted in the image below to import ARM template. Ultimately we are going to load the ‘arm_template.json’ file from the folder which we have extracted from the zip previously.

Restore Azure data factory

Since we are loading the ARM template file from the backup we have created it is a custom deployment. So instead of choosing the common templates, let’s build our own template from the editor.

Restore Azure data factory

In the next screen, you will see the edit template with a few lines of code, you can go ahead and delete that and click on ‘Load file’ option above to load the arm_template.json file that we have extracted from the zip file.

Restore Azure data factory

I have modified the name of the file which contained the old name to ‘adfdemo100222’ for identification.

Restore Azure data factory

After modifying the name now save the file where will be required to enter the connection string details with the resource groups and region.

For ‘blob storage connection string’ go to the storage and then access keys, click Show keys to copy the connection string details.

  1. Data factory name
  2. Connection string of blob storage where the sample files has been stored
  3. Connection string of SQL database

Restore Azure data factory

For SQL database connection string details go to the databaseà connection string tab in the left side pane. You can copy the string available in the ADO.NET tab and make sure you replace the password before pasting into the create deployment page.

Restore Azure data factory

Once done, go ahead and make sure it is successfully deployed.

Restore Azure data factory

We could see the copy of our restored pipeline from the old data factory instance. So you can go ahead to see how the restored pipeline is working.

Go to the pipeline and just run the pipeline and check the count of the table records in the database table. The count should be increased as the data from the file should have been appended in the table. In my case, the file contained 99 rows and there were 4 existing rows already taking the count to 103.

Restore Azure data factory

Restore Azure data factory

We could see that the table count has been increased to 203 rows, 103 existing and 99 appended in the second run.

Restore Azure data factory

Summary

This is the simplest form of backup and restore for an azure data factory pipeline using ARM template. Hope the live demo will come handy in case you want to implement such methods as well as for your understanding.

Tags
Show More

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Close