Azure Data Factory CI/CD using GitHub Actions

In this guide, we show how to do continuous integration and delivery in Azure Data Factory with GitHub Actions.

Lots of online resources, on implementing Continuous Integration and Continuous Deployment for Azure Data Factory using Azure DevOps:

• Continuous integration and delivery - Azure Data Factory | Microsoft Learn

• Automated publishing for continuous integration and delivery - Azure Data Factory | Microsoft Learn

• Deploy linked ARM templates with VSTS - Azure Data Factory & Azure Synapse | Microsoft Learn

• Azure Data Factory (ADF)— Continuous integration and delivery (CI/CD) | Medium

Some articles on using GitHub, GitHub Actions to implement ADF CI/CD:

• Azure Data Factory CI/CD with GitHub Actions

• Implementing CI/CD for Azure Data Factory using Github Actions

• Azure Data Factory CI/CD with GitHub Actions

GitHub code resources:

• Azure-DataFactory/SamplesV2/ContinuousIntegrationAndDelivery at main · Azure/Azure-DataFactory (github.com)

• JFolberth/adf-cicd-demo: Example of Azure Data Factory source control configuration with GitHub including CICD GitHub Actions.

• Joe-Code/adf-cicd-demo: Example of Azure Data Factory source control configuration with GitHub including CICD GitHub Actions.

• AdamPaternostro/Azure-Data-Factory-CI-CD-Source-Control: How do to CI/CD with Azure Data Factory (github.com)

---

Here is a detailed guide on how a simple CI/CD workflow can be implemented using GitHub Actions to promote ADF data pipelines from one environment to another.

Configure GitHub Repository Settings

After creating and initializing a new repository, go to Settings tab to further configure the detailed options for establishing operating boundary.

Under Collaborators and teams > Manage access > Add teams:

1. ADF Engineering Team granted with Role: Maintain

2. Admin Team granted with Role: Admin

3. Everyone Team granted with Role: Read

Under Collaborators and teams > Moderation options > Interaction limits, restrict repository interaction as such:

1. Enable Limit to existing users

2. Enable Limit to repository collaborators

Under Collaborators and teams > Moderation options > Code review limits:

1. Enable Limit to users explicitly granted read or higher access

Under Code and automation > Branches > Add rule for main branch with following setting enabled:

1. Require approvals (Required number of approvals before merging: x). Start with more reviewers and then slowly transit to peer review to increase speed of deployment into production

2. Require review from Code Owners

3. Restrict who can dismiss pull request reviews (Add the Admin Team. This is particularly useful for out-of-the-norm scenarios)

4. Allow specified actors to bypass required pull requests (Add the Admin Team. This is particularly useful for out-of-the-norm scenarios)

5. Require approval of the most recent reviewable push

6. Require status checks to pass before merging and Require branches to be up to date before merging

7. Require conversation resolution before merging

8. Lock branch

9. Do not allow bypassing the above settings

Under Security > Secrets and variables, set the Secrets:

1. AZURE_TENANT_ID (take from below steps)

2. AZURE_SUBSCRIPTION_ID (take from below steps)

3. AZURE_RESOURCE_GROUP (where your ADF is in)

4. AZURE_USER_ASSIGNED_MANAGED_IDENTITY_CLIENT_ID

5. AZURE_DATA FACTORY_NAME_DEV

6. AZURE_DATA_FACTORY_NAME_PROD

7. etc..

Under Security > Secrets and variables, set the Variables:

1. ACTIONS_RUNNER_DEBUG: false

2. ACTOINS_STEP_DEBUG: false

The above 2 flags are useful during initial setup. Set them to true to examine detailed logs during GitHub Actions run. See Enabling debug logging - GitHub Docs

Create folders and files required for GitHub Repository for CI/CD

1. Create these files in the repository:

2. “.github/workflows/main.yml”

3. “.github/workflows/validatepullrequest.yml”

4. “.github/pull_request_template”

5. “.github/CODEOWNERS”

Set up User Assigned Managed Identity for GitHub Actions to deploy ARM Templates

Create credentials for ARM deployment when code is pushed to main branch:

1. Go to Azure, search for Managed Identities and create a new User Assigned Managed Identity

2. Once created, go to Federated Credential and Add Credential

3. Under Federated credential scenario, select “GitHub Actions deploying Azure resources - Configure a GitHub issued token to impersonate this application and deploy to Azure”

4. Fill in the Organization, Repository

5. For Entity, select “Branch”

6. For Branch, select “main”

Create credentials for ARM deployment when pull request is raised:

1. Repeat steps 1 to 6.

2. Then edit Subject Identifier to put in:

   “repo:<your-org>/<your-repo>:pull_request”

Go to Settings > Properties and set these into the above created Secrets in GitHub Repository > Under Security > Secrets and variables

1. Tenant ID

2. Client ID

3. Subscription ID

Set up Azure Data Factory for CI/CD

Set up GitHub Actions to run the CI/CD workflow

ARM Template itself has limits: 256 variables, 800 resources (including copy count), 64 output value, 10 unique locations per subscription/tenant/management group scope, 24,576 characters in a template expression and 4mb file size for template and template parameter files and it affects ADF CI/CD workflow:

• I don't understand the ramificaitons of the message: "Your ARM template exceeds the limit of 256 parameters" after enabling Source Control with Azure Devops - Microsoft Q&A

• Azure datafactory issue with ARM templated - Stack Overflow

• Customize parameters to Azure Data Factory ARM Templates - Stack Overflow

• More Woes with ADF Deployments - bzzzt!

Setting up ADF CICD workflow to utilize Linked ARM Templates may overcome the 4mb file size limit but it does not solve the 256 parameters limit.

A simpler alternative to overcome the limit is just design the architecture to provision multiple Azure Data Factory instances upfront.