Azure DevOps Data Factory Dataset and pipeline CI/CD Parameters

Question

I am trying to build the CI/CD for Azure Data Factory using Azure DevOps. I am able to setup the Pipeline and Release. However I have a problem:

Dataset

I have 2 environments, DEV and PROD, How can I pass parameter in the CD pipeline to change the File path (e.g. dev and prod) in different deployment stage (dev and prod environment) in Sink and Source

enter image description here

Thank you for your help!

Answer 1

There is another approach to publish ADF, from master (collaboration) branch. You can define (replace) value for every single node (property) in json file (ADF object). It will resolve your problem as you can provide separate CSV config file per each environment (stage).

Example of CSV config file (config-stage-UAT.csv):

type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT

Then just run such cmdlet in PowerShell:

Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"

Check this out: azure.datafactory.tools (PowerShell module)