Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines? Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines? azure azure

Azure Data Factory - How can I trigger Scheduled/OneTime Pipelines?


I ran into this same issue, I needed to run my pipeline only when a local job was completed. To do that I modified the local job to kick off the pipeline as its last step. I have a write up here on how to start an ADF pipeline with C#. Here is the link to the ADF developer reference which might also be helpful. I also have an example here on how to trigger ADF pipelines from Azure Functions, if you are interested. This is using the same code from the first example but I get the benefit of running the whole process in the cloud and the ability to use the azure function scheduler.

Here is the relevant method to modify the pipeline. You would need to change the start and end dates based on when you want the slice to run.

public void StartPipeline(string resourceGroup, string dataFactory, string pipelineName, DateTime slice)    {        var pipeline = inner_client.Pipelines.Get(resourceGroup, dataFactory, pipelineName);        pipeline.Pipeline.Properties.Start = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T00:00:00Z");        pipeline.Pipeline.Properties.End = DateTime.Parse($"{slice.Date:yyyy-MM-dd}T23:59:59Z");        pipeline.Pipeline.Properties.IsPaused = false;        inner_client.Pipelines.CreateOrUpdate(resourceGroup, dataFactory, new PipelineCreateOrUpdateParameters()        {            Pipeline = pipeline.Pipeline        });    }


To trigger ADF you need to have input dataset in 'Ready' state. If it is in ready state you can manually go to Monitoring tab to manually 'Re-Run', if input dataset is not ready then you need to make that dataset ready to manually start ADF.


If you want to trigger the job only once then you can set StartDate and EndDate to be the same time:

pipeline.Pipeline.Properties.Start = DateTime.Parse($"{someDate:yyyy-MM-dd}T00:00:00Z");pipeline.Pipeline.Properties.End = DateTime.Parse($"{someDate:yyyy-MM-dd}T00:00:00Z");pipeline.Pipeline.Properties.IsPaused = false;