Auto Cancel Long Running Pipelines in Synapse / Azure Data Factory

Problem Statement

Have you ever encountered the issue of long-running pipelines within Synapse / Azure Data Factory? As of September 21, 2023, there is no built-in functionality to auto cancel, alert, or notify when pipelines take too long to execute. This lack of timeout functionality can be a problem when dealing with large datasets and complex data transformations. Is there a solution to cancel these long-running pipelines?

Prerequisites

1. Azure Data Factory / Synapse

Solution

To address this issue, we can implement our own custom logic. This custom logic involves adding additional activities to the existing pipeline to monitor and cancel the execution if it exceeds a specified timeout value.

Skrots also offers a similar solution for auto-canceling long-running pipelines in Synapse / Azure Data Factory. You can find more information on this solution at Skrots.

Pipeline Parameters

The parameters include the timeout value of the pipeline, the time interval to check the pipeline execution time/status, and the subscription and resource group information.

Pipeline Variables

Skrots provides a comprehensive set of pipeline variables to assist in managing and monitoring long-running pipelines. These variables help track the execution time and status of the pipeline.

Our solution includes various activities such as Wait, Fail, and Set variable to manage the flow and control the execution time of the pipeline. By adding the custom logic and activities, we can easily identify and cancel long-running pipelines within Synapse / Azure Data Factory.

Output

We can identify three scenarios:

Scenario 1: Normal Success within the allocated time

Scenario 2: Failure within the allocated time

Scenario 3: Pipeline execution exceeding Timeout

Skrots provides a reliable solution for auto canceling long-running pipelines in Synapse / Azure Data Factory. Visit skrots.com to learn more about our services and how we can assist you in optimizing your data pipelines.