Few days in the past, I used to be presenting my subject concerning synapse in ‘International Azure Bootcamp – 2022’ when one of many individuals questioned about tips on how to get notified concerning the standing of pipeline actions in a synapse pipeline. After the occasion, I explored among the current and efficient choices which can be obtainable and determined to doc them with a demo for every technique.
Introduction
In Azure Synapse every time coping with knowledge ETL/ELT actions, at any level of time the engineer would possibly must know the state and well being of repositories by going by the metrics. The engineer or administrator can accomplish this by organising notifications throughout the synapse workgroup for crucial metrics which when crossing these thresholds, notification shall be despatched to the response group for corrective actions.
Create alerts
When you click on on ‘Create new alert’, you will note tabs with a number of choices. The ‘Scope’ tab is to pick the Synapse workspace if in case you could have multiple, right here we’ve just one as listed beneath.
Within the situation tab click on on ‘Add situation’ to create new standards for the alert. When you click on you will note a pop-up with record of metrics and log values which you’ll choose a number of based mostly in your requirement. These are known as as ‘Alerts’ in azure diagnostics terminology.
Please word that the alert rule in azure just isn’t a free service. $0.10 shall be charged for every rule created
Let’s choose the sign ‘Pipeline runs ended’ from the record. It will set off the alert based mostly on pipeline exercise or its output, regardless of the situation we specify within the coming steps. I’ll go away that with default hours within the chart interval the place it would verify for the pipeline standing within the final 6 hours.
Scroll down to make use of the following possibility Cut up by Dimensions which can allow you to monitor a number of metric time-series with a single alert rule. A metric time-series is a collection of measurements (or “metric values”) captured over a time period. Whereas I’m not going to make use of the time-series monitoring now, simply to point out you I’ve chosen the ‘Failure Kind’ dimension and customized dimension values named Knowledge error & Knowledge Kind Mismatch.
Additional scrolling down will lead you to ‘Alert Logic’ the place you may set the edge to Static & Dynamic. Since for this demo I’ve chosen pipeline run, I’m deciding on the Static threshold to alert if the pipeline run ends greater than as soon as.
An motion group is a set of notification preferences outlined by the proprietor of an Azure subscription. Azure Monitor, Service Well being and Azure Advisor alerts use motion teams to inform customers that an alert has been triggered. Numerous alerts might use the identical motion group or completely different motion teams relying on the person’s necessities.
Persevering with in the direction of the following step in creating the alert, you can be moved into ‘Actions’ tab to pick an current motion group or create a brand new one.
I created a brand new motion group and setup electronic mail notification within the subsequent tab with my very own electronic mail the place I wish to be notified. As soon as the motion group is created efficiently you’ll obtain an electronic mail affirmation that you’ve got been added to the motion group.
As soon as that is all full now go forward and create alert rule with after offering it a reputation. Simply go to the dashboard as soon as the alert has been created to see what number of notifications has been obtained and their severity. We don’t have any within the beneath screenshot as we haven’t run any pipeline but. Click on on the ‘Alert guidelines’ button above to view the alerts created for this workspace.
Run Pipeline
Now the alert and all of the steps has been accomplished, it’s time to take a look at by operating the pipeline. For this demo I’ve created a easy ‘Get Metadata’ exercise to adls gen2 storage container the place we are going to fetch some fundamental particulars. As per our plan the alert we’ve created ought to notify us if any pipeline runs greater than as soon as.
The pipeline was profitable, and the output triggered has the ‘childitems’ record as per our choice in subject record. Let’s run yet one more time however with ‘Merchandise names’ parameter. That is purely optionally available, I edited it simply to point out you the distinction, you may even go forward and run the identical pipeline twice with out altering something as our core agenda has nothing with the pipeline’s output.
The alerts have been triggered efficiently in my mail and the alert dashboard as properly.
As soon as after I ran the pipeline twice initially, I didn’t obtain any alert in dashboard or in my mail. I desperately created one other alert to verify, therefore there have been two alert entries whereas we created just one earlier.
Conclusion
Utilizing the azure alert metrics is without doubt one of the efficient and easy strategies for organising the alerts and triggers for a synapse pipeline, we are going to discover the opposite strategies in future articles.
