Question

How to alert on error records once a day

  • 14 July 2023
  • 2 replies
  • 34 views

This is my first time working with the alerting rules and I’m struggling to figure out if it is possible to create an alert that will trigger multiple times when a threshold of error messages is reached, resetting after each alert. For example, my job typically has no error records, but recently in our test environment we have been receiving some due to bad input data. I’d like to build an alert that sends me an email once a day if any error records were received that day. I’m trying to avoid getting an alert for every batch of records that has an error record, especially if this alert will go into the production job.

 

Reading over the Alerts and Rules documentation, none of the alert types seem like they fulfill this case. I can get an alert as soon as I get any error records, but I don’t see a way to reset it after a time period except for looking at each batch of records. Any help here would be appreciated!


2 replies

Userlevel 5
Badge +1

@xarnthehero 

Could you please clarify if your pipeline is set to real-time data streaming mode, allowing you to receive notifications in the event of any data errors? If the pipeline is running in batch mode, this should not be an issue.

If you are comfortable sharing the pipeline with me, I would be glad to assist you with it.

Userlevel 2
Badge

@xarnthehero please look into Data SLAs and see if that fits the requirement

Reply