Skip to main content

I'm using oracle CDC pipeline, Sometimes it is not fetching any data, so in that cases I need to get notified via mail.

(Ie: We triggered the job , and job is running. But even after if 15 minutes the input is still zero. In this scenario I should get mail ).

Is there any way to get notified like this, If it is  either directly from streamsets UI or through Databricks it is acceptable?

@SELINJ depending upon the CDC origin configuration(Initial change, Maximum Transaction length etc) you can expect the pipeline not to process any records for a while when the pipeline is started for the first time. For instance, if the the ‘Maximum Transaction length’ is set to let’s say an hour then the pipeline will read the redo logs starting from an hour ago from the pipeline start time which can take some time. Also, if the Initial Change is set to latest then if there are no changes to the table since the pipeline started then also you won’t see any records flowing through.  Regarding the notification question, setting up some Data SLAs can be an alternative. 

Reply


Terms & Conditions