How to send a notification if SDC memory reaches specified threshold limit?

Overview:

• This document helps you to design the pipeline to send a notification if SDC memory reaches a specified threshold.

Configuration:

SDC memory notification pipeline looks as below way:

Cron Scheduler:

• This stage sends the records every 5 minutes to the next stage.
• This is similar to the Linux Cron scheduler.

Control Hub API:

• This stage sends the request to the Control Hub and pulls the SDC metrics.
• Rest API: https://cloud.streamsets.com/jobrunner/rest/v1/metrics/executors - Returns executor uptime, Memory/CPU usage by executor at JVM level
• Here is the output of the stage: 

Field Pivoter:

• This stage converts the array of JSON to individual records.
• Here is the output of the stage:

Field Type Converter:

• This stage converts the memory field from Double to Integer.
• Here is the configuration:

Expression Evaluator:

• This stage converts the Used Memory from Integer to Percentage.
• The configuration looks as below way:

Stream Selector:

• Add the condition for Used Memory to redirect the request to the notification stage.
• Here is the configuration:

Email executor:

• This stage sends an email notification if the SDC goes beyond the specified threshold limit.