Question

Different types of Events and their names

  • 3 January 2023
  • 2 replies
  • 32 views

Hi,

In the StreamSets documentation we found very lesser Event Types like:

  1. no-more-data
  2. Pipeline start
  3. Pipeline Stop

Other than these we are not able to able any other Event Type if it exists in the documentation.

Like, if I want to generate an event after the record count is 5000, what should be the event name to put in the below type of equation:

${record:eventType()=='no-more-data'}

 

Please share other Event types and their names for the usage.


2 replies

Userlevel 3
Badge

Hello @Ankur - Thank you for reaching out to the StreamSets Community. 

Each origin generates different types of the event, for example AmazonS3 generate 3 events as shown in the image. One of them is ‘no-more-data-event’ , you can use this to stop the pipeline with the help of pipeline finisher.

 

You can also check this for each origin by navigating to » SDC UI » select the Origin » Generated Events. 

Could you please let us know what exactly is the use-case? What would you like to do if 5000 records are generated?

 

Thank you - AkshayJ

Userlevel 4
Badge

Hi @Ankur 

 

Everything that Akshay mentioned +

 

Pipeline events:

  • Pipeline Start and Pipeline Stop 

Stage events:

If you refer to StreamSets Documentation, each Origin, processor, destination has a list of events that you could refer to.

 

Link is here.

Reply