Skip to main content
Question

How to add custom event record to event stream from Jython Evaluator Destroy Script?

  • March 10, 2022
  • 9 replies
  • 265 views

BiborSz
Fan

I am trying to save a single timestamp (latest timestamp from all records) after all batches finished running. However, while the event record gets created it never makes it to an Amazon S3 Destination Stage attached to the Event stream coming out of the Jython Evaluator. This is the code I’m using in the Destroy Script part:

last_batch_event = sdc.createEvent("last_batch_ended", 0)
last_batch_event.value = {"latestTimeStamp": sdc.state["latestTimeStamp"]}
sdc.toEvent(last_batch_event)

Any thoughts on what I am doing wrong?

9 replies

Drew Kreiger
Rock star
Forum|alt.badge.img
  • Senior Community Builder at StreamSets
  • 95 replies
  • March 16, 2022

@Brenna @dima @Pradeep 


BiborSz
Fan
  • Author
  • Fan
  • 4 replies
  • March 16, 2022

Adding an update: it magically started to work after not touching the code/ pipeline for a few days. I faced the same problem on another pipeline, though. After I closed and reopened the pipeline, the event started to get into the event stream. Is this the expected behavior? I feel I am missing something here...


Pradeep
StreamSets Employee
Forum|alt.badge.img+1
  • StreamSets Employee
  • 48 replies
  • March 17, 2022

Hi @BiborSz How are you verifying the event record exists?


BiborSz
Fan
  • Author
  • Fan
  • 4 replies
  • March 17, 2022

@Pradeep Two ways: 

  1. In preview - by selecting “Write to Destinations and Executors” and “Execute Pipeline Lifecycle Events”.
  2. Checking if the Destination/ Executor completed the configured task.

Manisha Srivastava
Fan

Hello.. i want to create an event to put  _success file in  a particular bucket. Can anyone tell me how to do this? What should i write in create event?@BiborSz


BiborSz
Fan
  • Author
  • Fan
  • 4 replies
  • March 23, 2022

@Manisha Srivastava As per the documentation: sdc.createEvent(<String type>, <Integer version>). So, if you wanted to use a specific event created this way in the Precondition of a next stage, you could do something like this:

new_event = sdc.createEvent("process-successful", 0) 
# you may add a counter and incrementally increase the version if it was relevant
sdc.toEvent(new_event)
# note: in a Jython scripting origin you would use the batch.addEvent(new_event) function

Then, in the next stage, you would check for ${record:eventType() == “process-successful”}.

I hope this helps.


Manisha Srivastava
Fan

Thank you for this.. for more specific, I need to generate event by groovy evaluator processor and then execute a task through executor (google cloud storage executor). What i am suppose to put as precondition in this @BiborSz 


BiborSz
Fan
  • Author
  • Fan
  • 4 replies
  • March 24, 2022

Sorry, I am not familiar with the Groovy evaluator. I’d have to defer to someone more knowledgeable. 


Manisha Srivastava
Fan

Thank you @BiborSz .. but yaa i have to look into this. Now i am facing another issue.. is there any way to get event generation in transformer pipeline?