Skip to main content
Question

How to add custom event record to event stream from Jython Evaluator Destroy Script?

  • 10 March 2022
  • 9 replies
  • 221 views

I am trying to save a single timestamp (latest timestamp from all records) after all batches finished running. However, while the event record gets created it never makes it to an Amazon S3 Destination Stage attached to the Event stream coming out of the Jython Evaluator. This is the code I’m using in the Destroy Script part:

last_batch_event = sdc.createEvent("last_batch_ended", 0)
last_batch_event.value = {"latestTimeStamp": sdc.statet"latestTimeStamp"]}
sdc.toEvent(last_batch_event)

Any thoughts on what I am doing wrong?

9 replies

Drew Kreiger
Rank
Userlevel 5
Badge

@Brenna @dima @Pradeep 

BiborSz
Rank

Adding an update: it magically started to work after not touching the code/ pipeline for a few days. I faced the same problem on another pipeline, though. After I closed and reopened the pipeline, the event started to get into the event stream. Is this the expected behavior? I feel I am missing something here...

Pradeep
Rank
Userlevel 3
Badge +1

Hi @BiborSz How are you verifying the event record exists?

BiborSz
Rank

@Pradeep Two ways: 

  1. In preview - by selecting “Write to Destinations and Executors” and “Execute Pipeline Lifecycle Events”.
  2. Checking if the Destination/ Executor completed the configured task.
Manisha Srivastava
Rank

Hello.. i want to create an event to put  _success file in  a particular bucket. Can anyone tell me how to do this? What should i write in create event?@BiborSz

BiborSz
Rank

@Manisha Srivastava As per the documentation: sdc.createEvent(<String type>, <Integer version>). So, if you wanted to use a specific event created this way in the Precondition of a next stage, you could do something like this:

new_event = sdc.createEvent("process-successful", 0) 
# you may add a counter and incrementally increase the version if it was relevant
sdc.toEvent(new_event)
# note: in a Jython scripting origin you would use the batch.addEvent(new_event) function

Then, in the next stage, you would check for ${record:eventType() == “process-successful”}.

I hope this helps.

Manisha Srivastava
Rank

Thank you for this.. for more specific, I need to generate event by groovy evaluator processor and then execute a task through executor (google cloud storage executor). What i am suppose to put as precondition in this @BiborSz 

BiborSz
Rank

Sorry, I am not familiar with the Groovy evaluator. I’d have to defer to someone more knowledgeable. 

Manisha Srivastava
Rank

Thank you @BiborSz .. but yaa i have to look into this. Now i am facing another issue.. is there any way to get event generation in transformer pipeline?

Reply


Terms & Conditions