Skip to main content

I am trying to save a single timestamp (latest timestamp from all records) after all batches finished running. However, while the event record gets created it never makes it to an Amazon S3 Destination Stage attached to the Event stream coming out of the Jython Evaluator. This is the code I’m using in the Destroy Script part:

last_batch_event = sdc.createEvent("last_batch_ended", 0)
last_batch_event.value = {"latestTimeStamp": sdc.statet"latestTimeStamp"]}
sdc.toEvent(last_batch_event)

Any thoughts on what I am doing wrong?

@Brenna @dima @Pradeep 

Adding an update: it magically started to work after not touching the code/ pipeline for a few days. I faced the same problem on another pipeline, though. After I closed and reopened the pipeline, the event started to get into the event stream. Is this the expected behavior? I feel I am missing something here...

Hi @BiborSz How are you verifying the event record exists?

@Pradeep Two ways: 

  1. In preview - by selecting “Write to Destinations and Executors” and “Execute Pipeline Lifecycle Events”.
  2. Checking if the Destination/ Executor completed the configured task.

Hello.. i want to create an event to put  _success file in  a particular bucket. Can anyone tell me how to do this? What should i write in create event?@BiborSz

@Manisha Srivastava As per the documentation: sdc.createEvent(<String type>, <Integer version>). So, if you wanted to use a specific event created this way in the Precondition of a next stage, you could do something like this:

new_event = sdc.createEvent("process-successful", 0) 
# you may add a counter and incrementally increase the version if it was relevant
sdc.toEvent(new_event)
# note: in a Jython scripting origin you would use the batch.addEvent(new_event) function

Then, in the next stage, you would check for ${record:eventType() == “process-successful”}.

I hope this helps.

Thank you for this.. for more specific, I need to generate event by groovy evaluator processor and then execute a task through executor (google cloud storage executor). What i am suppose to put as precondition in this @BiborSz 

Sorry, I am not familiar with the Groovy evaluator. I’d have to defer to someone more knowledgeable. 

Thank you @BiborSz .. but yaa i have to look into this. Now i am facing another issue.. is there any way to get event generation in transformer pipeline?

Reply


Terms & Conditions