Skip to main content
Solved

jdbc query used toTable data emptying

L
Fan

When the JDBC query component in executors is used to empty table data, it will not stop after starting the task. Note: pipeline finisher has been used on the java script component

 

SQL QUERY:delete from depart_passenger_info

Best answer by Rishi

Hi @lr123 , The general best practice is to use the Pipeline Finisher executor with origins that generate no-more-data events, check the list here. However,  JavaScript origin does not generate this no-more-data event  Which might explain this behaviour. 
 

Please checkout more details about  Pipeline Finisher executor to understand the workflow.

 

View original
Did this topic help you find an answer to your question?

3 replies

Rishi
StreamSets Employee
Forum|alt.badge.img
  • RishiStreamSets Employee
  • StreamSets Employee
  • 96 replies
  • Answer
  • January 7, 2022

Hi @lr123 , The general best practice is to use the Pipeline Finisher executor with origins that generate no-more-data events, check the list here. However,  JavaScript origin does not generate this no-more-data event  Which might explain this behaviour. 
 

Please checkout more details about  Pipeline Finisher executor to understand the workflow.

 

L
Fan
  • lr123Fan
  • Author
  • Fan
  • 1 reply
  • January 10, 2022
Rishi wrote:

Hi @lr123 , The general best practice is to use the Pipeline Finisher executor with origins that generate no-more-data events, check the list here. However,  JavaScript origin does not generate this no-more-data event  Which might explain this behaviour. 
 

Please checkout more details about  Pipeline Finisher executor to understand the workflow.

 

However, I later restarted the server environment and successfully completed this task by reusing JavaScript components. Surprise!

 

Rishi
StreamSets Employee
Forum|alt.badge.img
  • RishiStreamSets Employee
  • StreamSets Employee
  • 96 replies
  • January 10, 2022

@lr123 
With Javascript origin event generation logic has to be put into within script.For more information, see Event Generation for the Jython Scripting Origin  .  

For example, in below code I am create new event type `q_test_event` and the adding this event to batch. 

   try {
        offset++;
        record = sdc.createRecord('generated data');
        var value = prefix + entityName + ':' + offset.toString();
        record.value = value;
        cur_batch.add(record);
        newEventRecord = sdc.createEvent('q_test_event', 1)
	    newEventRecord.value = sdc.createMap(false)
        newEventRecord.value = record.value
        cur_batch.addEvent(newEventRecord)

 

And then in Pipeline Finisher Executor Pre-condition I added the below logic, so that as soon as this event received by Pipeline Finisher Executor , it will stop the pipeline. 

${record:eventType() == 'q_test_event'}

 

Reply


Terms & Conditions