Skip to main content

Hello, I am new to Streamsets, I have tried to find answer to this question.

I can see a worked example at my place of work where an Orchestration Pipeline can pass a variable down to a Transformer Job in the Runtime Parameters, 

eg…    

 

  1. Collector Pipeline Fragment (Jython)

for record in sdc.records:
  try:
    if :
     ...
    else:
       ...
    record.value"'LATEST_DATE'] = ….
    sdc.output.write(record)

 

  1. Orchestration Pipeline then executed Transformer Job node “branch_one_run” passing the value as a parameter in the Runtime Parameters
"LATEST_DATE": "${record:value("/LATEST_DATE")}",

 

  1. Subsequent Stream Splitter node in the Orchestrator pipeline can determine course of action to take based on whether the call to the Transformer job was successful… ${record:value('/orchestratorTasks/branch_one_run/success')==true}

But is it possible for the Orchestrator to also get a bespoke variable containing value / values back from the called Transformer job that it can use to pass as a parameter to subsequent Transformer Job Nodes?

 

I have seen the following post


which shows a way of picking up parm values from a table so perhaps that is the only way of doing this (using a table as the commarea).  The first transformer called writing its output to a table and then using the Orchestration JDBC node to get this data into a parm for subsequent job calls?

 

@CS_1970 

Please find below the reference document for orchestration pipeline.

https://academy.streamsets.com/learn/course/6/play/89/lab-build-an-orchestration-pipeline

As per my knowledge we can’t  pass job parameter from one job to another.

I might be wrong but can you please elaborate your case study in details ,so i can try to help you.

 

Thanks & Regards

Bikram_

Hello Bikram,  just as you responded I added a note to my original post regarding another Q+A that looks like a solution.  But would involve using a table to pass the output from one Transformer Job to then be retrieved by Orchestrator Pipeline and then useable to pass as a parameter to subsequent Trnsformer jobs.
 

 

ok if it helps in your case then its fine.

Thanks, yes I think that would work.  I haven’t been able to find anything (Google etc) on passing output from one Transformer level job back up to the Orchestrator directly.  But this method of picking up data from table to used for parms looks like next best thing.  Cheers.

Reply


Terms & Conditions