Environment:
- StreamSets Data Collector (SDC) 5.7.0 & 5.7.1
- An orchestration pipeline which uses Start Jobs stages
- The orchestration stage starts a job using a Job Template
- The orchestration stage is configured with the Run in Background setting disabled
Issue:
In StreamSets Data Collector (SDC) 5.7.0 and 5.7.1, orchestration pipelines that use a Start Jobs stage to start a job using a Job Template do not function properly when Run in Background is disabled. With this configuration, the pipeline should wait for the job to complete before moving to the next stage, but instead the pipeline starts the job and immediately moves on, which can cause unexpected results for downstream jobs.
Resolution:
There is a fix planned for this issue in the SDC 5.8.0 release. In the short term, if your organization has an active support contact, you can contact StreamSets Support to obtain a patch to resolve the issue.