Job not starting due to error: "System pipeline is still running. Please wait until it is stopped."

This issue might happen when the system Job is hanging in a deactivating state and this is preventing the main pipeline to start again.

In order to fix this problem, we can try to stop the system pipeline using the REST API. We can do it from SCH going to left navigation bar -> RESTful API -> Job Runner.

From these options, we need to use the one below and you need to provide the JobID of the main Job:

https://cloud.streamsets.com/jobrunner/rest/v1/job/<job-id>

Which will return a JSON with the pipeline information, we only need the system job ID. E.g.:

    "systemJobId": "ea7a137f-5235-4c1b-9916-9e8911e6c12f:org",

And with that JobID use the Stop Job API call:

https://cloud.streamsets.com/jobrunner/rest/v1/job/<system-job-id>/stop

In case the SCH REST API is being called from the UI, please note that the SystemJob ID needs to be add in the parameter selected in the image below:

This should stop the system job and allow us to start the main job again but there are some cases where we will need to acknowledge the error in the System Job so it can be in an INACTIVE state. We can do this with the API call below:

https://cloud.streamsets.com/jobrunner/rest/v1/job/<system-job-id>/acknowledgeError

Now the System Job-status should be INACTIVE and the main Job should be able to start again.