Question:
- How to import pipelines/jobs in the control hub using Rest API?
Answer:
Command to generate cookie:
curl -X POST -d '{"userName":"<username>", "password": "<password>"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:SCH" -c cookie.txt
Generated session token:
sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)
echo "Generated session token : $sessionToken"
Import pipelines:
curl -v 'https://cloud.streamsets.com/pipelinestore/rest/v1/pipelines/importPipelineCommits' -F "file=@/<path>/<pipelineName>.zip" -H "X-Requested-By:SCH" -H "Content-Type:multipart/form-data" -H "Accept-Encoding: gzip, deflate" --header "X-SS-User-Auth-Token:$sessionToken" -i
Import jobs:
curl -v 'https://cloud.streamsets.com/jobrunner/rest/v1/jobs/importJobs' -F "file=@/<path>/jobs.zip" -H "X-Requested-By:SCH" -H "Content-Type:multipart/form-data" -H "Accept-Encoding: gzip, deflate" --header "X-SS-User-Auth-Token:$sessionToken" -i
Troubleshooting Steps:
- Make sure you have @ in the file attribute -F "file=@/<path>"
- For importPipelineCommits/importJobs use zip file.
- If possible, import the pipeline/job manually in the control hub and see whether its successfully.
- If the import is unsuccessful, check the commit.id of the pipeline/job, it should be unique.
