We can publish a pipeline and run it as a job and also we can schedule a job to run it in desired time interval, but if we want to do the same thing using a python script then what will be the procedure?
Already have an account? Login
Login to the community
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
@sanket.bhendarkar - Thank you for reaching out to the StreamSets community. You can use the StreamSets SDK for Python which enables users to interact with StreamSets products programmatically, and is compatible with Python versions 3.4-3.9.
Here is the the GitHub Tutorials:
Following links will have example of how you can create a job and schedule it from SDK itself:
Creating jobs: https://docs.streamsets.com/sdk/latest/usage/sch/jobs.html
Job scheduling: https://docs.streamsets.com/sdk/latest/usage/sch/scheduled_tasks.html
Let me know if you have any further questions.
Thank you - AkshayJ
The GitHub link which is provided above suggests to have some prerequisites before running the codes in the terminal.
The 2nd prerequisites is to have StreamSets for SDK Installed and activated, but the link provided is not having anything. Can you please help me to install and activate the SDK.
@saikat_chatterjee - For activation, you would have to reach out to the Support or sales team to get the license key of the SDK.
Ok got it,
@saikat_chatterjee - I would request you to mark the command as best answer to resolve the thread.