Question

Run and schedule a job

  • 4 January 2023
  • 5 replies
  • 73 views

We can publish a pipeline and run it as a job and also we can schedule a job to run it in desired time interval, but if we want to do the same thing using a python script then what will be the procedure?


5 replies

Userlevel 4
Badge

Hello @sanket.bhendarkar  - Thank you for reaching out to the StreamSets community. You can use the StreamSets SDK for Python which enables users to interact with StreamSets products programmatically, and is compatible with Python versions 3.4-3.9.

 

Here is the the GitHub Tutorials: 

https://github.com/streamsets/tutorials/blob/master/sdk-tutorials/sch/tutorial-getting-started/README.md

Following links will have example of how you can create a job and schedule it from SDK itself:

Creating jobs: https://docs.streamsets.com/sdk/latest/usage/sch/jobs.html

Job scheduling: https://docs.streamsets.com/sdk/latest/usage/sch/scheduled_tasks.html

 

Let me know if you have any further questions.

Thank you - AkshayJ

The GitHub link which is provided above suggests to have some prerequisites before running the codes in the terminal.

The 2nd prerequisites is to have StreamSets for SDK Installed and activated, but the link provided is not having anything. Can you please help me to install and activate the SDK.

Userlevel 4
Badge

Hello @saikat_chatterjee  - For activation, you would have to reach out to the Support or sales team to get the license key of the SDK. 

Ok got it,
Thank you.

 

Userlevel 4
Badge

Thank you @saikat_chatterjee - I would request you to mark the command as best answer to resolve the thread.

Reply