Skip to main content

Hello community out there,

A newbie to streamsets world.
I am looking for automatic solution means via python script with pipeline, which exports all jobs, all pipelines by calling  StreamSets Control hub.

 

so I am in very initial stage of my script just for testing, to connect with Control hub and list of jobs.

 

script: streamsets_connection.py

import os
import sys
from streamsets.sdk import ControlHub

# Get environment variables
server_url = os.getenv('SDC_SERVER_URL')
credential_id = os.getenv('SDC_CREDENTIAL_ID')
token = os.getenv('SDC_TOKEN_ID')

# Validate environment variables
if not server_url or not credential_id or not token:
    print("One or more environment variables are not set.")
    exit(1)

# Connect to ControlHub
try:
    control_hub = ControlHub(server_url=server_url, credential_id=credential_id, token=token)
    print("ControlHub client initialized successfully.")
except Exception as e:
    print(f"Failed to initialize ControlHub client: {e}")
    exit(1)

def list_jobs():
    try:
        # Fetch all jobs
        jobs = control_hub.jobs.get_all()
        print("List of jobs:")
        if not jobs:
            print("No jobs found.")
        else:
            for job in jobs:
                print(f"Job ID: {job.job_id}, Name: {job.job_name}")
    except Exception as e:
        print(f"An error occurred while listing jobs: {e}")

if __name__ == '__main__':
    print("Starting job listing script...")
    list_jobs()
    print("Finished job listing script.")

 

output:

streamsets_connection.py
ControlHub client initialized successfully.
 

I am unable to list of jobs, any one guide me here why its not listing the jobs.

final goal is to have a script which can export all pipelines and all jobs. 

 

Thanks in advance.

 

 

 

 

 

 

 

 

 

 

@hami ,

  1. in the SCH UI, do you see jobs, if you log in using the same credentials that you are using in the script above? If yes, then yes the script will show jobs and you can proceed to 2 ) below. If not, either you need to create job/s. or you do not have permissions to the existing jobs.

2)Can you try the following please?

# Fetch all jobs
        jobs = control_hub.jobs
        print("List of jobs:")
        if not jobs:
            print("No jobs found.")
        else:
            for job in jobs:
                print(job)
  1. what is the whole output?
  2. also, does it print any errors etc?

Regards-

Kirti

Reply


Terms & Conditions