Question

how to add new stages to an existing pipelines using streamsets python SDK?

  • 7 October 2021
  • 5 replies
  • 99 views

Hi team,

how to add new stages to an existing pipelines using streamsets python SDK? 


5 replies

Userlevel 3
Badge

@mblahay There are a lot of nuances when it comes to designing SDK methods that could add a stage to an existing pipeline (e.g., how to have users specify how to remove the lanes between existing stages when inserting such a stage, what to do about complex pipelines with branches) that have led to us keeping this feature request in the backlog. For what it's worth, the SDK does supporting editing existing pipelines (e.g., to update existing stage configurations or set pipeline-level configurations), so if that functionality is needed, it's there. 

Userlevel 1

I’m new to the SDK and am researching what can be done with it. Am I reading correctly in this thread that although one may create a new pipeline, one cannot add stages to an existing pipeline? It seems odd that this would be missing, especially considering it must be implemented as part of the API that SCH uses when making modifications to pipelines.

Userlevel 6
Badge +3

@bonthunagireddy03 Thanks for the update.

Hi Dash, Thanks for the update, StreamSets Python SDK is not helping for this use case, hence reached out community to see if any alternative approaches are available. However, I got to know that this future is not available in the current SDK version though StreamSets support channel, and enhancement request is filed for the same.

Userlevel 6
Badge +3

Hi @bonthunagireddy03!

 

Here’s the link to StreamSets SDK for Python tutorials -- https://github.com/streamsets/tutorials/tree/master/sdk-tutorials/sch. Hope that helps.

 

Reply