Skip to main content

SCH High Availability/Fail-over and SDC/Pipeline/Job Fail-over


AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Question:

In the case of pipelines with origins that will run only in stand-alone mode (like HTTPClient, TCP Server, Directory, etc.), how are these deployed to a cluster of SDCs?

How would they run in the cluster? How and which of the rest of the SDCs will pickup the pipeline when an active SDC goes down?

 

 Answer:

A description of pipeline fail-over can be found here

Pipelines can be enabled for failover within Control Hub:

You can enable a job for pipeline failover. Enable pipeline failover to minimize downtime due to unexpected pipeline failures and to help you achieve high availability.
When a job is enabled for failover, Control Hub can restart a failed pipeline on another available Data Collector that is assigned all labels specified for the job, starting from the last-saved offset.

A description and information about how to set up Control Hub for High Availability can be found under Setting Up a Highly Available Environment in the On-Premises SCH Installation Guide.

Also, you can also read more about Pipeline Fail-over & Balancing Jobs Enabled for Fail-over.

NOTE: There is no way to assign the job to a particular SDC if all the SDC's contains the same label.

Did this topic help you find an answer to your question?
This topic has been closed for comments