Hello Team,
Could you please assist for a solution where we can switch to Data Collector A to Data Collector B for a Job if Job having error in between and go for retry.
Hello Team,
Could you please assist for a solution where we can switch to Data Collector A to Data Collector B for a Job if Job having error in between and go for retry.
Hi
Please go through the document: https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Jobs/PipelineFailover.html?hl=job%2Cfailover
This helps you to achieve your use-case. Let us know if you have any queries.
Could you please share configuration for same as we are getting ,
S3 As Destination :
S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.SdkClientException: Unable to execute HTTP request: Broken pipe (Write failed)
Our Configs :
Failover Retries per Data Collector :1
Global Failover Retries : 1
But it is still trying to connect to same Data Collector.
PS : Our use case is very simple which is SFTP to S3.
Edited :
In S3 Stage (Destination) we have “Retry Count” as 3
For Pipeline we have “Retry Attempts” as -1
In case of
first error job is Switching from “Data Collector A” to “Data Collector B”
Second Error job is Switching from “Data Collector B” to “Data Collector C”
So How the offsetting works here ? let’s say at beginning offset is “0”
Does it save the offset after first error and job will start from there when Data Collector switching is going on ? Or New Data Collector will start job from “0” ?
If you can help with this.
You can refer the documentation for more details:
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.