Skip to main content

The offset is always stored in the Control Hub (SCH). The Data Collector (SDC) stores the offset to the offset.json file and sends the information to the SCH periodically.
 
When 'Enable Failover' is checked, the offset is migratable i.e the SCH can start a job on a different SDC (SDC_2) with the last-saved offset (from the previous SDC - SDC_1, which is not available anymore).
 
If 'Enable Failover' is not checked, the offset is tied to a specific SDC ID and it does not migrate i.e the SCH cannot start a job with the last-saved offset from another SDC (SDC_1) on a different SDC (SDC_2). On a new SDC, the job will run the pipeline from the beginning.
 

Failover needs to be enabled for the offset to be migrated.

Be the first to reply!

Reply