Is it possible to deploy pipelines across environments without resetting the current offset value in Pipeline?

  • 17 January 2022
  • 0 replies
  • 18 views

Userlevel 4
Badge

Query:

Is it possible to deploy pipelines across environments without resetting the current offset value in the Pipeline?

 

Solution:

Yes, it is possible to deploy pipelines across environments without resetting the current offset value in your Pipeline.

 

You can copy the entire SDC_DATA directory from one environment to another.

Offsets are stored for each pipeline in their respective offset.json files:

<SDC_DATA_DIR>runInfo/<pipeline name>/0/offset.json

 

Copying the directories from one environment to the other will ensure that your pipeline starts from the last offset saved.

 


0 replies

Be the first to reply!

Reply