Skip to main content

Suppose the CDC origin detects a change in the table and captures it, and the pipeline is processing the data, if for some reason the data is being processed very slowly and you want to reset the origin and also delete the change it captured how will one do it ?

I have noticed that even if I stop the pipeline and start it again by saying reset origin and start it still has the change it captured earlier and starts processing it again. So I wanted to know a way to delete that change which it captured earlier in the previous run. 

Hello @lex03 - Thank you for reaching out to the StreamSets Community Platform. It is expected to see  origin processing all the message from starting when pipeline is started with reset origin option.

If you would like to fetch data at some particular point then you can use the Initial Change configuration. The initial change is the point in the Write-Ahead Logging (WAL) data where you want to start processing.

More information is available here:

https://docs.streamsets.com/portal/datacollector/4.4.x/help/datacollector/UserGuide/Origins/PostgreSQL.html#concept_cfs_4m4_n2b

 

Thank you,

Akshay Jadhav

Reply


Terms & Conditions