Skip to main content

I have a pipeline that uses a JDBC Query Consumer, running in incremental mode. The source table is in MSSQL.

My offset column is in datetime type, and is never null (although nullable).

While the job is running, I can retrieve the offset values in the GUI. However, after the job finishes successfully, when I try to retrieve the offset values, the result is empty.

In the data table, there are about 250k records, belonging to 5 distinct dates. I can’t see why the latest offset cannot be saved.

In the select query, I have this → SELECT * FROM dbo.PHIN_Bankruptcy_Load WHERE ReportDate > '${OFFSET}' ORDER BY ReportDate

I would appreciate any tip that can help me overcome this problem - how to make the pipeline save the offset at the end of the run?

@jyuan 

You can use Rest API to get the latest offset value:

 

https://eu01.hub.streamsets.com/jobrunner/rest/v1/job/{jobId}/committedOffsets

 

This API returns latest committed offsets.

 

Replace {jobId} with your job ID. You can get your job ID from here:

 

Click on ‘Show Additional Info’

 

 

I found the cause of my problem. In the Pipeline Finisher Executor, I forgot to unset the “reset origin” checkbox.

Ah! Well spotted :-)

Reply


Terms & Conditions