Solved

Why offset is available when the jbo is running, but empty when job is finished

  • 22 November 2022
  • 3 replies
  • 133 views

I have a pipeline that uses a JDBC Query Consumer, running in incremental mode. The source table is in MSSQL.

My offset column is in datetime type, and is never null (although nullable).

While the job is running, I can retrieve the offset values in the GUI. However, after the job finishes successfully, when I try to retrieve the offset values, the result is empty.

In the data table, there are about 250k records, belonging to 5 distinct dates. I can’t see why the latest offset cannot be saved.

In the select query, I have this → SELECT * FROM dbo.PHIN_Bankruptcy_Load WHERE ReportDate > '${OFFSET}' ORDER BY ReportDate

I would appreciate any tip that can help me overcome this problem - how to make the pipeline save the offset at the end of the run?

icon

Best answer by jyuan 22 November 2022, 23:54

View original

3 replies

Userlevel 4
Badge

@jyuan 

You can use Rest API to get the latest offset value:

 

https://eu01.hub.streamsets.com/jobrunner/rest/v1/job/{jobId}/committedOffsets

 

This API returns latest committed offsets.

 

Replace {jobId} with your job ID. You can get your job ID from here:

 

Click on ‘Show Additional Info’

 

 

I found the cause of my problem. In the Pipeline Finisher Executor, I forgot to unset the “reset origin” checkbox.

Userlevel 4
Badge

Ah! Well spotted :-)

Reply