Skip to main content
Solved

Why offset is available when the jbo is running, but empty when job is finished

  • November 22, 2022
  • 3 replies
  • 181 views

  • Opening Band
  • 8 replies

I have a pipeline that uses a JDBC Query Consumer, running in incremental mode. The source table is in MSSQL.

My offset column is in datetime type, and is never null (although nullable).

While the job is running, I can retrieve the offset values in the GUI. However, after the job finishes successfully, when I try to retrieve the offset values, the result is empty.

In the data table, there are about 250k records, belonging to 5 distinct dates. I can’t see why the latest offset cannot be saved.

In the select query, I have this → SELECT * FROM dbo.PHIN_Bankruptcy_Load WHERE ReportDate > '${OFFSET}' ORDER BY ReportDate

I would appreciate any tip that can help me overcome this problem - how to make the pipeline save the offset at the end of the run?

Best answer by jyuan

I found the cause of my problem. In the Pipeline Finisher Executor, I forgot to unset the “reset origin” checkbox.

View original
Did this topic help you find an answer to your question?

3 replies

saleempothiwala
Headliner
Forum|alt.badge.img

@jyuan 

You can use Rest API to get the latest offset value:

 

https://eu01.hub.streamsets.com/jobrunner/rest/v1/job/{jobId}/committedOffsets

 

This API returns latest committed offsets.

 

Replace {jobId} with your job ID. You can get your job ID from here:

 

Click on ‘Show Additional Info’

 

 


  • Author
  • Opening Band
  • 8 replies
  • Answer
  • November 22, 2022

I found the cause of my problem. In the Pipeline Finisher Executor, I forgot to unset the “reset origin” checkbox.


saleempothiwala
Headliner
Forum|alt.badge.img

Ah! Well spotted :-)


Reply