Solved

Origin related question

M
Rank

i have to get data from a server and i can only get 100 tables at a time from it. I have to run the pipeline again if there are more than 100 tables in it. how can i check whether the server is empty or has data left in it? how to automate the checking process

 

icon

Best answer by Rishi 27 June 2022, 16:08

View original

3 replies

Bikram
Rank
Userlevel 5
Badge +1

In Rest api , you can configure for the error code 400 , in this case it will throw error message which signifies no. more data in api.

M
Rank

what if i’m using rest api calls to get data. how to know whether the server is empty? does the above solution works for this case too?

 

Rishi
Rank
Userlevel 4
Badge

hi @Mahesh Assuming that you have JDBC Multitable consumer Origin, since you are reading from multiple table.

You can leverage the event record no-more-data generated by JDBC Multitable Consumer origin. ref : here

 

sdc.event.type Event type. Uses the following type:
  • no-more-data - Generated when the origin completes processing all data returned by the queries for all tables.
  • schema-finished - Generated when the origin completes processing all rows within a schema.
  • table-finished - Generated when the origin completes processing all rows within a table.

 

With the help of event stream you can stop a pipeline after processing all available data.  
more detailed can be found here 

 https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Solutions/StopPipeline.html#concept_kff_ykv_lz

 

And have scheduler configured in control hub to trigger this job periodically 

https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/Scheduler/ScheduledTasks_Creating.html#task_y5k_f1g_ndb

Reply


Terms & Conditions