i have to get data from a server and i can only get 100 tables at a time from it. I have to run the pipeline again if there are more than 100 tables in it. how can i check whether the server is empty or has data left in it? how to automate the checking process
Best answer by Rishi
hi
You can leverage the event record no-more-data generated by JDBC Multitable Consumer origin. ref : here
| sdc.event.type | Event type. Uses the following type:
|
With the help of event stream you can stop a pipeline after processing all available data.
more detailed can be found here
And have scheduler configured in control hub to trigger this job periodically
hi
You can leverage the event record no-more-data generated by JDBC Multitable Consumer origin. ref : here
| sdc.event.type | Event type. Uses the following type:
|
With the help of event stream you can stop a pipeline after processing all available data.
more detailed can be found here
And have scheduler configured in control hub to trigger this job periodically
what if i’m using rest api calls to get data. how to know whether the server is empty? does the above solution works for this case too?
+1In Rest api , you can configure for the error code 400 , in this case it will throw error message which signifies no. more data in api.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
