@Paawan , you intend to use Max Batch Size(Records)? More details here https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Origins/JDBCConsumer.html

@Paawan , as mentioned by @Pradeep , use Max Batch Size, reduce that to 1. It will process one record at a time, from origin to destination. Clearly this is going to have a high impact on performance.

It would be interesting to understand why you feel you need to process from end to end one record at a time - it’s generally a bit of an anti-pattern in Data Integration. But clearly doable.

@Pradeep @Giuseppe Mura each record in the result of the query gives me an id to get some xml data from an api and a table name to load that data in the respective table.

I don’t want to set the global batch size to 1, I want it to pick one id from the query results, which would be used in an api call which will result in multiple records, write these records and then pick the next id from the query result which may have to be written to a different table.

Would this be possible by setting the batch size on jdbc producer to 1?