Skip to main content
Question

Transfer of data


Diana Lacatus
Fan

Hi everyone,

I would like to know if it is possible to introduce a delay between the transfer of the records, in order to not send the whole information at once. I want to transfer the data record by record.

Could you help me, please?

4 replies

alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 73 replies
  • November 5, 2021

Hi @Diana Lacatus 
You can change the batch size to 1 which will make all the batches of size 1. 

Apart from that we have a delay processor that can introduce any delay between batches. I think that this might help you


dima
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 83 replies
  • November 5, 2021

Along with changing batch sizes at the origin, as @alex.sanchez pointed out, you can also set a rate limit at the pipeline level. These two in combination can probably help you get what you're looking for, @Diana Lacatus


Sanjeev
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 53 replies
  • November 5, 2021

and just in case you need to add a delay in passing the records to the target or next stage, you can use the Delay processor. 


Drew Kreiger
Rock star
Forum|alt.badge.img
  • Senior Community Builder at StreamSets
  • 95 replies
  • February 7, 2022

@Diana Lacatus did the responses above help? Please select a “best answer” for others to see what helped with your specific question. :)  


Reply