How do I change the max batch size in SDC?

  • 28 December 2021
  • 2 replies
  • 527 views

Userlevel 3
  • StreamSets Employee
  • 0 replies

Question:

How do you change max batch size for production?

 

Answer:

To be able to increase the max batch sizes for production, you will need to make changes to production.maxBatchSize in sdc.properties (If you are using Cloudera Manager, you can change it via StreamSets configuration -> Max Batch Size (Running)). By default, the max batch size is set to 1000.

After the changes, you need to reset the Data Collector.


2 replies

Userlevel 1

What exactly does “for production” mean in the above answer? If that is referring to a production environment, what then controls the batch size in a non-production environment?

Userlevel 3
Badge +1

@Sami , can you provide more context on the “batch size”.

 

I thought the batch size is set at each individual pipeline level, on the source stage configuration. 

 

Thanks.

 

Cheers,

Srini

Reply