Hi, I am using JDBC Multitable Origin Stage and I want to run 200+ pipelines in parallel but I have to maintain their process time as low as possible. I am configuring sdc.properties file but I still cannot find the proper values. I have configured max.stage.private.classloaders, runner.thread.pool.size, pipeline.max.runners.count but I still cannot minimize their run time. Whenever they are running in parallel, it seems that their usual individual run time takes longer. Also, whenever I run them in parallel, some pipelines are in STARTING state. They do not run immediately as their pipeline status is STARTING. May I ask what configurations should I consider for me to run these 200+ pipelines concurrently with consideration of lower run time?
Best answer by saleempothiwala
View original