Issue:
S3 whole file transfer taking loads of time if files in the folder are high.
Solution:
According to our documentation File Pool Size is the Maximum number of files that the origin stores in memory for processing after loading and sorting all files present on S3. Increasing this number can improve pipeline performance when Data Collector resources permit.
We can try to set up the File pool size in the s3 origin to a number bigger than a number of the files than the origin is trying to read. For example, you can set it to 20000 if you are reading up to 15.000 files.