Hi,
We are processing S3 files with the batch size of 1000, the output also we are planning to store in S3 only. But we have he input file of 10000 records so we are seeing 10 output files in s3. As per client requirement we need to create a single file.
Is there a way to create a single S3 output file from Streamsets.
Best answer by Rishi
Yes, with Whole File Data format you cannot modify individual record. I was assuming your pipeline like S3 → S3.
Other workaround I can think of :
Hii
You can use the whole file data format to transfer entire files from an origin system to a destination system ( i.e S3 to S3 in your case) . With the whole file data format, you can transfer any type of file. and this will result in one file as you desired.
Please configure Whole File data format at both origin and destination.
Yes, with Whole File Data format you cannot modify individual record. I was assuming your pipeline like S3 → S3.
Other workaround I can think of :
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
