I am facing issue i.e. reading data through S3origin within data transformer. I am able to read data through s3 origin in data collector.
Trying to read data from S3 origin and copy the same in different location using S3 destination. I am using EMR as an computing engine. Job runs for several mins. on EMR and completed successfully. There is no error in Logs (Both EMR and StreamSet pipeline Logs). Do get this below Warning but not sure this is causing issue or not. java.nio.file.NoSuchFileException: /data/transformer/runInfo/testRun__9e731964-6f21-4956-99fa-82206f3451f5__149e11c1-f697-11eb-b9dc-fd846d33049d__56e36c1c-f8c6-11eb-9295-0fa62e75e081@149e11c1-f697-11eb-b9dc-fd846d33049d/run1630923519827/driver-topLevelError.log
I have verified staging directory as well. seems like all required files are getting populated there which eventually being read through Spark submit.
At the end, Transformer pipeline ends with status START_ERROR: Job completed successfully.
This is a show stopper as of now as its seems to be very basic issue.
Appreciate any resolution and pointers to proceed further.