I define a pipeline with the processer called “Field Remover”, the job failed and can see all Input rows are shown in Error rows. However when check the log(log severity is ‘ALL’), I cannot see any error or error reason in the log, I can only see an warning “Exception thrown while trying to remove temporary directory /opt/streamsets-datacollector/tmp/sdc1627444042246270257”, how can I get the exact failed reason?
Some context of pipleline, it is to load some data from oracle origin to snowflake internal tables. and it can run well several days, but same workflow failed today, however if I remove the process “filed remover” now, I can see the pipeline run successfully. Any suggestions for this trouble shooting.