I just wanted to get help how the best to deal with embedded scripts inside a pipeline.
What is the best practice:
- Is there a way to see based on error shown in UI in which line the error occurred in embedded script ?
- Is there any way to debug into the script ?
- What is the best way to find root cause of an error (maybe exception handler with own logging - see below)
- Is there way to emulate the pipeline behavior to test script inside an IDE ?
(eclipse, intellij,...)
So an easy we to emulate the sdc.records object which is the input for an embedded script ?
What i have found out so far:
- The error shown in Pipeline Builder does not tell anything to relate to where the error
occurred in embedded script (at least i could not find any relationship as it just shows streamsets internal classes and not the script itself) - I can add my own exception handler inside my script and log info and stack trace out
via sdc.log.info or sdc.log.error
Use Scripting Processors to Debug StreamSets Data Collector Pipelines | by Bob Plotts | Medium
Many thanks for your help.