Skip to main content
Question

Custom Stages in StreamSets Transformer pipelines


How easy would it be to create a custom Scala or PySpark stage, that can output an array of spark data frames (like it can receive for input) rather than just one?

2 replies

Bikram
Headliner
Forum|alt.badge.img+1
  • Headliner
  • 486 replies
  • January 31, 2023

@collid 

In this case the custom function will help you in fixing the issue.

Create the custom function scala and deploy the JAR in in streamsets transformer .In scala processor , import the UDF in it to use the function.


  • Author
  • Roadie
  • 6 replies
  • February 1, 2023

Thanks for the response!

 

Could you possibly share an example? as i’m a little unsure on what you mean exactly. 

 

Thanks


Reply