Question

Running SQL scripts using Streamsets Transformer

  • 6 February 2023
  • 1 reply
  • 26 views
R
Rank

Hi Everyone,
I am new to Streamsets and working on a project which has some specific requirement.
We need to use the Transformer.

We have run several sql scripts to create table and insert data into them from another table using Streamsets Transformer. The table creation needs to be done on AWS S3.

I am using Spark SQL Query, but when i am trying to run 2-3 sql scripts within one Spark SQL Query component it is giving an error.

Requesting your help on below:
1. Can we run multiple scripts within one Spark SQL Query component? If yes, how.
2. What is the best way to execute a set of sql scripts that needs to be run in a sequential manner in Streamsets transformer.?
Any help or advise would be much appreciated.

Thanks a lot in advance.

1 reply

saleempothiwala
Rank
Userlevel 4
Badge

@Rachit_Agarwal 

The way Transformer works is that every processor generates 1 or more DataFrames that are passed as input to the next processor. So Spark SQL Query is expecting a dataframe on which query will be executed and a new dataframe will be created.

 

If you want to run multiple queries then you will have to use multiple processors.

Reply


Terms & Conditions