Skip to main content

Use Case - We have to pass dynamic date in Spark SQL Query stage in Streamsets,
We have got an output as a dynamic date using pyspark code, now we need to use output in Spark SQL stages within SQL scripts. How to refer this date output variable in SQL scripts.

Any help and advise would be great.

@Rachit_2022 

I am using the below script to retrieve data based on current date value from the table and i can see data is coming out correctly.

Please let me know if you are looking some other scenario for your requirement and happy to help you on it.

select * from $table
where LastExtractDateTime = current_date

e.g LastExtractDateTime = 2023-06-23 and current_date = 2023-06-23 ,Its matching and getting data.

 

Kindly provide details of your need .

 

Thanks & Regards

Bikram_

Hi @Bikram ,

Thanks a lot for taking out time to view and reply to this post.

Can you please share any sample pipeline that you have for above scenario...we are looking for similar solution that you have mentioned in your reply.

Thanks for the help.

@Rachit_2022 

Attached the pipeline for your reference.

Reply


Terms & Conditions