Skip to main content

Hi,

We have a Scala spark Job where we read couple of CSV files into data frames and create temporary views ("createOrReplaceTempView") on those data frames. Then we write business rules logic in Spark SQL using the tempororary views created earlier and write the result into another dataframe.

We tried to incoroporate this Scala Spark Job in Transformer using Scala Stage. But we are facing an error while creating “createOrReplaceTempView”. Could you please help resolve the issue and let us know how we can incorporate Scala Spark Job in Streamsets Transformer?

 

Thanks,

Pradeep

Hi @Pradeep_Bala Could you please let us know the error message you are facing? And along with that please share the pipeline with us.

NOTE: If you have an account for support, you can open a support ticket to troubleshoot this issue

@Pradeep_Bala ,

can you please try to use “createGlobalTempview” in your scala and i hope it will help to solve your problem.

I was facing similar issues while using the local temp view but after some analysis , i found creating globalview is effective and its working fine for me.

Please let me know if it helps.

 

Regards,

Bikram_

Reply


Terms & Conditions