Hi Team,
I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.
Is there any workaround we have in SDC. Below are my datasets,
Dataset 1 - CSV
Dataset 2 - Snowflake table
Any way to join Dataset 1 and Dataset 2 in SDC.
TIA
+1
In SDC, there isn't a built-in join operation, but StreamSets Transformer makes it straightforward to manage.
Unlike SDC, where configuring more than one origin can be challenging, in your case, you can simply read data from a CSV file and then utilize JDBC lookup to retrieve data from Snowflake based on the information obtained from the CSV file.
e.g
Data received from CSV is emp = ‘Bikram’
In jdbc lookup
select emp from Test
where emp = ‘${record:value(‘/emp’)}’
This will check if emp data is available in snowflake db or not , we can add as much filter condition .
Please give a try and let me know if it helps.
Thanks & Regards
Bikram_
I have never used JDBC look up, if you can help me with sample configuration then it will be easy for me to implement.
+1
Please find below the screenshot for your reference.
In where condition , left side value is from table and right side value from the source .
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
