How to query BigQuery table in the pipeline and use the result for further processing

  • 24 August 2023
  • 0 replies

Hi Friends,

I am new to StreamSets and trying to build a pipeline to handle event data coming from Google Cloud pub/sub to Google Cloud BigQuery. I am stuck with a problem and need your help.

Whenever an event enters the pipeline, I will have to check in the Bigquery tables whether the event already exists in the tables. We have a unique id in the event against which i can query the BigQuery table to check and further continue in the pipeline. I tried various options(including jdbc lookup) but no luck.

Can you please help?


Thanks in Advance.




0 replies

Be the first to reply!