Question

JDBC Query Consumer ignores query Interval

  • 23 May 2022
  • 3 replies
  • 69 views

Hi Streamsets Community,

 

just started out and tried the JDBC Query Consumer in DataCollector. My query executes correctly (offset etc.) but I is continuously executed regardless what I put into the query interval. I tried different values like ${5 * MINUTES} ${5 * HOURS} and also just plain integers like 10000. Any Idea what I’m doing wrong?

 

Thanks Marco


3 replies

This was my misinterpretation, I saw the yellow circle updating all the time in monitoring mode and thought that this was an execution.

I execute the pipeline in a draft run and I see Processing Timer refresh all the time with now records fetched (correct due to offset). I connect to a Postgres demo db and write to a Kafka producer. Environment is self managed docker.

 

Userlevel 3
Badge

Hi Marco,

 

When you say it is continuously executed, what do you mean (and how did you determine that)? Also, can you share some details about your pipeline and what environments it’s connecting to?

Reply