Skip to main content
Question

JDBC Query Consumer not populating jdbc.tables record header

  • September 20, 2022
  • 7 replies
  • 64 views

Hi! I have a pipeline that uses a JDBC multitable consumer origin and that populates the jdbc.tables header attribute, but when I change out the origin to a JDBC query consumer it is not longer populated. Any advice on why this might be happening?

 

7 replies

alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 73 replies
  • September 20, 2022

Hi @annbandy,

In JDBC Query Consumer, you have to check the "Create JDBC Header Attributes" configuration in order to get that produced.

Thanks


  • Author
  • Fan
  • 3 replies
  • September 20, 2022

HI @alex.sanchez

Create JDBC Header Attributes is set to true! They’re all populated except for jdbc.tables.


alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 73 replies
  • September 20, 2022

So, we are getting the table names from the resultSet metadata, depends on your query, that can actually be empty, and I feel this is the case.

If you try with a simpler query: “select *  from table” it should work straight, but depending on the query it can be empty on the results sets, and we might not be able to read it.

As a workaround, given that you know the query in advance, you can also set it using some field processor.


  • Author
  • Fan
  • 3 replies
  • September 20, 2022

I am using a simple select * from already. However I am using parameter to specify the schema and table. So the query looks like select * from ${schema}.${table}. Do you think this could be cause?


alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 73 replies
  • September 20, 2022

Which SDC version are you using? JDBC Multitable had that option long time back, but only newer versions from JDBC Query Consumer are populating those headers


  • Author
  • Fan
  • 3 replies
  • September 20, 2022

I am using 4.1.0


alex.sanchez
StreamSets Employee
Forum|alt.badge.img
  • StreamSets Employee
  • 73 replies
  • September 20, 2022

Hi @annbandy,

 

I’ve asked someone in the Engineering team to take a deeper look and seems that you were always correct, we will investigate further and try to get that fixed on a future version, please keep an eye on the release notes, and thank you for reporting it