Question

Consuming MySQL binlog data in JDBC producer

  • 2 August 2023
  • 6 replies
  • 40 views

I am trying to build CDC pipeline to migrate MySQL database to MySQL database in different server.
Here’s the data collector pipeline I’ve created. 
 

As per the documentation here, JDBC producers should be able to process binlog data. I’ve used Field remover to only use /Data and /Table fields. 

 

When I run the pipeline I am getting error that input record has no data for <schema>.<table> . How can I create a pipeline to consume binlog records?


6 replies

Please feel free to let me know if more details are needed.

Userlevel 5
Badge +1

@HIMANSHU_SURANA 

Kindly just preview the pipeline and see if you manage to fetch CDC data from binary log.

In place of JDBC producer , please add jdbc tee processor to handle the CDC data.

Please find below the document for your reference and i hope it will help you.

 

https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Pipeline_Design/CDC-Overview.html#concept_apw_l2c_ty

https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Processors/JDBCTee.html#concept_qfd_tpm_5y

Hi @Bikram , thank you for your reply.

 

I used the JDBC tee as you suggested and also tested om preview mode. The data is coming as a record from binlog as expected. But I’m still seeing an error in JDBC tee during processing mode (which was same as without JDBC TEE). : Record1-Error JDBC_90 - Record doesn't have any columns for table

 

Userlevel 5
Badge +1

@HIMANSHU_SURANA 

When you are using the jdbc producer or jdbc tee , please make sure the column names exactly match with the source column name else it will throw error.

 

Kindly check on it .

 

Hi @Bikram , thank you for your reply. 

My objective is to migrate all tables from one database, is there a way to dynamically map this in jdbc tee?
 

This is my setup
 

And since I have flattened the record fields, how can I use it in column mapping?
Below is the input for JDBC tee, please see these columns will change based on table so how should I map it? Also see, that is homogeneous migraiton the destination column names are same as source.
 

 

Userlevel 5
Badge +1

 

@HIMANSHU_SURANA 

we have option for column mapping . May i know if you try this .

 

 

Reply