Skip to main content
Solved

How to configure JDBC Multitable consumer & Azure data lake storage using python sdk 5.0 ?

  • January 25, 2023
  • 3 replies
  • 112 views
S
Fan

We were trying to configure, JDBC multitable consumer using python SDK 5.0. We were able to set below configurations:

JDBC_consumer = pipeline_builder.add_stage('JDBC Multitable Consumer')JDBC_consumer.jdbc_connection_string = 'jdbc:mysql://mysqldb:3306/zomato'JDBC_consumer.username = '****'JDBC_consumer.password = '****'

However, not able to set table config.

 

Same with Azure Data Lake Storage, we are able to set below configurations:

Azure_storage.data_format = 'DELIMITED'Azure_storage.authentication_method = 'Shared Key'Azure_storage.account_shared_key = '***'

However, not able to set Account FQDN & Storage Container / File System

 

 

Please guide us with these configurations.

Best answer by Bikram

@Satyam_Tope 

 

sdc = sch.data_collectors.get(url='http://12345:18630')
builder = sch.get_pipeline_builder(engine_type='data_collector', engine_id=sdc.id)
jdbc_mt1 = builder.add_stage('JDBC Multitable Consumer')
#jdbc_mt.configuration['tableJdbcConfigBean.tableConfigs'][0]['schema'] = 'EMP'
jdbc_mt1.table_configs[0]['schema'] = 'EMP'
jdbc_mt1.table_configs[0]['tablePattern'] = 'Data'
trash = builder.add_stage('Trash')
jdbc_mt1 >> trash
pipeline = builder.build('Bikrams 2nd pipeline')
sch.publish_pipeline(pipeline, commit_message='fourth commit of Bikrams pipeline')

View original
Did this topic help you find an answer to your question?

saleempothiwala
Headliner
Forum|alt.badge.img

Hi @Satyam_Tope 

 

Easiest way is to create a pipeline in UI, set these properties and then use SDK to read the stage and see what values have been set for these properties. You will get the stage property name and respected values.

 

You can then use SDK to set these in your new pipeline

S
1 person likes this
  • S
    Satyam_Tope
Bikram
Headliner
Forum|alt.badge.img+1
  • BikramHeadliner
  • Headliner
  • January 25, 2023

@Satyam_Tope 

 

sdc = sch.data_collectors.get(url='http://12345:18630')
builder = sch.get_pipeline_builder(engine_type='data_collector', engine_id=sdc.id)
jdbc_mt1 = builder.add_stage('JDBC Multitable Consumer')
#jdbc_mt.configuration['tableJdbcConfigBean.tableConfigs'][0]['schema'] = 'EMP'
jdbc_mt1.table_configs[0]['schema'] = 'EMP'
jdbc_mt1.table_configs[0]['tablePattern'] = 'Data'
trash = builder.add_stage('Trash')
jdbc_mt1 >> trash
pipeline = builder.build('Bikrams 2nd pipeline')
sch.publish_pipeline(pipeline, commit_message='fourth commit of Bikrams pipeline')

saleempothiwala
S
2 people like this
  • saleempothiwala
    saleempothiwala
  • S
    Satyam_Tope
S
Fan

Hi @saleempothiwala 

Can you guide me with the steps needed to read the stages from SDK as mentioned in your solution.

Reply


Terms & Conditions