Skip to main content

Hi, I am new to StreamSets so I may miss a thing here, if this is the case please apologize!

Wanted to configure the PostgreSQL metadata processor and found that the only way to connect is to directly configure the JDBC connection string and credentials. There seems no way to use a connection previously configured, like when setting up a JDBC (multitable) consumer or JDBC Producer.

Is this a defect/missing feature or do I misunderstand something?

 

Thank you in advance!

@pedromanuel If you are designing the pipelines through StreamSet’s control-hub or dataops platform, you can utilize Connections feature for the same. Refer https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/Connections/GettingStarted.html#concept_fxv_rp5_4mb for more details.

Hi @Pradeep , thank you for your reply! This was exactly what I was trying to do and it is working fine for origins (like the JDBC Multitable Consumer) or destinations (like the JDBC Producer). But for the JDBC metadata processor, which is supposed to use the exact same connection as the JDBC producer, there is no option to select a Connection, i can only fill out the connection details like JDBC url, credentials etc. For clarity, I append two screen shots, one for the producer (a destination) where I can select a connection and one for the processor, where I only can specify the JDBC details but not a connection...

 

 

Does this makes sense?

Hi @pedromanuel, you are correct. Connection can not be used in Postgresql Metadata processor currently. Will be raising a feature request for the same. Lookout for release notes for future releases. I guess you are already updating connection details manually and please continue to use this as a temporary workaround.

Thank you @Pradeep !

Reply


Terms & Conditions