Skip to main content
Solved

PostgreSQL Metadata processor does not accept preconfigured connection?

P
Opening Band

Hi, I am new to StreamSets so I may miss a thing here, if this is the case please apologize!

Wanted to configure the PostgreSQL metadata processor and found that the only way to connect is to directly configure the JDBC connection string and credentials. There seems no way to use a connection previously configured, like when setting up a JDBC (multitable) consumer or JDBC Producer.

Is this a defect/missing feature or do I misunderstand something?

 

Thank you in advance!

Best answer by Pradeep

Hi @pedromanuel, you are correct. Connection can not be used in Postgresql Metadata processor currently. Will be raising a feature request for the same. Lookout for release notes for future releases. I guess you are already updating connection details manually and please continue to use this as a temporary workaround.

View original
    Did this topic help you find an answer to your question?

    4 replies

    Pradeep
    StreamSets Employee
    Forum|alt.badge.img+1
    • PradeepStreamSets Employee
    • StreamSets Employee
    • 48 replies
    • August 9, 2022

    @pedromanuel If you are designing the pipelines through StreamSet’s control-hub or dataops platform, you can utilize Connections feature for the same. Refer https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/Connections/GettingStarted.html#concept_fxv_rp5_4mb for more details.

    P
    Opening Band
    • pedromanuelOpening Band
    • Author
    • Opening Band
    • 12 replies
    • August 10, 2022

    Hi @Pradeep , thank you for your reply! This was exactly what I was trying to do and it is working fine for origins (like the JDBC Multitable Consumer) or destinations (like the JDBC Producer). But for the JDBC metadata processor, which is supposed to use the exact same connection as the JDBC producer, there is no option to select a Connection, i can only fill out the connection details like JDBC url, credentials etc. For clarity, I append two screen shots, one for the producer (a destination) where I can select a connection and one for the processor, where I only can specify the JDBC details but not a connection...

     

     

    Does this makes sense?

    Pradeep
    StreamSets Employee
    Forum|alt.badge.img+1
    • PradeepStreamSets Employee
    • StreamSets Employee
    • 48 replies
    • Answer
    • August 10, 2022

    Hi @pedromanuel, you are correct. Connection can not be used in Postgresql Metadata processor currently. Will be raising a feature request for the same. Lookout for release notes for future releases. I guess you are already updating connection details manually and please continue to use this as a temporary workaround.

    P
    1 person likes this
    • P
      pedromanuel
    P
    Opening Band
    • pedromanuelOpening Band
    • Author
    • Opening Band
    • 12 replies
    • August 11, 2022

    Thank you @Pradeep !

    Reply


    Terms & Conditions