Skip to main content

I have migrated from 3.18.2 to 3.19.0.

I'm trying to connect to Confluent Kafka on Streamsets version 3.19.0 I keep receiving this error 
KAFKA_14 - Security properties should be specified in the Security tab, not in the Kafka properties

In the Security Tab there does not seem to be an option to use the sasl.jaas.config API-Key approach. Everything seems to require a truststore etc. 

 

When using 3.18.2 Streamsets I was able to use the Kafka Tab / Kafka Config for all the connection info.

 

Example: 

p
    {
        "key": "ssl.endpoint.identification.algorithm",
        "value": "https"
    },
    {
        "key": "sasl.mechanism",
        "value": "PLAIN"
    },
    {
        "key": "request.timeout.ms",
        "value": "20000"
    },
    {
        "key": "retry.backoff.ms",
        "value": "500"
    },
    {
        "key": "sasl.jaas.config",
        "value": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxxxxxxxxxxxxxx\" password=\"xxxxxxxxxxxxxxxxxxxxx\";"
    },
    {
        "key": "security.protocol",
        "value": "SASL_SSL"
    },
    {
        "key": "basic.auth.credentials.source",
        "value": "USER_INFO"
    },
    {
        "key": "acks",
        "value": "all"
    },
    {
        "key": "connections.max.idle.ms",
        "value": "60000"
    },
    {
        "key": "metadata.max.age.ms",
        "value": "60000"
    },
    {
        "key": "client.id",
        "value": "dev.ea.mytestapp.arvo.18"
    }
]

Reply all

Reply to author

Forward

Hi @miketod09 

 

In SDC release 3.19, StreamSets released the following functionality:

The Kafka Consumer, Kafka Multitopic Consumer, and Kafka Producer stages include a Security tab used to configure one of the following options to connect securely to Kafka:

  • SSL/TLS encryption
  • SSL/TLS encryption and authentication
  • Kerberos authentication
  • Kerberos authentication on SSL/TLS

So if you were using any other security properties then it would give you an error to say you have to select the property from the new tab rather than use kafka properties manually.

 

From release 4.0, StreamSets added a new option in Kafka Advanced properties

Kafka stages - Kafka stages include an Override Stage Configurations property that enables the Kafka properties defined in the stage to override other stage properties.

So if you upgrade to 4.0, you will be able to override the security properties with your own ones.

 

Hope it helps!

Reply


Terms & Conditions