After upgrade to 3.3~3.4.2, cluster streaming pipeline fails with 'Caused by: java.lang.SecurityException: java.io.IOException: ${SDC_CONF}/ldap-login.conf (No such file or directory)'.


Userlevel 4
Badge

Issue:

After upgrading Data Collector to 3.3~3.4.2, cluster streaming pipelines pointing to a non-kerberized Kafka endpoint fails with the following exception:

Caused by: java.lang.SecurityException: java.io.IOException: ${SDC_CONF}/ldap-login.conf (No such file or directory)

 

Versions affected:

StreamSets Data Collector 3.3~3.4.2.  The issue has been resolved in the 3.4.3 minor release (3.4.3+), and 3.5 major release (3.5+).

 

Solution:

This issue will occur if your Data Collector is kerberized, but the Kafka endpoint is non-kerberized.  As a workaround, obtain the $SDC_CONF/ldap-login.conf file from your Data Collector install and distribute it to the same known path on your gateway Data Collector and all YARN worker nodes.  Afterwards, include the following Java runtime parameter in the sdc environment file:

-Djava.security.auth.login.config=<path_to_ldap-login.conf>

For example, for a non-Cloudera managed Data Collector instance, append the following to the sdc-env.sh/sdcd-env.sh file:

export SDC_JAVA_OPTS="${SDC_JAVA_OPTS} -Djava.security.auth.login.config=<path_to_ldap-login.conf>"

For a Cloudera-managed instance, you may add just the parameter in 'java opts' or add the above export line in the sdc-env.sh safety valve.

 

Additional notes:  As of 3.3+, the cluster streaming pipelines can now communicate with a kerberized Kafka endpoint.  If you have upgraded to 3.3+ and are switching over to a kerberized Kafka endpoint for your cluster streaming pipelines, then it is expected to ensure that the relevant JAAS file containing the 'KafkaClient' kerberos login information exists on each YARN worker node.


0 replies

Be the first to reply!

Reply