ONLY FOR AGENTS!
We noticed in our code that when there are multiple HDFS destinations in one pipeline, we actually create multiple connections to HDFS (one for each) since we use a private classloader for each destination. This could, in theory, cause the replay issues we are seeing.
WARN UserGroupInformation - PriviledgedActionException ... (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2017-01-04 23:01:55,812 [user:*admin] [pipeline:...] [thread:LeaseRenewer:...] WARN Client - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2017-01-04 23:01:55,813 [user:*admin] [pipeline:...] [thread:LeaseRenewer:...] WARN UserGroupInformation - PriviledgedActionException as:... (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2017-01-04 23:01:55,814 [user:*admin] [pipeline:...] [thread:LeaseRenewer:...] WARN LeaseRenewer - Failed to renew lease for [DFSClient_NONMAPREDUCE_1591117506_208] for 30 seconds. Will retry shortly ...
Explanation of the issue:
Kerberos is designed in a way that it is expected that a process multiplexes all its logical authenticated connections with a server over a single physical authenticated connection. Hadoop client does this, but it does it within the scope of a client space (the classloader). Because SDC is designed to support multiple (versions of) Hadoop clients, each one has its own private classloader. Most of the time, this own private classloader is unnecessary for multiple Hadoop clients of the same version of Hadoop, but for simplicity/generality that it is how it was implemented in SDC. In the patch we provided you, we introduced a flag to ensure that all Hadoop clients talking to the same version of Hadoop use the same classloader.
Fix available from SDC 2.5.0.0
The user has to add to SDC_JAVA_OPTS the following property to enable the change:
Dcom.streamsets.pipeline.stage.destination.hdfs.HdfsDTarget.no.private.classloader
This is not documented so the users wouldn't get confused and use this property in some wrong cases. For example, this shouldn't be used in the case when more HDFS targets within one pipeline/SDC are writing to different clusters.
