Issue:
When running a pipeline with Hadoop stages using impersonation, the pipeline fails. The application logs may show connection exceptions similar to the following repeating:
INFO client.ConfiguredRMFailoverProxyProvider: Failing over to <rm_1>
INFO retry.RetryInvocationHandler: Exception while invoking getNewApplication of class ApplicationClientProtocolPBClientImpl over <rm_1> after XX fail over attempts. Trying to fail over immediately.
INFO client.ConfiguredRMFailoverProxyProvider: Failing over to <rm_2>
WARN ipc.Client: Failed to connect to server: <rm_server>: retries get failed due to exceeded maximum allowed retries number: 0
java.net.ConnectException: Connection refused Solution:
It is important to first check if it is a general network connection issue or not. Try to run telnet from the gateway Data Collector host to the YARN ResourceManager host and port. If the connection is fine, then look at the ResourceManger logs around the time of the connection exceptions. Check to see if there is an exception similar to the following:
INFO org.apache.hadoop.ipc.Server: Connection from <IP>:<PORT> for protocol org.apache.hadoop.yarn.api.ApplicationClientProtocolPB is unauthorized for user <proxy_user> (auth:PROXY) via <sdc_user> (auth:KERBEROS)If you see an exception such as the above in the ResourceManager logs, it is because the impersonation privilege settings for the SDC user are not set on the Hadoop cluster. You will need to set the following properties on the Hadoop cluster:
hadoop.proxyuser.<sdc_user>.groups = <groups>
hadoop.proxyuser.<sdc_user>.users = <users>
hadoop.proxyuser.<sdc_user>.hosts = <hosts>Afterwards, make sure to synchronize the Hadoop configuration files on the Data Collector host.
