Issue:
Kudu pipeline failing with "delaying RPC due to Illegal state: Not the leader" exception:
KUDU_03 - Errors while interacting with Kudu: Row error for primary key="...", tablet=null, server=null, status=Runtime error: can not complete before timeout: KuduRpc(method=GetTableLocations, tablet=Kudu Master, attempt=..., DeadlineTracker(timeout=..., elapsed=...), Traces: [0ms] sending RPC to server , [10ms] delaying RPC due to Illegal state: Not the leader. Local UUID: ..., Raft Consensus state: current_term: ... leader_uuid: "..." ....... (see attached for full version)
Solution:
The exception seems to be related to the KUDU-2343 issue.
This issue appears to be happening when the Kudu Master elections taking place between the master nodes. Kudu client (StreamSets) connection to the old Master persists - which is met with the appropriate "delaying RPC due to Illegal state: Not the leader" error message, since the application is attempting to connect to the old Master which is now a follower.
Currently, the only available libraries with the fix are Kudu client libraries 1.7.
