When a kerberos-related issue is suspected, especially in relation to expired tickets, it is helpful to check the state of the kerberos renewal thread that runs as part of the Data Collector.
The kerberos renewal thread is a thread that is started upon SDC start which periodically does a Kerberos re-login operation to keep SDC's kerberos ticket from expiring. The time interval of the re-logins are based on the lifetime of the ticket, and it is done at a frequency to ensure the ticket never expires.
At each attempted re-login, the renewal thread will log the activity in the sdc.log, so that you can keep track of these re-logins. The sdc.log will contain the following lines:
<timestamp> [user:] [pipeline:] [runner:] [thread:Kerberos-Renewal-Thread-<principal>] INFO SecurityContext - Attempting re-login
<timestamp> [user:] [pipeline:] [runner:] [thread:Kerberos-Renewal-Thread-<principal>] INFO SecurityContext - Login, principal '<principal>'As long as you see these re-login attempts, the renewal thread is running and executing the re-logins.
In some scenarios, such as a temporary network issue or KDC-related failure, the re-login may fail. When this happens, the renewal thread will also stop. The sdc.log will contain the following line in this situation:
ERROR SecurityContext - Stopping renewal thread because of exception: <exception>The exception that caused the re-login to fail will be printed (<exception>). In this situation, you will need to restart SDC to get the renewal thread up again. Also, make sure to rectify the environmental issue that had caused the re-login to fail and consequently, the renewal thread to die.
