Issue
A pipeline fails due to a Kafka Stage with Kerberos authentication enabled is throwing the stage exception KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: xxxx
Background
There are two general types of errors which can happen which prevent a user from connecting to a secured endpoint: Authentication and Authorization.
- Authentication errors - have to do with a credential failures, i.e. a Kerberos ticket is unable to be granted due to the wrong credentials being used, denying the principal from obtaining a service ticket.
- Authorization errors - occur after the user has made it passed authentication, but there is some additional Access Control Logic (ACL) which is preventing this user from accessing some resource (i.e. missing permissions to access a certain Kafka Consumer group).
Resolution
The GroupAuthorizationException signifies that the Kafka consumer has successfully authenticated and obtained a service ticket to access the Kafka broker, but this principal is not authorized to access some Kafka resource (in this case the consumer group).
Since the authentication is succeeding, this narrows down the issue since it’s not between StreamSets and Kerberos, but rather something on the Kafka ACL-side which needs to grant this user permission to the consumer group.
Further reading regarding Kafka ACL Authorization:
https://developer.confluent.io/learn-kafka/security/authorization/