Sometimes we notice kafka consumer is fetching more number of records than in the origin. Let us take following assumptions to understand the issue.
- You have 250 records in a kafka topic and assuming there are no new events getting streamed to this topic. Batch size of kafka consumer is also set to 250.
- Kafka consumer in the pipeline is setup with a consumer group to read from the topic in (1)
- Destination is HTTP Client which makes calls to configured API end-points to consume the relevant kafka data. Destination could be anything and not just HTTP Client.
- You observe kafka consumerās record count is increasing by 250 every few mins. Eg: 500, 750, 1000 etc.
Troubleshooting:
First thing we need to verify is "Stage Batch Processing Timer (in seconds)" and note how much time each stage is taking to process.
If you see the destination or processor configured next to Kafka consumer origin is taking more than 90% time of the pipeline run(90% is just a random threshold. you can say affected stage is taking significant processing time compared to the origin) then it is likely that destination or processor is taking longer time to process the origin records and if the records are not consumed in set timeout of kafka it will send the same batch of records again.
Solution:
To fix the issue, below configs can be set according to the max processing time of the destination.
1) max.poll.interval.ms
2) session.timeout.ms: While setting this please make sure the value is within the range of broker config group.min.session.timeout.ms and group.max.session.timeout.ms
We would also recommend to go through https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html to understand more about these configurations.
