Issue:
Pipeline with Kafka consumer origin is running but not showing any records ingested, even though the topic contains data.
Solution:
By default, Kafka consumer has the 'auto.offset.reset' parameter set to 'latest'. This means when there is no prior offset saved or the offset is out of range for the consumer group, the consumer group will start from the latest offset. Thus, the consumer will start at the latest offset, and it will only ingest new data that has come in after.
If you would like to ingest from the earliest available offset, in the Kafka origin, add 'auto.offset.reset=earliest' in the 'Kafka Configuration' section. This will make the consumer group specified begin from the earliest offset. However, keep in mind that if you had already started a consumer group from 'latest' already and want to switch to 'earliest', you will need to also specify a new consumer group.
This is also covered in our documentation here.
NOTE:
In Kafka version 0.9.0 or later, options are 'latest' / 'earliest' as mentioned above.
In Kafka versions before 0.9.0, options are 'largest' / 'smallest' instead.
