Skip to main content

Can I guarantee record order when writing to Kinesis?

  • January 10, 2022
  • 0 replies
  • 215 views

AkshayJadhav
StreamSets Employee
Forum|alt.badge.img

Question: 

Could I guarantee record order when writing to Kinesis ? Yes/no.

 

Answer:

The Kinesis Producer Library (KPL) we use from Amazon sends requests asynchronously. We've added an option called "Preserve Record Order" that will ensure that each record is flushed to Kinesis before we write the next one. Unfortunately, this means that you'll get an ordering guarantee but at a pretty severe throughput penalty.

Additionally, be mindful that for a stream with more than 1 shard, the partitioner you select will also affect where each record is placed.

Please see this issue on the Kinesis Producer Library (KPL) Github for additional information.

Did this topic help you find an answer to your question?

0 replies

Be the first to reply!

Reply