The Problem
Message queues are foundational infrastructure, but building one that's both fast and reliable is notoriously difficult. Here's how we approached it.
Exactly-Once Semantics
Most distributed systems can only guarantee at-least-once delivery. Messages might be delivered multiple times during network partitions or node failures.
We needed exactly-once semantics for financial transactions. Charging a customer twice is not acceptable.
Our Solution: Idempotency Keys
Every message in our system carries a unique idempotency key. Consumers track which keys they've processed in a distributed set with a 7-day TTL.
When a message arrives, the consumer checks if its key exists in the set. If yes, the message is acknowledged but not reprocessed.