A Detailed Guide to Idempotency, Delivery Semantics, and Deduplication
What happens when a service sends a request to charge a customer, but the request times out with no response? The burning question is whether the charge went through. Or should it be retried?
Two different things could have happened. The charge succeeded, and the confirmation was lost on the way back, or the request never reached the payment service at all. Both possibilities produce identical outputs, which makes it difficult to figure out what happened and the next action to be taken. Retrying risks charging the customer twice. On the other hand, declining to retry risks never charging them at all.
Idempotency is the property that makes the retry safe. An operation is idempotent when applying it more than once produces the same state as applying it once. For example, setting an account balance to 500 is idempotent, because even the tenth time to execute this operation, the outcome will be the same. In contrast, adding 500 to a balance is not idempotent, since every time it is executed, the balance amount changes. Most operations that matter in a business system resemble the second one.
In this article, we will look at the following topics in detail:
The three different delivery semantics available for developers.
The three points where duplicates enter a producer, broker, and consumer path, and why a fix at one point does nothing for the other two
The difference between an operation that is idempotent by nature and an endpoint engineered to behave that way
What does an idempotency key need to work, and how can it fail?
Why every deduplication scheme has a time limit, and what the guarantee is worth once that limit passes
What “exactly-once” means in real-world systems, and where each guarantee ends?
