Exactly-Once Semantics
"We need exactly-once delivery" is one of the most frequently requested and most frequently misunderstood requirements in distributed systems. It sounds obvious — process each message exactly once — but it's remarkably hard to achieve, and the cost depends on which layer you want the guarantee at.
The Three Delivery Guarantees
At-Most-Once
Send the message, don't retry. If it gets lost, it's gone.
Producer → Message → Consumer
If consumer crashes: message is lost ✗
When to use: Fire-and-forget metrics, heartbeats, cache invalidation hints. Losing some events is acceptable; re-processing them would be wrong (e.g., sending a notification twice is worse than not sending it).
At-Least-Once
Retry until you get an acknowledgment. The consumer may process the message multiple times.
Producer → Message → Consumer
Consumer crashes before ACK:
Producer → Message (retry) → Consumer
Consumer processes it again ⚠
When to use: Most messaging systems default. The application is responsible for handling duplicates (via idempotency).
Exactly-Once
Process each message exactly once. Neither loss nor duplication.
Producer → Message → Consumer
Consumer crashes before ACK:
System ensures message is re-delivered but effect is applied only once ✓
When to use: Financial transactions, inventory decrements, order processing — any operation where the result of running it twice is different from running it once.
Why It's Hard
Exactly-once requires coordination between three parties: producer, broker, and consumer. Any of them can fail at any time.
Scenario: Payment service publishing "charge $100" to Kafka
1. Payment service sends message to Kafka
2. Kafka broker receives and stores it
3. Network drops before ACK reaches producer
4. Producer doesn't know: did Kafka get the message?
5. Producer retries
6. Now the message is in Kafka twice
7. Consumer processes both: customer charged $200 ✗
The fundamental problem: after a failure, you cannot distinguish "message was received" from "message was received and ACK was lost."
Delivery vs Processing: The Distinction That Resolves the Argument
Engineers argue about exactly-once for years because the two sides are usually talking about different things. One side says "exactly-once is impossible", the other says "Kafka has had exactly-once since 2017". Both are right, because they mean different words.
| Term | What it promises | Achievable? |
|---|---|---|
| Exactly-once delivery | The bytes of a message cross the network to the receiver exactly one time, no matter what fails | No. Not over an unreliable network with crashing processes |
| Exactly-once processing (often called effectively-once) | The observable effect of handling a message is applied exactly once, even if the message itself arrives several times | Yes, inside a boundary you control |
Delivery is about transport. Processing is about state. You give up on the first and build the second out of three ingredients: at-least-once delivery (so nothing is lost), deduplication or idempotency (so repeats do nothing), and atomicity (so "I did the work" and "I recorded that I did the work" can't come apart).
Why exactly-once delivery cannot exist
The argument is short. It is essentially the Two Generals problem applied to a single message.
Suppose a sender wants to deliver message to receiver exactly once, over a network that may drop or delay messages, where either process may crash. Consider what does after sending :
- If never retransmits, then whenever the network drops , it is delivered zero times. That is at-most-once.
- If retransmits until it receives an acknowledgement, then whenever received but the acknowledgement was lost, sends again and receives it twice. That is at-least-once.
- Could wait for some other signal before deciding? Any signal is itself a message that can be lost. From 's point of view, " was lost" and " arrived and the reply was lost" produce exactly the same observations: silence. No finite exchange of messages makes them distinguishable, because the last message of any exchange can be the one that is lost.
So any sender behaviour that never loses a message must sometimes duplicate it, and any behaviour that never duplicates must sometimes lose. The receiver can hide duplicates, but only by remembering what it has already seen. Once it does that, you are doing exactly-once processing, which is the achievable thing.
sequenceDiagram
participant S as Sender
participant N as Network
participant R as Receiver
S->>N: m (attempt 1)
N->>R: m
R->>R: apply effect
R--xS: ack lost
Note over S: Silence. Was m lost, or the ack?
S->>N: m (attempt 2, retry)
N->>R: m again
Note over R: Without a memory of m, effect applied twice
Crashes make it worse
The argument above only needs message loss. Add crashes and it gets harder: the receiver can crash after applying the effect and before recording that it did. On restart it has no memory of at all. This is why the "record that I did it" step must be atomic with the effect itself. A separate write to a dedup table after the effect has the same gap.
Correction (in place)
An earlier version of this page attributed the impossibility of end-to-end exactly-once to "the same reasons as FLP". That is not quite right. The FLP result concerns deterministic consensus in an asynchronous system with one crash. The impossibility of exactly-once delivery is simpler and closer to the Two Generals problem: with lossy links you cannot tell a lost message from a lost acknowledgement. The "Real Answer" section below has been updated to say so.
Where the boundary sits
Exactly-once processing is always scoped to a boundary: a set of state that can be updated atomically together with the record of what has been consumed. Inside the boundary, the system can make "consume input, update state, record progress" a single atomic step. Crossing the boundary (calling an external API, sending an email, writing to a second database) puts you back at at-least-once, and the receiving side must deduplicate.
flowchart LR
subgraph B1["Boundary: one Kafka cluster"]
IN["Input topic"] --> P["Streams app"]
P --> OUT["Output topic"]
P --> OFF["Consumer offsets"]
end
subgraph B2["Boundary: one Postgres database"]
T["Business tables"]
D["Dedup / offset table"]
end
OUT -->|"at-least-once"| C["Sink consumer"]
C -->|"one local transaction"| T
C --> D
C -->|"at-least-once + idempotency key"| E["External payment API"]
Every technique on this page is a way of drawing that boundary so the critical state sits inside it:
| Technique | The boundary it relies on |
|---|---|
| Idempotency key at an API | The server's database holds both the key and the result |
| Dedup table | The consumer's database holds both the processed-ID set and the business update |
| Offsets stored in the sink | The sink database holds both consumer offsets and results |
| Kafka transactions | One Kafka cluster holds output records and consumer offsets |
| Flink checkpoints + 2PC sink | Flink state, source positions and a transactional sink commit together |
| Transactional outbox | The service database holds both business change and the pending event |
Idempotency: The Foundation
An operation is idempotent if applying it multiple times has the same effect as applying it once.
Idempotent:
SET balance = 100 ✓ (running twice still = 100)
DELETE WHERE id=5 ✓ (second delete finds nothing)
NOT idempotent:
balance += 100 ✗ (running twice = +200)
INSERT INTO orders ✗ (running twice = duplicate row)
Making operations idempotent is the most practical path to exactly-once behavior.
Idempotency Keys
Assign every operation a unique ID. The server stores processed IDs. If it sees the same ID again, return the cached result without reprocessing.
POST /payments
{
"idempotency_key": "pay_usr123_order456_1234567890",
"amount": 100,
"currency": "USD"
}
Server behavior:
1. Check: has idempotency_key been processed?
- Yes: return stored result (no re-processing)
- No: process, store result with key, return result
Used by: Stripe (every API call accepts Idempotency-Key), Braintree, Twilio.
Key design:
- Must be unique per operation attempt (not per user or per session)
- Include enough context:
{user_id}_{order_id}_{timestamp}or use UUID - Store with TTL (Stripe keeps keys for 24 hours)
Natural vs Synthetic Idempotency
Some operations are idempotent by their nature; others have to be made idempotent by attaching an identity.
| Kind | Example | How repeats are absorbed |
|---|---|---|
| Natural (state-setting) | SET status = 'shipped', PUT /users/7 {name: "Ana"} | The second application writes the same value |
| Natural (set membership) | Add tag vip to a set | Sets ignore duplicate members |
| Natural (max/min) | "highest offset seen = max(current, 42)" | max is idempotent |
| Synthetic (keyed) | balance += 100 tagged with txn_id | Server rejects a txn_id it already applied |
| Synthetic (versioned) | Update guarded by WHERE version = 7 | Second attempt finds version = 8 and changes nothing |
A useful design habit is to convert deltas into absolute state wherever the domain allows. "Set the order to PAID with payment p_91" survives redelivery. "Add one to the paid-orders counter" does not, unless it carries a key.
Beware operations that look idempotent but are not under reordering. SET status = 'shipped' is idempotent on its own, but if a delayed retry of an older SET status = 'packed' arrives after it, the order moves backwards. Idempotency handles duplicates. It does not handle stale messages. For those you need versions, sequence numbers or a state machine that refuses backwards transitions.
Implementing Idempotency Keys Properly
The three-line description above hides two hard cases: concurrent duplicates (the retry arrives while the first attempt is still running) and crashes mid-request (the first attempt died after doing some work). A robust implementation stores the key before doing the work, with a status.
CREATE TABLE idempotency_keys (
key TEXT PRIMARY KEY,
request_hash TEXT NOT NULL, -- fingerprint of the request body
status TEXT NOT NULL, -- 'in_progress' | 'completed'
response_code INT,
response_body JSONB,
locked_until TIMESTAMPTZ, -- lease for the in-progress attempt
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
// Express-style handler, Postgres via node-postgres
async function createPayment(req: Request, res: Response) {
const key = req.header('Idempotency-Key');
if (!key) return res.status(400).json({ error: 'Idempotency-Key required' });
const hash = sha256(JSON.stringify(req.body));
const client = await pool.connect();
try {
await client.query('BEGIN');
// Claim the key. Exactly one concurrent request can insert it.
const claimed = await client.query(
`INSERT INTO idempotency_keys (key, request_hash, status, locked_until)
VALUES ($1, $2, 'in_progress', now() + interval '30 seconds')
ON CONFLICT (key) DO NOTHING
RETURNING key`, [key, hash]);
if (claimed.rowCount === 0) {
const { rows: [row] } = await client.query(
'SELECT * FROM idempotency_keys WHERE key = $1 FOR UPDATE', [key]);
await client.query('COMMIT');
if (row.request_hash !== hash)
return res.status(422).json({ error: 'key reused with different body' });
if (row.status === 'completed')
return res.status(row.response_code).json(row.response_body); // replay
return res.status(409).json({ error: 'request in progress, retry later' });
}
// Do the work in the SAME transaction as the key record.
const payment = await insertPaymentRow(client, req.body);
const body = { id: payment.id, status: 'succeeded' };
await client.query(
`UPDATE idempotency_keys
SET status = 'completed', response_code = 201, response_body = $2
WHERE key = $1`, [key, body]);
await client.query('COMMIT');
return res.status(201).json(body);
} catch (e) {
await client.query('ROLLBACK'); // key claim disappears with the work
throw e;
} finally {
client.release();
}
}
Points worth noticing:
- Atomicity is the whole trick. Because the key row and the payment row commit together, there is no state where the payment exists but the key does not, or the reverse. If the process crashes mid-transaction, both vanish and the retry starts clean.
- Request fingerprinting. Reusing a key with a different body is a client bug. Stripe returns an error in this case rather than silently replaying the old response, and so should you.
- In-progress handling. A concurrent duplicate gets a
409, not a second execution. The client retries later and receives the stored response. - The external-call problem. If the work includes a call to another system (a card network, an email provider), that call is outside your transaction. You cannot roll it back. The usual answer is to pass your idempotency key (or one derived from it) downstream, and to split the request into recoverable steps, each recorded in the database, so a retry resumes from the last completed step. The billing deep dive at /production-blueprints/billing-engine/idempotency and the API design page at /system-design/api-design/idempotency go further on this.
Who Generates the Key?
The key must be generated by the party that retries, at the moment it first decides to perform the operation, and then reused for every retry of that operation.
| Generated by | Good for | Pitfall |
|---|---|---|
| Client, once per user intent (UUIDv4 on "Pay" click) | Public APIs, mobile apps | Must persist the key locally across app restarts, or a restart mints a new key and a duplicate charge |
Derived from business identity (order_456:charge) | Internal services, one charge per order | Breaks if the business genuinely allows two charges for the same order |
| Upstream message ID (Kafka topic-partition-offset, event UUID) | Consumers | Producer retries before idempotent producers could create two copies with different offsets. Prefer a UUID assigned by the original producer |
| Server, per HTTP request | Nothing useful | A retry is a new request, so it gets a new key. This defeats the purpose |
Note the existing advice above to include a timestamp in the key: that only works if the timestamp is captured once when the operation is first attempted and stored with it. A timestamp read again on each retry produces a different key every time.
The Transactional Outbox Pattern
The most reliable pattern for publishing events from a service without losing them or duplicating them.
Problem: You want to update a database AND publish a message. What if the DB commits but the message publish fails? Or the message publishes but the DB rolls back?
// WRONG — can publish without DB commit, or DB commit without publishing
db.update(order) // step 1
messageQueue.publish() // step 2 — what if this crashes?
Solution: Write the message to an outbox table in the same transaction as the business data. A separate process reads the outbox and publishes.
graph LR
A[Service] -->|1. Same TX| B[(Database)]
B -->|orders table updated| B
B -->|outbox row inserted| B
C[Outbox Relay] -->|2. Poll outbox| B
C -->|3. Publish| D[Message Broker]
C -->|4. Mark sent| B
BEGIN;
UPDATE orders SET status = 'confirmed' WHERE id = 123;
INSERT INTO outbox (id, topic, payload, created_at)
VALUES (gen_random_uuid(), 'order.confirmed',
'{"order_id": 123}', NOW());
COMMIT;
The outbox relay (e.g., Debezium reading the WAL, or a polling job):
- Reads unprocessed outbox rows
- Publishes to Kafka/RabbitMQ
- Marks rows as published (or deletes them)
Guarantees: At-least-once delivery to the broker. The broker must be idempotent or the consumer must deduplicate.
Used by: Any service doing event-driven architecture correctly. Debezium + CDC (Change Data Capture) is the most common implementation.
Why the Naive Approaches Fail
There are four orderings of "commit DB" and "publish event", and every one without an outbox has a failure window:
| Approach | Crash point | Result |
|---|---|---|
| Commit, then publish | After commit, before publish | DB says confirmed, no event. Downstream never learns. Lost event |
| Publish, then commit | After publish, before commit (or commit fails) | Event says confirmed, DB rolled back. Phantom event |
| Publish inside the open transaction | Publish succeeds, commit fails | Same phantom event. The broker is not part of your DB transaction |
| Distributed 2PC (XA) across DB and broker | Coordinator crash | Works in theory. Most brokers (Kafka included) do not support XA, and 2PC blocks on coordinator failure. See distributed transactions |
The outbox replaces the cross-system atomicity problem with a single-database transaction (cheap, well understood) plus an at-least-once copy step (easy to retry).
A Polling Relay
CREATE TABLE outbox (
id BIGSERIAL PRIMARY KEY, -- monotonically increasing, gives an order
event_id UUID NOT NULL UNIQUE, -- stable ID consumers dedupe on
aggregate_id TEXT NOT NULL, -- becomes the Kafka key, keeps per-entity order
topic TEXT NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;
# Relay loop. Run one instance per outbox, or many with SKIP LOCKED.
def relay_once(conn, producer, batch=500):
with conn.transaction():
rows = conn.execute("""
SELECT id, event_id, aggregate_id, topic, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT %s
FOR UPDATE SKIP LOCKED
""", (batch,)).fetchall()
for r in rows:
producer.produce(r.topic, key=r.aggregate_id, value=r.payload,
headers={"event_id": str(r.event_id)})
producer.flush() # wait for broker acks (acks=all) before marking
conn.execute("UPDATE outbox SET published_at = now() WHERE id = ANY(%s)",
([r.id for r in rows],))
If the relay crashes after flush() but before the UPDATE commits, the same rows are published again on restart. That is the at-least-once gap, and it is why every outbox event carries a stable event_id that consumers dedupe on. An idempotent Kafka producer does not close this gap: a restarted relay is a new producer session, so the broker sees the resend as new records.
Two ordering caveats:
- With
SKIP LOCKEDand several relay workers, events for the sameaggregate_idcan be published out of order. If per-entity order matters, partition relay work byaggregate_idor run a single relay. BIGSERIALorder is allocation order, not commit order. A transaction that took ID 101 can commit after one that took ID 102. A relay that remembers "last ID published = 102" would skip 101 forever. Selecting onpublished_at IS NULL(as above) rather than on a high-water mark avoids that.
CDC Relay (Debezium)
Instead of polling, a change-data-capture connector tails the database's replication log (Postgres logical decoding, MySQL binlog) and turns each committed outbox insert into a Kafka record. Debezium ships an "outbox event router" transform for exactly this. Advantages: no polling load, events arrive in commit order, and the outbox rows can be deleted immediately after insert (the log still carries them). The connector stores its log position in Kafka and, after a crash, resumes from the last recorded position, so it can republish a few events. Downstream dedup is still required. See Kafka Connect and the full pattern page at /architecture-patterns/event-driven/transactional-outbox.
The Inbox: The Outbox's Mirror Image
The outbox guarantees an event leaves exactly once in effect. The inbox guarantees it is applied exactly once in effect. The consumer writes the incoming event_id into an inbox (dedup) table in the same transaction as its business change:
sequenceDiagram
participant OS as Order Service DB
participant R as Relay
participant K as Kafka
participant C as Billing Consumer
participant BD as Billing DB
OS->>OS: TX - update order + insert outbox row (event_id E1)
R->>OS: read unpublished rows
R->>K: publish E1
R--xOS: crash before marking published
R->>K: publish E1 again after restart
K->>C: E1
C->>BD: TX - insert inbox E1 + create invoice
K->>C: E1 (duplicate)
C->>BD: TX - insert inbox E1 conflicts, roll back, skip
Note over C,BD: Invoice created once
Outbox on the producer side plus inbox on the consumer side is the standard recipe for exactly-once effect between two services that each own a database. Neither needs a distributed transaction.
Kafka Exactly-Once
Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017). It covers:
1. Idempotent Producer
Prevents duplicates between producer and broker caused by retries.
Producer sends: msg(seq=1, data=X)
Network drop before ACK
Producer retries: msg(seq=1, data=X)
Broker deduplicates: "already saw seq=1 from this producer" → ignores ✓
Each producer gets a PID (Producer ID). Each message has a sequence number. The broker maintains state per (PID, partition) and deduplicates.
Enable:
props.put("enable.idempotence", "true");
How the idempotent producer works
- On startup the producer asks the cluster for a producer ID and an epoch.
- For every partition it writes to, it numbers record batches with a sequence number starting at 0.
- The partition leader remembers, per producer ID, the sequence numbers of the most recent batches it appended (the last five). A batch whose sequence is already there is acknowledged as success without being appended again. A batch whose sequence skips ahead indicates lost data and is rejected with an out-of-order sequence error.
- This state is written into the log with the batches, so it survives leader failover: the new leader rebuilds it from the log.
Requirements and defaults:
| Setting | Required value | Why |
|---|---|---|
acks | all | A batch acknowledged by only the leader could vanish in failover, breaking the sequence |
retries | > 0 | Retrying is the point. Duplicates are now safe |
max.in.flight.requests.per.connection | ≤ 5 | The broker tracks the last five batches per producer and partition. Ordering is preserved even with retries |
Since Apache Kafka 3.0, enable.idempotence=true and acks=all are the producer defaults, so modern clients get duplicate-free retries unless they opt out or set conflicting options.
What it does not cover
The idempotent producer only removes duplicates caused by the producer's own internal retries within one session. It does not help when:
- The application calls
send()twice for the same logical event (for example, a relay that re-reads an outbox row after restarting). The second call gets a new sequence number and is a new record. - The producer process restarts. Without a
transactional.id, a restarted producer gets a new producer ID, and the broker has no way to link it to the old one. - Writes span multiple partitions and you need them all-or-nothing. That is what transactions add.
2. Transactions
Allows atomically writing to multiple partitions and topics.
producer.initTransactions();
producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", orderId, "confirmed"));
producer.send(new ProducerRecord<>("notifications", userId, "order_confirmed"));
producer.commitTransaction(); // atomic — either both land or neither
Transaction coordinator: A special Kafka broker component that manages transaction state using a __transaction_state internal topic.
The transaction protocol, step by step
Every transactional producer is configured with a transactional.id, a stable name that survives restarts (for example payments-enricher-partition-3). The coordinator for that ID is the leader of the __transaction_state partition the ID hashes to.
sequenceDiagram
participant P as Producer
participant TC as Transaction Coordinator
participant L as Partition Leaders
participant GC as Group Coordinator
P->>TC: InitProducerId (transactional.id)
TC->>TC: bump epoch, abort or finish any open txn from older epoch
TC-->>P: producer id + epoch
P->>TC: AddPartitionsToTxn (orders-0, notifications-2)
TC->>TC: log state Ongoing with partition list
P->>L: produce records (pid, epoch, seq)
P->>TC: AddOffsetsToTxn (consumer group)
P->>GC: TxnOffsetCommit (input offsets)
P->>TC: EndTxn COMMIT
TC->>TC: log PrepareCommit
TC->>L: write COMMIT markers
TC->>GC: write COMMIT marker on offsets partition
TC->>TC: log CompleteCommit
TC-->>P: committed
What each piece achieves:
InitProducerIdwith a bumped epoch fences zombies. If an old instance of the sametransactional.idis still alive (say it was stuck in a GC pause and its replacement has started), its epoch is now stale, and every broker rejects its writes with a fencing error. This is the same idea as a fencing token, with the epoch playing the token's role.- Registering partitions with the coordinator before writing tells it where to write commit or abort markers later.
- Records are written to the real partitions immediately, not buffered. They are simply not visible to
read_committedconsumers yet. - Offsets go through the transaction too.
sendOffsetsToTransactionturns "I consumed up to offset N" into part of the same atomic unit as the output records. This is what makes read-process-write exactly-once. - Commit is a two-phase commit run by the coordinator. Once
PrepareCommitis durably logged in__transaction_state, the transaction will commit, even if the coordinator crashes: the new coordinator reads the log and finishes writing markers. That logged decision is the commit point.
Readers: isolation.level
Consumers choose what they see:
isolation.level | Sees |
|---|---|
read_uncommitted (default) | Every record, including those from transactions later aborted |
read_committed | Only committed transactional records (and non-transactional ones). Aborted records are filtered out |
A read_committed consumer cannot read past the last stable offset (LSO): the offset of the first record belonging to a still-open transaction. A transaction that stays open for a long time therefore stalls every read_committed consumer of that partition, even for records from other producers that have committed. This is the main operational cost of transactions, and why transaction.timeout.ms matters (the coordinator aborts transactions that run longer).
Forgetting read_committed
The most common EOS bug is a downstream consumer left at the default read_uncommitted. Aborted records, including the partial output of a crashed transaction, are then processed as if real. Exactly-once on the write side is undone on the read side.
A full read-process-write loop with the plain clients
Kafka Streams does all this for you, but the raw version shows what is happening:
Properties pp = new Properties();
pp.put("bootstrap.servers", "kafka:9092");
pp.put("transactional.id", "enricher-" + instanceId); // stable across restarts
pp.put("enable.idempotence", "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(pp, ser, ser);
Properties cp = new Properties();
cp.put("bootstrap.servers", "kafka:9092");
cp.put("group.id", "enricher");
cp.put("enable.auto.commit", "false"); // offsets go via the txn
cp.put("isolation.level", "read_committed"); // don't read aborted input
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(cp, de, de);
producer.initTransactions(); // fences older instances
consumer.subscribe(List.of("payments"));
while (true) {
ConsumerRecords<String, String> batch = consumer.poll(Duration.ofMillis(200));
if (batch.isEmpty()) continue;
producer.beginTransaction();
try {
for (ConsumerRecord<String, String> r : batch) {
producer.send(new ProducerRecord<>("payments-enriched", r.key(), enrich(r.value())));
}
producer.sendOffsetsToTransaction(nextOffsets(batch), consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException e) {
producer.close(); // a newer instance owns this transactional.id, stop
throw e;
} catch (KafkaException e) {
producer.abortTransaction();
rewindToLastCommitted(consumer); // re-read the batch, its output was discarded
}
}
nextOffsets must return, for each partition, the offset of the last processed record plus one (Kafka commits the position of the next record to read, not the last one read). Off-by-one here reprocesses one record per partition after every restart.
Passing consumer.groupMetadata() (rather than just a group ID string) is the KIP-447 improvement. Before it, fencing during consumer-group rebalances needed one transactional producer per input partition. With it, a single producer per application instance is enough, because the group coordinator can fence stale offset commits using the consumer group generation. Kafka Streams exposes this as exactly_once_v2 (introduced as exactly_once_beta in 2.5, renamed in 3.0).
3. Exactly-Once in Kafka Streams
The full end-to-end guarantee: read-process-write atomically.
Consumer reads msg from input topic
Consumer processes (transforms, enriches)
Producer writes to output topic
Consumer commits offset
All four steps are atomic — either all happen or none
props.put("processing.guarantee", "exactly_once_v2");
Cost: Lower throughput and higher latency than at-least-once. Only guarantees exactly-once within the Kafka ecosystem — downstream systems (databases, external APIs) need their own idempotency.
Correction (in place)
This line previously stated a flat "~20% throughput reduction". The overhead is not a fixed number. It depends mostly on how often transactions commit. The design document for KIP-98 reported only a few percent throughput loss for transactional producers committing every 100 ms with 1 KB messages, because the per-transaction cost (coordinator round trips and markers) is amortised over many records. Kafka Streams commits every 100 ms by default under EOS (versus 30 s under at-least-once), so the effective cost and the added end-to-end latency (records become visible to read_committed readers only at commit) grow as you shorten the commit interval. Measure with your own record sizes and commit interval.
What Kafka EOS covers, and what it does not
| Covered | Not covered |
|---|---|
| Duplicate appends from producer retries | An application calling send() twice for the same event |
| Atomic writes across many partitions and topics | Writes to anything outside Kafka (databases, HTTP calls, emails) |
| Atomic "consume offsets + produce outputs" | Side effects inside your processing function (a call to an API during enrich() runs once per attempt) |
Fencing of zombie instances with the same transactional.id | Two instances configured with different IDs for the same work |
| State stores in Kafka Streams (changelogs are written in the transaction) | Consumers reading with read_uncommitted |
The third row in the right column is the one that surprises people. If your Streams processor calls a REST service on every record, an aborted and retried transaction calls it again. Kafka can roll back its own records, not someone else's database.
Flink: Checkpoints and Two-Phase-Commit Sinks
Apache Flink takes a different route to the same goal. Instead of a transaction per batch of records, it takes consistent snapshots of the entire running job at intervals and, on failure, rewinds everything to the last snapshot.
Exactly-once state via checkpoints
Flink's checkpointing is a variant of the Chandy-Lamport snapshot algorithm, adapted for dataflow graphs (Flink's papers call it asynchronous barrier snapshotting).
- The JobManager's checkpoint coordinator injects a barrier with checkpoint ID into every source.
- Each source records its position (for Kafka, the offset per partition) as part of checkpoint , then emits the barrier downstream, in-line with records.
- When an operator with several inputs receives barrier on one input, it waits (aligns) until barrier has arrived on all inputs, buffering records that arrive after the barrier on the faster inputs. Then it snapshots its state and forwards the barrier.
- When every sink has acknowledged barrier , checkpoint is complete: a consistent cut through the whole job, where every operator's state reflects exactly the records before the barrier.
sequenceDiagram
participant JM as Checkpoint Coordinator
participant S as Kafka Source
participant O as Stateful Operator
participant K as Sink
participant ST as Durable Storage
JM->>S: trigger checkpoint 42
S->>ST: save offsets for checkpoint 42
S->>O: barrier 42 (in the data stream)
O->>ST: snapshot state for checkpoint 42
O->>K: barrier 42
K->>ST: snapshot sink state
K-->>JM: ack 42
O-->>JM: ack 42
S-->>JM: ack 42
JM->>JM: checkpoint 42 complete
JM->>K: notify checkpoint 42 complete
On failure, Flink restarts the job, restores every operator's state from the last completed checkpoint, and resets the sources to the offsets saved in it. Records after the checkpoint are re-read and re-processed. Internal state ends up as if each record was processed once, because the effects of the replayed records were never part of the restored state.
Note what that means: records are processed more than once; state reflects them once. That is the "processing vs delivery" distinction again. Flink also offers unaligned checkpoints (barriers overtake buffered records, which are saved as part of the checkpoint) to keep checkpoints fast under backpressure, still with exactly-once state.
The sink problem
Replaying records is harmless for internal state, but a sink that wrote to the outside world between the last checkpoint and the crash has already emitted output for the replayed records. After the rewind those outputs are emitted again. Exactly-once state does not give exactly-once output unless the sink cooperates. There are two ways:
| Sink strategy | How | Example |
|---|---|---|
| Idempotent sink | Writes are upserts keyed by a deterministic key, so replays overwrite with identical values | Key-value stores, Elasticsearch with document IDs, JDBC upserts |
| Transactional (two-phase-commit) sink | Output since the last checkpoint goes into an open external transaction that commits only when the checkpoint completes | Kafka sink with transactions, file sinks that rename on commit |
Two-phase-commit sinks
Flink's two-phase-commit sink pattern ties an external transaction to the checkpoint lifecycle. The Flink checkpoint is the prepare phase. Notification of checkpoint completion is the commit phase.
sequenceDiagram
participant F as Flink Sink Task
participant JM as Checkpoint Coordinator
participant K as Kafka
F->>K: begin txn T7, write records
JM->>F: barrier for checkpoint 42
F->>K: flush T7 (pre-commit)
F->>F: store T7 id in checkpoint 42 state
F->>K: begin txn T8 for new records
F-->>JM: ack checkpoint 42
JM->>JM: all tasks acked, checkpoint 42 complete
JM->>F: notifyCheckpointComplete 42
F->>K: commit T7
Note over F,K: Crash after 42 completes but before commit? Recovery restores 42, which names T7, and commits it
The lifecycle methods (from the classic TwoPhaseCommitSinkFunction; the newer Sink V2 API has the same shape with a writer and a committer):
| Method | When | Must do |
|---|---|---|
beginTransaction | Start, and after each pre-commit | Open a fresh external transaction |
preCommit | Checkpoint barrier reaches the sink | Flush everything, so that commit cannot fail for lack of data. Then stash the transaction handle in checkpoint state |
commit | Checkpoint is complete | Commit. Must be idempotent, because after a crash Flink may call it again for a transaction that already committed |
abort | Failure before the checkpoint completed | Roll back. Output for these records will be produced again on replay |
Why this is safe: an external transaction commits only after the checkpoint that contains its handle is durable. If Flink crashes before checkpoint completion, it restores the previous checkpoint and aborts the open transaction; the records are replayed into a new transaction. If it crashes after completion but before commit finished, the restored checkpoint names the transaction, and recovery commits it.
Configuring Flink → Kafka exactly-once
KafkaSink<String> sink = KafkaSink.<String>builder()
.setBootstrapServers("kafka:9092")
.setRecordSerializer(KafkaRecordSerializationSchema.builder()
.setTopic("orders-enriched")
.setValueSerializationSchema(new SimpleStringSchema())
.build())
.setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
.setTransactionalIdPrefix("orders-enricher") // must be unique per job on the cluster
.setProperty("transaction.timeout.ms", "900000") // 15 min, see warning below
.build();
env.enableCheckpointing(60_000, CheckpointingMode.EXACTLY_ONCE);
Transaction timeout vs checkpoint interval
The external transaction stays open from one checkpoint until the next one completes, plus however long recovery takes. If the broker aborts it first (Kafka's broker cap transaction.max.timeout.ms defaults to 15 minutes), the output is lost: Flink will try to commit a transaction Kafka has already discarded. Set the producer's transaction.timeout.ms well above "checkpoint interval + worst checkpoint duration + expected downtime", and no higher than the broker cap (raise the cap if needed). The opposite mistake also bites: the Kafka client's own default transaction.timeout.ms is 60 seconds, shorter than many checkpoint intervals. The legacy FlinkKafkaProducer defaulted to one hour, which exceeds the default broker cap and is rejected unless one of the two is changed.
Two more consequences:
- Output latency equals the checkpoint interval. Downstream
read_committedconsumers see nothing until the checkpoint completes. A 60-second checkpoint interval means up to 60 seconds of delay plus checkpoint time. Exactly-once trades latency for correctness. - Downstream consumers must use
read_committed, as with any Kafka transaction.
Kafka Streams vs Flink, side by side
| Aspect | Kafka Streams EOS | Flink EOS |
|---|---|---|
| Unit of atomicity | One transaction per task per commit interval | One global checkpoint across the whole job |
| State durability | Changelog topics written inside the transaction | Snapshots in durable storage (S3, HDFS) |
| Recovery | Each task restores its state from changelog, resumes from committed offsets | Whole job (or affected region) rewinds to last checkpoint |
| Default commit cadence under EOS | 100 ms | Whatever checkpoint interval you set (often tens of seconds) |
| Sources and sinks | Kafka only | Any source that can replay, any sink that is idempotent or transactional |
| Deployment | A library inside your app | A cluster (JobManager plus TaskManagers) |
For more on stream processors, see Kafka Streams and exactly-once processing in stream pipelines.
Exactly-Once Across Different Layers
| Layer | Mechanism | Cost |
|---|---|---|
| Producer → Broker | Idempotent producer (sequence numbers) | Low |
| Broker durability | Replication + acks=all | Medium |
| Consumer → Application | Idempotency keys + deduplication | Medium |
| Application → Database | Transactional outbox or DB transactions | Medium |
| End-to-end (Kafka Streams) | Transactions + atomic offset commits | High |
| Cross-service | Saga pattern + compensating transactions | Very high |
Practical Patterns
Deduplication Table
CREATE TABLE processed_events (
event_id UUID PRIMARY KEY,
processed_at TIMESTAMP DEFAULT NOW()
);
-- On message receive:
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING *;
-- If nothing returned: already processed, skip
-- If row returned: process and commit atomically
The comment "process and commit atomically" carries all the weight. The insert and the business update must be in one transaction, and the broker acknowledgement (offset commit) must come after that transaction commits:
def handle(msg, conn):
with conn.transaction(): # BEGIN ... COMMIT
fresh = conn.execute(
"INSERT INTO processed_events (event_id) VALUES (%s) "
"ON CONFLICT (event_id) DO NOTHING RETURNING event_id",
(msg.headers["event_id"],)).fetchone()
if fresh:
conn.execute("UPDATE accounts SET balance = balance + %s WHERE id = %s",
(msg.amount, msg.account_id))
consumer.commit(msg) # ack only after the DB commit
Walk through every crash point:
| Crash between | On restart | Effect count |
|---|---|---|
Receive and BEGIN | Message redelivered, not in table, processed | 1 |
| Inside the transaction | Transaction rolled back, message redelivered, processed | 1 |
| DB commit and offset commit | Message redelivered, insert conflicts, skipped | 1 |
| After offset commit | Not redelivered | 1 |
Swap the order (ack first, then DB) and the third row becomes "message never redelivered, DB never updated": a lost message. Split the dedup insert and the business update into two transactions and a crash between them gives either a skipped effect or a doubled one, depending on which you do first.
How long to remember
A dedup table that grows forever eventually becomes a problem. Choose a retention window longer than the maximum redelivery horizon: the longest time a duplicate can possibly show up after the original.
| Duplicate source | Typical horizon |
|---|---|
| Producer retries | Seconds to minutes |
| Consumer crash and rebalance | Minutes |
| Outbox relay restart | Minutes |
| Operator replays a topic from an old offset | Days, or "the topic's retention" |
| Client app retries after being offline | Hours to days |
Deduplication past the window is not guaranteed, which is a deliberate, documented trade-off. Partition the table by day (PARTITION BY RANGE (processed_at)) so expiry is a cheap DROP PARTITION rather than a slow DELETE.
Store Offsets With the Results
When the consumer's output lives in a single transactional database, there is a cleaner option than a dedup table: store the consumer's input offsets in the same database, in the same transaction as the results, and ignore the broker's offset storage.
CREATE TABLE consumer_offsets (
consumer_group TEXT,
topic TEXT,
partition INT,
next_offset BIGINT NOT NULL,
PRIMARY KEY (consumer_group, topic, partition)
);
def on_partitions_assigned(consumer, partitions):
for tp in partitions: # resume from the DB, not from Kafka
row = db.fetchone("SELECT next_offset FROM consumer_offsets "
"WHERE consumer_group=%s AND topic=%s AND partition=%s",
(GROUP, tp.topic, tp.partition))
consumer.seek(tp, row.next_offset if row else 0)
def process_batch(records):
with db.transaction():
for r in records:
apply_business_change(r)
for (topic, part), last in last_offset_per_partition(records).items():
db.execute("""INSERT INTO consumer_offsets VALUES (%s,%s,%s,%s)
ON CONFLICT (consumer_group, topic, partition)
DO UPDATE SET next_offset = EXCLUDED.next_offset""",
(GROUP, topic, part, last + 1))
The database is the single source of truth for both "what I have applied" and "where I am in the input", and they can never disagree. No per-message dedup table is needed, because the database never re-applies an offset it has already recorded. The trade-offs: duplicates that originate upstream (two records with the same event at different offsets) are not caught, and the approach only works when every effect goes to that one database. Kafka Connect sinks such as the JDBC sink and many Flink sinks use variants of this idea.
Two consumers, one partition
During a consumer-group rebalance, an old consumer that has not yet noticed it lost a partition can still be finishing a batch while the new owner starts on the same records. With offsets in the database, guard the offset update so it only moves forward and only from the expected position:
UPDATE consumer_offsets SET next_offset = $new
WHERE consumer_group = $g AND topic = $t AND partition = $p
AND next_offset = $expected; -- 0 rows: someone else advanced it, roll back
This turns the offset row into a compare-and-swap that fences the stale consumer.
Conditional Update (Compare-and-Swap)
-- Only apply if we haven't already applied this version
UPDATE orders
SET status = 'shipped', version = version + 1
WHERE id = $1 AND version = $expected_version;
-- If 0 rows affected: already processed (version mismatch) → skip
Fencing with Tokens
1. Acquire distributed lock → get token T
2. Perform work with token T
3. Write result with condition: "only if T is not older than the highest token the store has seen"
4. Release lock
If lock expires and another process gets token T+1:
Old process's write (with token T) is rejected ✓
(Correction: this step previously read "only if current token is T". The store does not usually know which token is "current". It only knows the highest token it has accepted, and rejects anything lower. See leases and fencing.)
Fencing prevents a stale worker from applying an effect after its lease ran out. It is complementary to deduplication, which prevents the same message from being applied twice. Exactly-once pipelines with leased partition ownership usually need both. Kafka's producer epoch and consumer-group generation are fencing tokens of exactly this kind.
Deterministic Processing
Replay-based schemes (Kafka Streams, Flink, offsets-in-DB) assume that reprocessing the same input yields the same output. Code that reads the wall clock, generates random IDs, or calls a service whose answer changes breaks that assumption: after a replay the output differs from what the aborted attempt produced. Because the aborted attempt's output was discarded, this is usually harmless for correctness, but it matters when output IDs are used for dedup further downstream. Derive IDs from input (event_id + ":out:" + n) rather than from UUID.randomUUID(), and take timestamps from the record, not from now().
The Real Answer
True end-to-end exactly-once delivery across network boundaries is theoretically impossible in the general case: with lossy links and crashing processes, a sender cannot tell a lost message from a lost acknowledgement (the Two Generals problem, see above). This is a different result from FLP, which is about reaching consensus. What systems provide is:
- Idempotent operations — safe to retry, same result
- Deduplication — detect and discard duplicates
- Atomic transactions — within a single system boundary
When someone says "exactly-once," they usually mean "at-least-once delivery with idempotent processing." That's achievable, practical, and what you should build. True "the effect happens exactly once regardless of failures" requires a transaction coordinator and is only practical within a single system boundary (like Kafka Streams).
End-to-End Worked Examples
Example 1: A mobile "Pay" button to a card processor
Goal: the customer is charged once per tap of "Pay", despite flaky mobile networks, app restarts and server crashes.
sequenceDiagram
participant App as Mobile App
participant API as Payments API
participant DB as Payments DB
participant PSP as Card Processor
App->>App: generate key K1, persist locally
App->>API: POST /charges, Idempotency-Key K1
API->>DB: TX - claim K1 (in_progress), insert charge row (pending)
API->>PSP: authorize, idempotency key derived from K1
PSP-->>API: approved
API->>DB: TX - charge = succeeded, K1 = completed with response
API--xApp: response lost (tunnel)
App->>API: POST /charges, Idempotency-Key K1 (retry)
API->>DB: K1 completed, read stored response
API-->>App: 201 (replayed), no new charge
Walk the failure points:
| Failure | What happens | Charged |
|---|---|---|
| Request never reaches API | App retries with K1, first real attempt | Once |
| API crashes after claiming K1, before calling the processor | Transaction never committed, K1 not claimed, retry starts clean | Once |
| API crashes after the processor approved, before recording it | K1 is stuck in_progress with a pending charge. A recovery job (or the next retry after the lease expires) re-calls the processor with the same derived key, and the processor returns the original approval rather than charging again | Once |
| Response lost on the way back | Retry with K1 returns the stored response | Once |
| App killed and restarted | K1 was persisted before the first send, so the retry reuses it | Once |
| App generates a new key on restart (bug) | Server sees a brand-new operation | Twice |
Note that the first commit in the diagram must be committed before the processor call, not held open across it. Holding a database transaction open across a network call to a third party pins a connection and row locks for an unbounded time. The "pending row, then external call, then update" sequence is a tiny saga, each step durable, each step resumable.
The last row is the lesson. Every layer did its job except the one that owned the key. The idempotency key has to be minted by whoever retries, and stored before the first attempt.
Example 2: Kafka topic into Postgres, with an email at the end
Goal: for each OrderPlaced event, insert an invoice row exactly once and send the customer one email.
- The order service writes the order and an outbox row in one transaction. Debezium publishes the outbox row to
orders.eventswith headerevent_id = E. (At-least-once into Kafka.) - The invoicing consumer reads
orders.eventsand, in one Postgres transaction, insertsEintoinbox, inserts the invoice, and inserts anemail_outboxrow keyed byE. Then it commits its Kafka offset. (Exactly-once effect in the invoicing DB.) - An email worker reads
email_outbox, calls the email provider passingEas the provider's idempotency key (or dedupe ID), and marks the row sent. (At-least-once to the provider, deduplicated by the provider if it supports keys.)
If the email provider does not support idempotency keys, step 3 is at-least-once and the customer may, rarely, get two emails: when the worker crashes after the provider accepted the send and before the row was marked. You cannot close that gap from your side. You can only shrink it (mark immediately after the call, keep the call fast) and decide that a rare duplicate email is acceptable. Many teams decide exactly that, and it is a reasonable decision as long as it is a conscious one.
flowchart LR
A["Order DB + outbox"] -->|"CDC, at-least-once"| K["orders.events"]
K -->|"at-least-once"| C["Invoicing consumer"]
C -->|"one TX: inbox + invoice + email_outbox"| D["Invoicing DB"]
D -->|"poll"| W["Email worker"]
W -->|"key = event_id"| E["Email provider"]
Example 3: Flink job, Kafka to Kafka, with a stateful count
Goal: count purchases per product per minute and publish the counts, exactly once.
- Source: Kafka
purchases, offsets saved in Flink checkpoints (not committed to the consumer group for correctness). - Operator: keyed tumbling window, counts in RocksDB state, snapshotted every 30 seconds.
- Sink:
KafkaSinkwithEXACTLY_ONCEandtransaction.timeout.ms = 900000. - Downstream dashboard consumer:
isolation.level=read_committed.
Failure at second 47 after checkpoint 12 completed at second 30:
- TaskManager dies. Flink restarts tasks and restores state and source offsets from checkpoint 12.
- Window counts revert to their value at second 30. Purchases from seconds 30 to 47 are re-read from Kafka and counted again, from the restored state, so each counts once.
- The sink's open transaction (holding output written after checkpoint 12) is aborted. Its records never become visible to
read_committedreaders. - Output is recomputed and written into a new transaction, which commits when checkpoint 13 completes.
The dashboard sees each per-minute count once, about 30 seconds later than it would under at-least-once. If the dashboard consumer used read_uncommitted, it would have seen the aborted partial output too, and double-counted.
Failure Scenarios Catalogue
| # | Scenario | Symptom | Root cause | Fix |
|---|---|---|---|---|
| 1 | Producer retry after lost ack | Duplicate records in topic | Non-idempotent producer | enable.idempotence=true (default since Kafka 3.0) |
| 2 | Zombie instance after GC pause | Two instances writing the same output | No fencing | Stable transactional.id. The newer instance's initTransactions() fences the old one |
| 3 | Consumer crashes after DB write, before offset commit | Message processed twice | Ack and effect not atomic | Dedup table or offsets in the same DB transaction |
| 4 | Offset committed before processing | Messages silently lost on crash | Auto-commit or ack-first | Disable auto-commit, commit after the effect |
| 5 | Downstream reads read_uncommitted | Aborted output appears, counts double | Reader isolation default | isolation.level=read_committed |
| 6 | Long-running transaction | All read_committed consumers stall | LSO pinned by open transaction | Shorter transactions, sensible transaction.timeout.ms |
| 7 | Flink transaction aborted by broker during long outage | Output for a checkpoint lost | Txn timeout below recovery time | Raise transaction.timeout.ms and broker cap |
| 8 | Dedup TTL shorter than replay horizon | Duplicates after an operator replays old data | Retention too short | Size the window to the real horizon, document it |
| 9 | Side effect inside a stream processor | External API called on every retry | EOS covers Kafka state only | Move side effects to a sink with its own idempotency |
| 10 | Client mints a new idempotency key per retry | Double charges | Key generated at the wrong layer | Generate once per intent, persist before first send |
| 11 | Outbox relay with high-water-mark polling | Some events never published | Sequence allocation order differs from commit order | Poll on published_at IS NULL, or use CDC |
| 12 | Same idempotency key reused with different body | Wrong response replayed | Client bug | Fingerprint requests, reject mismatches with an error |
War Story (illustrative)
A ledger team moved its balance updates onto Kafka Streams with exactly_once_v2 and declared victory. Two weeks later, finance found a few hundred customers with notification emails sent twice and one partner bank that had received duplicate settlement instructions. The Streams topology was correct: every balance in the output topics was right. The problem was a mapValues step that called the partner bank's HTTP API directly "to keep latency low". Each time a task aborted a transaction (during rebalances after deploys), the batch was reprocessed and the HTTP call was made again. Kafka rolled back its own records. It had no way to roll back a bank's. The fix was to emit "settlement requested" records to a topic, and have a separate sink service call the bank with the record's event ID as the bank's idempotency reference, recording each acknowledged call in its own database.
Real Numbers
These are orders of magnitude to reason with, not benchmarks. Measure your own setup.
| Mechanism | Added cost per operation | Notes |
|---|---|---|
| Idempotent producer | Negligible | Sequence tracking is per batch, not per record. It is on by default in current Kafka |
| Kafka transaction | A few extra round trips per transaction (coordinator, markers) | Amortised over the batch. Overhead shrinks as commit interval grows |
| Kafka Streams EOS | Latency ≈ commit interval (100 ms default) | Output becomes visible to read_committed readers only at commit |
| Flink 2PC sink | Latency ≈ checkpoint interval plus checkpoint duration | 10 s to a few minutes is typical |
| Dedup table insert (Postgres, indexed UUID) | One extra index insert per message, in an existing transaction | Usually sub-millisecond. Watch table size and bloat |
| Idempotency-key API lookup | One indexed read (and on first call, an insert) | Small next to the work itself |
| Outbox via CDC | Write amplification of one row per event, plus replication-log reading | Removes the dual-write window entirely |
The broad rule: per-record costs are cheap, and the expensive part of exactly-once is the commit cadence, which trades latency for overhead.
Decision Guide
flowchart TD
Q1{"Is a duplicate effect harmful?"} -->|"No (metrics, cache hints)"| AL["At-least-once, no dedup"]
Q1 -->|"Yes"| Q2{"Can the operation be made naturally idempotent?"}
Q2 -->|"Yes (upsert, set state)"| NI["At-least-once + idempotent writes"]
Q2 -->|"No"| Q3{"Is everything inside one system?"}
Q3 -->|"Kafka to Kafka"| KS["Kafka transactions or Streams exactly_once_v2"]
Q3 -->|"Kafka to one database"| DB["Dedup table or offsets in the same DB TX"]
Q3 -->|"Stateful stream job, many sinks"| FL["Flink checkpoints + 2PC or idempotent sinks"]
Q3 -->|"Crosses services or third parties"| XS["Outbox + inbox + idempotency keys at every hop"]
| Situation | Recommended approach | Avoid |
|---|---|---|
| Public payments API | Client idempotency keys, stored atomically with the result | Server-generated keys |
| Service publishing events after DB changes | Transactional outbox with CDC | Dual writes |
| Consumer updating its own DB | Inbox or dedup table in the same transaction, ack after commit | Auto-commit offsets |
| Kafka-to-Kafka transformation | Kafka Streams exactly_once_v2 | Hand-rolled consumer plus producer without transactions |
| Windowed aggregations with external sinks | Flink with transactional or idempotent sinks | Non-idempotent JDBC inserts |
| Notifications, emails, SMS | At-least-once with provider-side dedup keys, accept rare duplicates | Pretending it is exactly-once |
| Cross-service business process | Sagas with idempotent steps and compensations | Distributed 2PC across services |
Related reading: idempotent consumers, the message-queue view of exactly-once, Kafka internals, and timeouts, retries and backoff for the retry side of the story.
Interview Questions
Q1. Is exactly-once delivery possible? Not over an unreliable network with failures: a sender cannot distinguish a lost message from a lost acknowledgement, so it must either risk loss (no retry) or risk duplication (retry). What is achievable is exactly-once processing: at-least-once delivery plus deduplication or idempotency, made atomic with the effect inside some boundary.
Q2. What does Kafka's idempotent producer guarantee, and what doesn't it?
It removes duplicates caused by the producer's own retries within a session, per partition, using producer ID and sequence numbers. It does not dedupe application-level double sends, does not survive a restart without a transactional.id, and gives no cross-partition atomicity.
Q3. Explain how Kafka transactions make read-process-write exactly-once.
The output records and the input offsets are committed in one transaction coordinated by the transaction coordinator, which uses a logged two-phase commit and writes commit markers to every involved partition. Readers with read_committed only see committed data. A stable transactional.id with epochs fences zombie instances.
Q4. Your Kafka Streams app is configured for exactly-once, yet a downstream service was called twice. Why? The call was a side effect inside the processor. EOS covers Kafka records and state stores only. An aborted transaction (from a rebalance or crash) replays the batch, and the side effect runs again. Move it to a sink that uses an idempotency key.
Q5. How does Flink achieve exactly-once state, and why is that not enough for exactly-once output? Periodic barrier-based snapshots of all operator state plus source offsets. On failure it restores and replays. Records are processed again, but state reflects them once. External outputs written since the last checkpoint would be emitted twice, so the sink must be idempotent or transactional (two-phase commit tied to checkpoint completion).
Q6. In the dedup-table pattern, in what order do you commit the DB transaction and the broker ack, and why? DB first, ack second. A crash between them causes a redelivery that the dedup table absorbs. The reverse order loses the message on a crash between them.
Q7. Why is the outbox pattern needed? Why not publish to Kafka inside the DB transaction? The broker is not part of the database transaction. If the publish succeeds and the commit fails, you have a phantom event. If you publish after commit, a crash loses the event. The outbox puts the event in the same local transaction and relays it at-least-once.
Q8. How long should a dedup table keep keys? Longer than the longest time a duplicate can appear: producer retries, rebalances, relay restarts, and any operational replay. Document the horizon, because duplicates beyond it are not caught.
Q9. Design a Flink job's Kafka sink for exactly-once. What configuration bites people?
DeliveryGuarantee.EXACTLY_ONCE, a unique transactional ID prefix, read_committed downstream, and a transaction timeout larger than checkpoint interval plus recovery time but within the broker's transaction.max.timeout.ms. Too short, and the broker aborts transactions Flink still intends to commit.
Q10. What is the difference between idempotency and fencing? Idempotency makes applying the same operation twice harmless. Fencing makes operations from a stale owner (whose lease or epoch has expired) be rejected. A pipeline with leased partition ownership usually needs both: one for duplicates, the other for zombies.
Q11. When is "offsets stored in the sink database" better than a dedup table? When all effects go to one transactional database: it needs no per-message table and can never disagree with itself. It does not catch duplicates introduced upstream (the same event at two offsets), so it pairs well with an idempotent or transactional producer upstream.
Q12. A product manager asks for "exactly-once email delivery". What do you say? You can guarantee that the system requests each email exactly once in effect, and you can pass a dedup key to the provider if it supports one. If the provider has no idempotency support, a crash between the provider accepting the email and you recording it can produce a duplicate. You can make it rare. You cannot make it impossible. Usually a rare duplicate email is acceptable, and saying so explicitly is better than promising something no system can deliver.