On this page

Exactly-Once Semantics

"We need exactly-once delivery" is one of the most frequently requested and most frequently misunderstood requirements in distributed systems. It sounds obvious — process each message exactly once — but it's remarkably hard to achieve, and the cost depends on which layer you want the guarantee at.


The Three Delivery Guarantees

At-Most-Once

Send the message, don't retry. If it gets lost, it's gone.

Producer → Message → Consumer
If consumer crashes: message is lost ✗

When to use: Fire-and-forget metrics, heartbeats, cache invalidation hints. Losing some events is acceptable; re-processing them would be wrong (e.g., sending a notification twice is worse than not sending it).

At-Least-Once

Retry until you get an acknowledgment. The consumer may process the message multiple times.

Producer → Message → Consumer
Consumer crashes before ACK:
Producer → Message (retry) → Consumer
Consumer processes it again ⚠

When to use: Most messaging systems default. The application is responsible for handling duplicates (via idempotency).

Exactly-Once

Process each message exactly once. Neither loss nor duplication.

Producer → Message → Consumer
Consumer crashes before ACK:
System ensures message is re-delivered but effect is applied only once ✓

When to use: Financial transactions, inventory decrements, order processing — any operation where the result of running it twice is different from running it once.


Why It's Hard

Exactly-once requires coordination between three parties: producer, broker, and consumer. Any of them can fail at any time.

Scenario: Payment service publishing "charge $100" to Kafka

1. Payment service sends message to Kafka
2. Kafka broker receives and stores it
3. Network drops before ACK reaches producer
4. Producer doesn't know: did Kafka get the message?
5. Producer retries
6. Now the message is in Kafka twice
7. Consumer processes both: customer charged $200 ✗

The fundamental problem: after a failure, you cannot distinguish "message was received" from "message was received and ACK was lost."


Delivery vs Processing: The Distinction That Resolves the Argument

Engineers argue about exactly-once for years because the two sides are usually talking about different things. One side says "exactly-once is impossible", the other says "Kafka has had exactly-once since 2017". Both are right, because they mean different words.

TermWhat it promisesAchievable?
Exactly-once deliveryThe bytes of a message cross the network to the receiver exactly one time, no matter what failsNo. Not over an unreliable network with crashing processes
Exactly-once processing (often called effectively-once)The observable effect of handling a message is applied exactly once, even if the message itself arrives several timesYes, inside a boundary you control

Delivery is about transport. Processing is about state. You give up on the first and build the second out of three ingredients: at-least-once delivery (so nothing is lost), deduplication or idempotency (so repeats do nothing), and atomicity (so "I did the work" and "I recorded that I did the work" can't come apart).

Why exactly-once delivery cannot exist

The argument is short. It is essentially the Two Generals problem applied to a single message.

Suppose a sender wants to deliver message to receiver exactly once, over a network that may drop or delay messages, where either process may crash. Consider what does after sending :

  1. If never retransmits, then whenever the network drops , it is delivered zero times. That is at-most-once.
  2. If retransmits until it receives an acknowledgement, then whenever received but the acknowledgement was lost, sends again and receives it twice. That is at-least-once.
  3. Could wait for some other signal before deciding? Any signal is itself a message that can be lost. From 's point of view, " was lost" and " arrived and the reply was lost" produce exactly the same observations: silence. No finite exchange of messages makes them distinguishable, because the last message of any exchange can be the one that is lost.

So any sender behaviour that never loses a message must sometimes duplicate it, and any behaviour that never duplicates must sometimes lose. The receiver can hide duplicates, but only by remembering what it has already seen. Once it does that, you are doing exactly-once processing, which is the achievable thing.

sequenceDiagram
    participant S as Sender
    participant N as Network
    participant R as Receiver
    S->>N: m (attempt 1)
    N->>R: m
    R->>R: apply effect
    R--xS: ack lost
    Note over S: Silence. Was m lost, or the ack?
    S->>N: m (attempt 2, retry)
    N->>R: m again
    Note over R: Without a memory of m, effect applied twice

Crashes make it worse

The argument above only needs message loss. Add crashes and it gets harder: the receiver can crash after applying the effect and before recording that it did. On restart it has no memory of at all. This is why the "record that I did it" step must be atomic with the effect itself. A separate write to a dedup table after the effect has the same gap.

Correction (in place)

An earlier version of this page attributed the impossibility of end-to-end exactly-once to "the same reasons as FLP". That is not quite right. The FLP result concerns deterministic consensus in an asynchronous system with one crash. The impossibility of exactly-once delivery is simpler and closer to the Two Generals problem: with lossy links you cannot tell a lost message from a lost acknowledgement. The "Real Answer" section below has been updated to say so.

Where the boundary sits

Exactly-once processing is always scoped to a boundary: a set of state that can be updated atomically together with the record of what has been consumed. Inside the boundary, the system can make "consume input, update state, record progress" a single atomic step. Crossing the boundary (calling an external API, sending an email, writing to a second database) puts you back at at-least-once, and the receiving side must deduplicate.

flowchart LR
    subgraph B1["Boundary: one Kafka cluster"]
        IN["Input topic"] --> P["Streams app"]
        P --> OUT["Output topic"]
        P --> OFF["Consumer offsets"]
    end
    subgraph B2["Boundary: one Postgres database"]
        T["Business tables"]
        D["Dedup / offset table"]
    end
    OUT -->|"at-least-once"| C["Sink consumer"]
    C -->|"one local transaction"| T
    C --> D
    C -->|"at-least-once + idempotency key"| E["External payment API"]

Every technique on this page is a way of drawing that boundary so the critical state sits inside it:

TechniqueThe boundary it relies on
Idempotency key at an APIThe server's database holds both the key and the result
Dedup tableThe consumer's database holds both the processed-ID set and the business update
Offsets stored in the sinkThe sink database holds both consumer offsets and results
Kafka transactionsOne Kafka cluster holds output records and consumer offsets
Flink checkpoints + 2PC sinkFlink state, source positions and a transactional sink commit together
Transactional outboxThe service database holds both business change and the pending event

Idempotency: The Foundation

An operation is idempotent if applying it multiple times has the same effect as applying it once.

Idempotent:
  SET balance = 100   ✓  (running twice still = 100)
  DELETE WHERE id=5   ✓  (second delete finds nothing)

NOT idempotent:
  balance += 100      ✗  (running twice = +200)
  INSERT INTO orders  ✗  (running twice = duplicate row)

Making operations idempotent is the most practical path to exactly-once behavior.

Idempotency Keys

Assign every operation a unique ID. The server stores processed IDs. If it sees the same ID again, return the cached result without reprocessing.

POST /payments
{
  "idempotency_key": "pay_usr123_order456_1234567890",
  "amount": 100,
  "currency": "USD"
}

Server behavior:

1. Check: has idempotency_key been processed?
   - Yes: return stored result (no re-processing)
   - No: process, store result with key, return result

Used by: Stripe (every API call accepts Idempotency-Key), Braintree, Twilio.

Key design:

  • Must be unique per operation attempt (not per user or per session)
  • Include enough context: {user_id}_{order_id}_{timestamp} or use UUID
  • Store with TTL (Stripe keeps keys for 24 hours)

Natural vs Synthetic Idempotency

Some operations are idempotent by their nature; others have to be made idempotent by attaching an identity.

KindExampleHow repeats are absorbed
Natural (state-setting)SET status = 'shipped', PUT /users/7 {name: "Ana"}The second application writes the same value
Natural (set membership)Add tag vip to a setSets ignore duplicate members
Natural (max/min)"highest offset seen = max(current, 42)"max is idempotent
Synthetic (keyed)balance += 100 tagged with txn_idServer rejects a txn_id it already applied
Synthetic (versioned)Update guarded by WHERE version = 7Second attempt finds version = 8 and changes nothing

A useful design habit is to convert deltas into absolute state wherever the domain allows. "Set the order to PAID with payment p_91" survives redelivery. "Add one to the paid-orders counter" does not, unless it carries a key.

Beware operations that look idempotent but are not under reordering. SET status = 'shipped' is idempotent on its own, but if a delayed retry of an older SET status = 'packed' arrives after it, the order moves backwards. Idempotency handles duplicates. It does not handle stale messages. For those you need versions, sequence numbers or a state machine that refuses backwards transitions.

Implementing Idempotency Keys Properly

The three-line description above hides two hard cases: concurrent duplicates (the retry arrives while the first attempt is still running) and crashes mid-request (the first attempt died after doing some work). A robust implementation stores the key before doing the work, with a status.

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  request_hash  TEXT        NOT NULL,          -- fingerprint of the request body
  status        TEXT        NOT NULL,          -- 'in_progress' | 'completed'
  response_code INT,
  response_body JSONB,
  locked_until  TIMESTAMPTZ,                   -- lease for the in-progress attempt
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
// Express-style handler, Postgres via node-postgres
async function createPayment(req: Request, res: Response) {
  const key = req.header('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key required' });
  const hash = sha256(JSON.stringify(req.body));

  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    // Claim the key. Exactly one concurrent request can insert it.
    const claimed = await client.query(
      `INSERT INTO idempotency_keys (key, request_hash, status, locked_until)
       VALUES ($1, $2, 'in_progress', now() + interval '30 seconds')
       ON CONFLICT (key) DO NOTHING
       RETURNING key`, [key, hash]);

    if (claimed.rowCount === 0) {
      const { rows: [row] } = await client.query(
        'SELECT * FROM idempotency_keys WHERE key = $1 FOR UPDATE', [key]);
      await client.query('COMMIT');
      if (row.request_hash !== hash)
        return res.status(422).json({ error: 'key reused with different body' });
      if (row.status === 'completed')
        return res.status(row.response_code).json(row.response_body); // replay
      return res.status(409).json({ error: 'request in progress, retry later' });
    }

    // Do the work in the SAME transaction as the key record.
    const payment = await insertPaymentRow(client, req.body);
    const body = { id: payment.id, status: 'succeeded' };
    await client.query(
      `UPDATE idempotency_keys
       SET status = 'completed', response_code = 201, response_body = $2
       WHERE key = $1`, [key, body]);
    await client.query('COMMIT');
    return res.status(201).json(body);
  } catch (e) {
    await client.query('ROLLBACK');   // key claim disappears with the work
    throw e;
  } finally {
    client.release();
  }
}

Points worth noticing:

  • Atomicity is the whole trick. Because the key row and the payment row commit together, there is no state where the payment exists but the key does not, or the reverse. If the process crashes mid-transaction, both vanish and the retry starts clean.
  • Request fingerprinting. Reusing a key with a different body is a client bug. Stripe returns an error in this case rather than silently replaying the old response, and so should you.
  • In-progress handling. A concurrent duplicate gets a 409, not a second execution. The client retries later and receives the stored response.
  • The external-call problem. If the work includes a call to another system (a card network, an email provider), that call is outside your transaction. You cannot roll it back. The usual answer is to pass your idempotency key (or one derived from it) downstream, and to split the request into recoverable steps, each recorded in the database, so a retry resumes from the last completed step. The billing deep dive at /production-blueprints/billing-engine/idempotency and the API design page at /system-design/api-design/idempotency go further on this.

Who Generates the Key?

The key must be generated by the party that retries, at the moment it first decides to perform the operation, and then reused for every retry of that operation.

Generated byGood forPitfall
Client, once per user intent (UUIDv4 on "Pay" click)Public APIs, mobile appsMust persist the key locally across app restarts, or a restart mints a new key and a duplicate charge
Derived from business identity (order_456:charge)Internal services, one charge per orderBreaks if the business genuinely allows two charges for the same order
Upstream message ID (Kafka topic-partition-offset, event UUID)ConsumersProducer retries before idempotent producers could create two copies with different offsets. Prefer a UUID assigned by the original producer
Server, per HTTP requestNothing usefulA retry is a new request, so it gets a new key. This defeats the purpose

Note the existing advice above to include a timestamp in the key: that only works if the timestamp is captured once when the operation is first attempted and stored with it. A timestamp read again on each retry produces a different key every time.

A crash before the ACK makes at-least-once delivery hand you the same message twice. You can't stop the redelivery. What you can do is have the consumer check the idempotency key and skip work it has already done, so the side effect happens once.

The Transactional Outbox Pattern

The most reliable pattern for publishing events from a service without losing them or duplicating them.

Problem: You want to update a database AND publish a message. What if the DB commits but the message publish fails? Or the message publishes but the DB rolls back?

// WRONG — can publish without DB commit, or DB commit without publishing
db.update(order)        // step 1
messageQueue.publish()  // step 2 — what if this crashes?

Solution: Write the message to an outbox table in the same transaction as the business data. A separate process reads the outbox and publishes.

graph LR
    A[Service] -->|1. Same TX| B[(Database)]
    B -->|orders table updated| B
    B -->|outbox row inserted| B
    C[Outbox Relay] -->|2. Poll outbox| B
    C -->|3. Publish| D[Message Broker]
    C -->|4. Mark sent| B
BEGIN;
  UPDATE orders SET status = 'confirmed' WHERE id = 123;
  INSERT INTO outbox (id, topic, payload, created_at)
    VALUES (gen_random_uuid(), 'order.confirmed',
            '{"order_id": 123}', NOW());
COMMIT;

The outbox relay (e.g., Debezium reading the WAL, or a polling job):

  1. Reads unprocessed outbox rows
  2. Publishes to Kafka/RabbitMQ
  3. Marks rows as published (or deletes them)

Guarantees: At-least-once delivery to the broker. The broker must be idempotent or the consumer must deduplicate.

Used by: Any service doing event-driven architecture correctly. Debezium + CDC (Change Data Capture) is the most common implementation.

Why the Naive Approaches Fail

There are four orderings of "commit DB" and "publish event", and every one without an outbox has a failure window:

ApproachCrash pointResult
Commit, then publishAfter commit, before publishDB says confirmed, no event. Downstream never learns. Lost event
Publish, then commitAfter publish, before commit (or commit fails)Event says confirmed, DB rolled back. Phantom event
Publish inside the open transactionPublish succeeds, commit failsSame phantom event. The broker is not part of your DB transaction
Distributed 2PC (XA) across DB and brokerCoordinator crashWorks in theory. Most brokers (Kafka included) do not support XA, and 2PC blocks on coordinator failure. See distributed transactions

The outbox replaces the cross-system atomicity problem with a single-database transaction (cheap, well understood) plus an at-least-once copy step (easy to retry).

A Polling Relay

CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,         -- monotonically increasing, gives an order
  event_id     UUID        NOT NULL UNIQUE,   -- stable ID consumers dedupe on
  aggregate_id TEXT        NOT NULL,          -- becomes the Kafka key, keeps per-entity order
  topic        TEXT        NOT NULL,
  payload      JSONB       NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;
# Relay loop. Run one instance per outbox, or many with SKIP LOCKED.
def relay_once(conn, producer, batch=500):
    with conn.transaction():
        rows = conn.execute("""
            SELECT id, event_id, aggregate_id, topic, payload
            FROM outbox
            WHERE published_at IS NULL
            ORDER BY id
            LIMIT %s
            FOR UPDATE SKIP LOCKED
        """, (batch,)).fetchall()
        for r in rows:
            producer.produce(r.topic, key=r.aggregate_id, value=r.payload,
                             headers={"event_id": str(r.event_id)})
        producer.flush()          # wait for broker acks (acks=all) before marking
        conn.execute("UPDATE outbox SET published_at = now() WHERE id = ANY(%s)",
                     ([r.id for r in rows],))

If the relay crashes after flush() but before the UPDATE commits, the same rows are published again on restart. That is the at-least-once gap, and it is why every outbox event carries a stable event_id that consumers dedupe on. An idempotent Kafka producer does not close this gap: a restarted relay is a new producer session, so the broker sees the resend as new records.

Two ordering caveats:

  • With SKIP LOCKED and several relay workers, events for the same aggregate_id can be published out of order. If per-entity order matters, partition relay work by aggregate_id or run a single relay.
  • BIGSERIAL order is allocation order, not commit order. A transaction that took ID 101 can commit after one that took ID 102. A relay that remembers "last ID published = 102" would skip 101 forever. Selecting on published_at IS NULL (as above) rather than on a high-water mark avoids that.

CDC Relay (Debezium)

Instead of polling, a change-data-capture connector tails the database's replication log (Postgres logical decoding, MySQL binlog) and turns each committed outbox insert into a Kafka record. Debezium ships an "outbox event router" transform for exactly this. Advantages: no polling load, events arrive in commit order, and the outbox rows can be deleted immediately after insert (the log still carries them). The connector stores its log position in Kafka and, after a crash, resumes from the last recorded position, so it can republish a few events. Downstream dedup is still required. See Kafka Connect and the full pattern page at /architecture-patterns/event-driven/transactional-outbox.

The Inbox: The Outbox's Mirror Image

The outbox guarantees an event leaves exactly once in effect. The inbox guarantees it is applied exactly once in effect. The consumer writes the incoming event_id into an inbox (dedup) table in the same transaction as its business change:

sequenceDiagram
    participant OS as Order Service DB
    participant R as Relay
    participant K as Kafka
    participant C as Billing Consumer
    participant BD as Billing DB
    OS->>OS: TX - update order + insert outbox row (event_id E1)
    R->>OS: read unpublished rows
    R->>K: publish E1
    R--xOS: crash before marking published
    R->>K: publish E1 again after restart
    K->>C: E1
    C->>BD: TX - insert inbox E1 + create invoice
    K->>C: E1 (duplicate)
    C->>BD: TX - insert inbox E1 conflicts, roll back, skip
    Note over C,BD: Invoice created once

Outbox on the producer side plus inbox on the consumer side is the standard recipe for exactly-once effect between two services that each own a database. Neither needs a distributed transaction.


Kafka Exactly-Once

Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017). It covers:

1. Idempotent Producer

Prevents duplicates between producer and broker caused by retries.

Producer sends: msg(seq=1, data=X)
Network drop before ACK
Producer retries: msg(seq=1, data=X)
Broker deduplicates: "already saw seq=1 from this producer" → ignores ✓

Each producer gets a PID (Producer ID). Each message has a sequence number. The broker maintains state per (PID, partition) and deduplicates.

Enable:

props.put("enable.idempotence", "true");

How the idempotent producer works

  • On startup the producer asks the cluster for a producer ID and an epoch.
  • For every partition it writes to, it numbers record batches with a sequence number starting at 0.
  • The partition leader remembers, per producer ID, the sequence numbers of the most recent batches it appended (the last five). A batch whose sequence is already there is acknowledged as success without being appended again. A batch whose sequence skips ahead indicates lost data and is rejected with an out-of-order sequence error.
  • This state is written into the log with the batches, so it survives leader failover: the new leader rebuilds it from the log.

Requirements and defaults:

SettingRequired valueWhy
acksallA batch acknowledged by only the leader could vanish in failover, breaking the sequence
retries> 0Retrying is the point. Duplicates are now safe
max.in.flight.requests.per.connection≤ 5The broker tracks the last five batches per producer and partition. Ordering is preserved even with retries

Since Apache Kafka 3.0, enable.idempotence=true and acks=all are the producer defaults, so modern clients get duplicate-free retries unless they opt out or set conflicting options.

What it does not cover

The idempotent producer only removes duplicates caused by the producer's own internal retries within one session. It does not help when:

  • The application calls send() twice for the same logical event (for example, a relay that re-reads an outbox row after restarting). The second call gets a new sequence number and is a new record.
  • The producer process restarts. Without a transactional.id, a restarted producer gets a new producer ID, and the broker has no way to link it to the old one.
  • Writes span multiple partitions and you need them all-or-nothing. That is what transactions add.

2. Transactions

Allows atomically writing to multiple partitions and topics.

producer.initTransactions();
producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", orderId, "confirmed"));
producer.send(new ProducerRecord<>("notifications", userId, "order_confirmed"));
producer.commitTransaction();  // atomic — either both land or neither

Transaction coordinator: A special Kafka broker component that manages transaction state using a __transaction_state internal topic.

The transaction protocol, step by step

Every transactional producer is configured with a transactional.id, a stable name that survives restarts (for example payments-enricher-partition-3). The coordinator for that ID is the leader of the __transaction_state partition the ID hashes to.

sequenceDiagram
    participant P as Producer
    participant TC as Transaction Coordinator
    participant L as Partition Leaders
    participant GC as Group Coordinator
    P->>TC: InitProducerId (transactional.id)
    TC->>TC: bump epoch, abort or finish any open txn from older epoch
    TC-->>P: producer id + epoch
    P->>TC: AddPartitionsToTxn (orders-0, notifications-2)
    TC->>TC: log state Ongoing with partition list
    P->>L: produce records (pid, epoch, seq)
    P->>TC: AddOffsetsToTxn (consumer group)
    P->>GC: TxnOffsetCommit (input offsets)
    P->>TC: EndTxn COMMIT
    TC->>TC: log PrepareCommit
    TC->>L: write COMMIT markers
    TC->>GC: write COMMIT marker on offsets partition
    TC->>TC: log CompleteCommit
    TC-->>P: committed

What each piece achieves:

  1. InitProducerId with a bumped epoch fences zombies. If an old instance of the same transactional.id is still alive (say it was stuck in a GC pause and its replacement has started), its epoch is now stale, and every broker rejects its writes with a fencing error. This is the same idea as a fencing token, with the epoch playing the token's role.
  2. Registering partitions with the coordinator before writing tells it where to write commit or abort markers later.
  3. Records are written to the real partitions immediately, not buffered. They are simply not visible to read_committed consumers yet.
  4. Offsets go through the transaction too. sendOffsetsToTransaction turns "I consumed up to offset N" into part of the same atomic unit as the output records. This is what makes read-process-write exactly-once.
  5. Commit is a two-phase commit run by the coordinator. Once PrepareCommit is durably logged in __transaction_state, the transaction will commit, even if the coordinator crashes: the new coordinator reads the log and finishes writing markers. That logged decision is the commit point.

Readers: isolation.level

Consumers choose what they see:

isolation.levelSees
read_uncommitted (default)Every record, including those from transactions later aborted
read_committedOnly committed transactional records (and non-transactional ones). Aborted records are filtered out

A read_committed consumer cannot read past the last stable offset (LSO): the offset of the first record belonging to a still-open transaction. A transaction that stays open for a long time therefore stalls every read_committed consumer of that partition, even for records from other producers that have committed. This is the main operational cost of transactions, and why transaction.timeout.ms matters (the coordinator aborts transactions that run longer).

Forgetting read_committed

The most common EOS bug is a downstream consumer left at the default read_uncommitted. Aborted records, including the partial output of a crashed transaction, are then processed as if real. Exactly-once on the write side is undone on the read side.

A full read-process-write loop with the plain clients

Kafka Streams does all this for you, but the raw version shows what is happening:

Properties pp = new Properties();
pp.put("bootstrap.servers", "kafka:9092");
pp.put("transactional.id", "enricher-" + instanceId);   // stable across restarts
pp.put("enable.idempotence", "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(pp, ser, ser);

Properties cp = new Properties();
cp.put("bootstrap.servers", "kafka:9092");
cp.put("group.id", "enricher");
cp.put("enable.auto.commit", "false");                  // offsets go via the txn
cp.put("isolation.level", "read_committed");            // don't read aborted input
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(cp, de, de);

producer.initTransactions();                            // fences older instances
consumer.subscribe(List.of("payments"));

while (true) {
    ConsumerRecords<String, String> batch = consumer.poll(Duration.ofMillis(200));
    if (batch.isEmpty()) continue;
    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, String> r : batch) {
            producer.send(new ProducerRecord<>("payments-enriched", r.key(), enrich(r.value())));
        }
        producer.sendOffsetsToTransaction(nextOffsets(batch), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException e) {
        producer.close();          // a newer instance owns this transactional.id, stop
        throw e;
    } catch (KafkaException e) {
        producer.abortTransaction();
        rewindToLastCommitted(consumer);   // re-read the batch, its output was discarded
    }
}

nextOffsets must return, for each partition, the offset of the last processed record plus one (Kafka commits the position of the next record to read, not the last one read). Off-by-one here reprocesses one record per partition after every restart.

Passing consumer.groupMetadata() (rather than just a group ID string) is the KIP-447 improvement. Before it, fencing during consumer-group rebalances needed one transactional producer per input partition. With it, a single producer per application instance is enough, because the group coordinator can fence stale offset commits using the consumer group generation. Kafka Streams exposes this as exactly_once_v2 (introduced as exactly_once_beta in 2.5, renamed in 3.0).

3. Exactly-Once in Kafka Streams

The full end-to-end guarantee: read-process-write atomically.

Consumer reads msg from input topic
Consumer processes (transforms, enriches)
Producer writes to output topic
Consumer commits offset
All four steps are atomic — either all happen or none
props.put("processing.guarantee", "exactly_once_v2");

Cost: Lower throughput and higher latency than at-least-once. Only guarantees exactly-once within the Kafka ecosystem — downstream systems (databases, external APIs) need their own idempotency.

Correction (in place)

This line previously stated a flat "~20% throughput reduction". The overhead is not a fixed number. It depends mostly on how often transactions commit. The design document for KIP-98 reported only a few percent throughput loss for transactional producers committing every 100 ms with 1 KB messages, because the per-transaction cost (coordinator round trips and markers) is amortised over many records. Kafka Streams commits every 100 ms by default under EOS (versus 30 s under at-least-once), so the effective cost and the added end-to-end latency (records become visible to read_committed readers only at commit) grow as you shorten the commit interval. Measure with your own record sizes and commit interval.

What Kafka EOS covers, and what it does not

CoveredNot covered
Duplicate appends from producer retriesAn application calling send() twice for the same event
Atomic writes across many partitions and topicsWrites to anything outside Kafka (databases, HTTP calls, emails)
Atomic "consume offsets + produce outputs"Side effects inside your processing function (a call to an API during enrich() runs once per attempt)
Fencing of zombie instances with the same transactional.idTwo instances configured with different IDs for the same work
State stores in Kafka Streams (changelogs are written in the transaction)Consumers reading with read_uncommitted

The third row in the right column is the one that surprises people. If your Streams processor calls a REST service on every record, an aborted and retried transaction calls it again. Kafka can roll back its own records, not someone else's database.


Apache Flink takes a different route to the same goal. Instead of a transaction per batch of records, it takes consistent snapshots of the entire running job at intervals and, on failure, rewinds everything to the last snapshot.

Exactly-once state via checkpoints

Flink's checkpointing is a variant of the Chandy-Lamport snapshot algorithm, adapted for dataflow graphs (Flink's papers call it asynchronous barrier snapshotting).

  1. The JobManager's checkpoint coordinator injects a barrier with checkpoint ID into every source.
  2. Each source records its position (for Kafka, the offset per partition) as part of checkpoint , then emits the barrier downstream, in-line with records.
  3. When an operator with several inputs receives barrier on one input, it waits (aligns) until barrier has arrived on all inputs, buffering records that arrive after the barrier on the faster inputs. Then it snapshots its state and forwards the barrier.
  4. When every sink has acknowledged barrier , checkpoint is complete: a consistent cut through the whole job, where every operator's state reflects exactly the records before the barrier.
sequenceDiagram
    participant JM as Checkpoint Coordinator
    participant S as Kafka Source
    participant O as Stateful Operator
    participant K as Sink
    participant ST as Durable Storage
    JM->>S: trigger checkpoint 42
    S->>ST: save offsets for checkpoint 42
    S->>O: barrier 42 (in the data stream)
    O->>ST: snapshot state for checkpoint 42
    O->>K: barrier 42
    K->>ST: snapshot sink state
    K-->>JM: ack 42
    O-->>JM: ack 42
    S-->>JM: ack 42
    JM->>JM: checkpoint 42 complete
    JM->>K: notify checkpoint 42 complete

On failure, Flink restarts the job, restores every operator's state from the last completed checkpoint, and resets the sources to the offsets saved in it. Records after the checkpoint are re-read and re-processed. Internal state ends up as if each record was processed once, because the effects of the replayed records were never part of the restored state.

Note what that means: records are processed more than once; state reflects them once. That is the "processing vs delivery" distinction again. Flink also offers unaligned checkpoints (barriers overtake buffered records, which are saved as part of the checkpoint) to keep checkpoints fast under backpressure, still with exactly-once state.

The sink problem

Replaying records is harmless for internal state, but a sink that wrote to the outside world between the last checkpoint and the crash has already emitted output for the replayed records. After the rewind those outputs are emitted again. Exactly-once state does not give exactly-once output unless the sink cooperates. There are two ways:

Sink strategyHowExample
Idempotent sinkWrites are upserts keyed by a deterministic key, so replays overwrite with identical valuesKey-value stores, Elasticsearch with document IDs, JDBC upserts
Transactional (two-phase-commit) sinkOutput since the last checkpoint goes into an open external transaction that commits only when the checkpoint completesKafka sink with transactions, file sinks that rename on commit

Two-phase-commit sinks

Flink's two-phase-commit sink pattern ties an external transaction to the checkpoint lifecycle. The Flink checkpoint is the prepare phase. Notification of checkpoint completion is the commit phase.

sequenceDiagram
    participant F as Flink Sink Task
    participant JM as Checkpoint Coordinator
    participant K as Kafka
    F->>K: begin txn T7, write records
    JM->>F: barrier for checkpoint 42
    F->>K: flush T7 (pre-commit)
    F->>F: store T7 id in checkpoint 42 state
    F->>K: begin txn T8 for new records
    F-->>JM: ack checkpoint 42
    JM->>JM: all tasks acked, checkpoint 42 complete
    JM->>F: notifyCheckpointComplete 42
    F->>K: commit T7
    Note over F,K: Crash after 42 completes but before commit? Recovery restores 42, which names T7, and commits it

The lifecycle methods (from the classic TwoPhaseCommitSinkFunction; the newer Sink V2 API has the same shape with a writer and a committer):

MethodWhenMust do
beginTransactionStart, and after each pre-commitOpen a fresh external transaction
preCommitCheckpoint barrier reaches the sinkFlush everything, so that commit cannot fail for lack of data. Then stash the transaction handle in checkpoint state
commitCheckpoint is completeCommit. Must be idempotent, because after a crash Flink may call it again for a transaction that already committed
abortFailure before the checkpoint completedRoll back. Output for these records will be produced again on replay

Why this is safe: an external transaction commits only after the checkpoint that contains its handle is durable. If Flink crashes before checkpoint completion, it restores the previous checkpoint and aborts the open transaction; the records are replayed into a new transaction. If it crashes after completion but before commit finished, the restored checkpoint names the transaction, and recovery commits it.

KafkaSink<String> sink = KafkaSink.<String>builder()
    .setBootstrapServers("kafka:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("orders-enriched")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("orders-enricher")   // must be unique per job on the cluster
    .setProperty("transaction.timeout.ms", "900000")   // 15 min, see warning below
    .build();

env.enableCheckpointing(60_000, CheckpointingMode.EXACTLY_ONCE);

Transaction timeout vs checkpoint interval

The external transaction stays open from one checkpoint until the next one completes, plus however long recovery takes. If the broker aborts it first (Kafka's broker cap transaction.max.timeout.ms defaults to 15 minutes), the output is lost: Flink will try to commit a transaction Kafka has already discarded. Set the producer's transaction.timeout.ms well above "checkpoint interval + worst checkpoint duration + expected downtime", and no higher than the broker cap (raise the cap if needed). The opposite mistake also bites: the Kafka client's own default transaction.timeout.ms is 60 seconds, shorter than many checkpoint intervals. The legacy FlinkKafkaProducer defaulted to one hour, which exceeds the default broker cap and is rejected unless one of the two is changed.

Two more consequences:

  • Output latency equals the checkpoint interval. Downstream read_committed consumers see nothing until the checkpoint completes. A 60-second checkpoint interval means up to 60 seconds of delay plus checkpoint time. Exactly-once trades latency for correctness.
  • Downstream consumers must use read_committed, as with any Kafka transaction.
AspectKafka Streams EOSFlink EOS
Unit of atomicityOne transaction per task per commit intervalOne global checkpoint across the whole job
State durabilityChangelog topics written inside the transactionSnapshots in durable storage (S3, HDFS)
RecoveryEach task restores its state from changelog, resumes from committed offsetsWhole job (or affected region) rewinds to last checkpoint
Default commit cadence under EOS100 msWhatever checkpoint interval you set (often tens of seconds)
Sources and sinksKafka onlyAny source that can replay, any sink that is idempotent or transactional
DeploymentA library inside your appA cluster (JobManager plus TaskManagers)

For more on stream processors, see Kafka Streams and exactly-once processing in stream pipelines.


Exactly-Once Across Different Layers

LayerMechanismCost
Producer → BrokerIdempotent producer (sequence numbers)Low
Broker durabilityReplication + acks=allMedium
Consumer → ApplicationIdempotency keys + deduplicationMedium
Application → DatabaseTransactional outbox or DB transactionsMedium
End-to-end (Kafka Streams)Transactions + atomic offset commitsHigh
Cross-serviceSaga pattern + compensating transactionsVery high

Practical Patterns

Deduplication Table

CREATE TABLE processed_events (
  event_id UUID PRIMARY KEY,
  processed_at TIMESTAMP DEFAULT NOW()
);

-- On message receive:
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING *;
-- If nothing returned: already processed, skip
-- If row returned: process and commit atomically

The comment "process and commit atomically" carries all the weight. The insert and the business update must be in one transaction, and the broker acknowledgement (offset commit) must come after that transaction commits:

def handle(msg, conn):
    with conn.transaction():                       # BEGIN ... COMMIT
        fresh = conn.execute(
            "INSERT INTO processed_events (event_id) VALUES (%s) "
            "ON CONFLICT (event_id) DO NOTHING RETURNING event_id",
            (msg.headers["event_id"],)).fetchone()
        if fresh:
            conn.execute("UPDATE accounts SET balance = balance + %s WHERE id = %s",
                         (msg.amount, msg.account_id))
    consumer.commit(msg)                           # ack only after the DB commit

Walk through every crash point:

Crash betweenOn restartEffect count
Receive and BEGINMessage redelivered, not in table, processed1
Inside the transactionTransaction rolled back, message redelivered, processed1
DB commit and offset commitMessage redelivered, insert conflicts, skipped1
After offset commitNot redelivered1

Swap the order (ack first, then DB) and the third row becomes "message never redelivered, DB never updated": a lost message. Split the dedup insert and the business update into two transactions and a crash between them gives either a skipped effect or a doubled one, depending on which you do first.

How long to remember

A dedup table that grows forever eventually becomes a problem. Choose a retention window longer than the maximum redelivery horizon: the longest time a duplicate can possibly show up after the original.

Duplicate sourceTypical horizon
Producer retriesSeconds to minutes
Consumer crash and rebalanceMinutes
Outbox relay restartMinutes
Operator replays a topic from an old offsetDays, or "the topic's retention"
Client app retries after being offlineHours to days

Deduplication past the window is not guaranteed, which is a deliberate, documented trade-off. Partition the table by day (PARTITION BY RANGE (processed_at)) so expiry is a cheap DROP PARTITION rather than a slow DELETE.

Store Offsets With the Results

When the consumer's output lives in a single transactional database, there is a cleaner option than a dedup table: store the consumer's input offsets in the same database, in the same transaction as the results, and ignore the broker's offset storage.

CREATE TABLE consumer_offsets (
  consumer_group TEXT,
  topic          TEXT,
  partition      INT,
  next_offset    BIGINT NOT NULL,
  PRIMARY KEY (consumer_group, topic, partition)
);
def on_partitions_assigned(consumer, partitions):
    for tp in partitions:                       # resume from the DB, not from Kafka
        row = db.fetchone("SELECT next_offset FROM consumer_offsets "
                          "WHERE consumer_group=%s AND topic=%s AND partition=%s",
                          (GROUP, tp.topic, tp.partition))
        consumer.seek(tp, row.next_offset if row else 0)

def process_batch(records):
    with db.transaction():
        for r in records:
            apply_business_change(r)
        for (topic, part), last in last_offset_per_partition(records).items():
            db.execute("""INSERT INTO consumer_offsets VALUES (%s,%s,%s,%s)
                          ON CONFLICT (consumer_group, topic, partition)
                          DO UPDATE SET next_offset = EXCLUDED.next_offset""",
                       (GROUP, topic, part, last + 1))

The database is the single source of truth for both "what I have applied" and "where I am in the input", and they can never disagree. No per-message dedup table is needed, because the database never re-applies an offset it has already recorded. The trade-offs: duplicates that originate upstream (two records with the same event at different offsets) are not caught, and the approach only works when every effect goes to that one database. Kafka Connect sinks such as the JDBC sink and many Flink sinks use variants of this idea.

Two consumers, one partition

During a consumer-group rebalance, an old consumer that has not yet noticed it lost a partition can still be finishing a batch while the new owner starts on the same records. With offsets in the database, guard the offset update so it only moves forward and only from the expected position:

UPDATE consumer_offsets SET next_offset = $new
WHERE consumer_group = $g AND topic = $t AND partition = $p
  AND next_offset = $expected;     -- 0 rows: someone else advanced it, roll back

This turns the offset row into a compare-and-swap that fences the stale consumer.

Conditional Update (Compare-and-Swap)

-- Only apply if we haven't already applied this version
UPDATE orders
SET status = 'shipped', version = version + 1
WHERE id = $1 AND version = $expected_version;

-- If 0 rows affected: already processed (version mismatch) → skip

Fencing with Tokens

1. Acquire distributed lock → get token T
2. Perform work with token T
3. Write result with condition: "only if T is not older than the highest token the store has seen"
4. Release lock

If lock expires and another process gets token T+1:
Old process's write (with token T) is rejected ✓

(Correction: this step previously read "only if current token is T". The store does not usually know which token is "current". It only knows the highest token it has accepted, and rejects anything lower. See leases and fencing.)

Fencing prevents a stale worker from applying an effect after its lease ran out. It is complementary to deduplication, which prevents the same message from being applied twice. Exactly-once pipelines with leased partition ownership usually need both. Kafka's producer epoch and consumer-group generation are fencing tokens of exactly this kind.

Deterministic Processing

Replay-based schemes (Kafka Streams, Flink, offsets-in-DB) assume that reprocessing the same input yields the same output. Code that reads the wall clock, generates random IDs, or calls a service whose answer changes breaks that assumption: after a replay the output differs from what the aborted attempt produced. Because the aborted attempt's output was discarded, this is usually harmless for correctness, but it matters when output IDs are used for dedup further downstream. Derive IDs from input (event_id + ":out:" + n) rather than from UUID.randomUUID(), and take timestamps from the record, not from now().


The Real Answer

True end-to-end exactly-once delivery across network boundaries is theoretically impossible in the general case: with lossy links and crashing processes, a sender cannot tell a lost message from a lost acknowledgement (the Two Generals problem, see above). This is a different result from FLP, which is about reaching consensus. What systems provide is:

  1. Idempotent operations — safe to retry, same result
  2. Deduplication — detect and discard duplicates
  3. Atomic transactions — within a single system boundary

When someone says "exactly-once," they usually mean "at-least-once delivery with idempotent processing." That's achievable, practical, and what you should build. True "the effect happens exactly once regardless of failures" requires a transaction coordinator and is only practical within a single system boundary (like Kafka Streams).


End-to-End Worked Examples

Example 1: A mobile "Pay" button to a card processor

Goal: the customer is charged once per tap of "Pay", despite flaky mobile networks, app restarts and server crashes.

sequenceDiagram
    participant App as Mobile App
    participant API as Payments API
    participant DB as Payments DB
    participant PSP as Card Processor
    App->>App: generate key K1, persist locally
    App->>API: POST /charges, Idempotency-Key K1
    API->>DB: TX - claim K1 (in_progress), insert charge row (pending)
    API->>PSP: authorize, idempotency key derived from K1
    PSP-->>API: approved
    API->>DB: TX - charge = succeeded, K1 = completed with response
    API--xApp: response lost (tunnel)
    App->>API: POST /charges, Idempotency-Key K1 (retry)
    API->>DB: K1 completed, read stored response
    API-->>App: 201 (replayed), no new charge

Walk the failure points:

FailureWhat happensCharged
Request never reaches APIApp retries with K1, first real attemptOnce
API crashes after claiming K1, before calling the processorTransaction never committed, K1 not claimed, retry starts cleanOnce
API crashes after the processor approved, before recording itK1 is stuck in_progress with a pending charge. A recovery job (or the next retry after the lease expires) re-calls the processor with the same derived key, and the processor returns the original approval rather than charging againOnce
Response lost on the way backRetry with K1 returns the stored responseOnce
App killed and restartedK1 was persisted before the first send, so the retry reuses itOnce
App generates a new key on restart (bug)Server sees a brand-new operationTwice

Note that the first commit in the diagram must be committed before the processor call, not held open across it. Holding a database transaction open across a network call to a third party pins a connection and row locks for an unbounded time. The "pending row, then external call, then update" sequence is a tiny saga, each step durable, each step resumable.

The last row is the lesson. Every layer did its job except the one that owned the key. The idempotency key has to be minted by whoever retries, and stored before the first attempt.

Example 2: Kafka topic into Postgres, with an email at the end

Goal: for each OrderPlaced event, insert an invoice row exactly once and send the customer one email.

  1. The order service writes the order and an outbox row in one transaction. Debezium publishes the outbox row to orders.events with header event_id = E. (At-least-once into Kafka.)
  2. The invoicing consumer reads orders.events and, in one Postgres transaction, inserts E into inbox, inserts the invoice, and inserts an email_outbox row keyed by E. Then it commits its Kafka offset. (Exactly-once effect in the invoicing DB.)
  3. An email worker reads email_outbox, calls the email provider passing E as the provider's idempotency key (or dedupe ID), and marks the row sent. (At-least-once to the provider, deduplicated by the provider if it supports keys.)

If the email provider does not support idempotency keys, step 3 is at-least-once and the customer may, rarely, get two emails: when the worker crashes after the provider accepted the send and before the row was marked. You cannot close that gap from your side. You can only shrink it (mark immediately after the call, keep the call fast) and decide that a rare duplicate email is acceptable. Many teams decide exactly that, and it is a reasonable decision as long as it is a conscious one.

flowchart LR
    A["Order DB + outbox"] -->|"CDC, at-least-once"| K["orders.events"]
    K -->|"at-least-once"| C["Invoicing consumer"]
    C -->|"one TX: inbox + invoice + email_outbox"| D["Invoicing DB"]
    D -->|"poll"| W["Email worker"]
    W -->|"key = event_id"| E["Email provider"]

Goal: count purchases per product per minute and publish the counts, exactly once.

  • Source: Kafka purchases, offsets saved in Flink checkpoints (not committed to the consumer group for correctness).
  • Operator: keyed tumbling window, counts in RocksDB state, snapshotted every 30 seconds.
  • Sink: KafkaSink with EXACTLY_ONCE and transaction.timeout.ms = 900000.
  • Downstream dashboard consumer: isolation.level=read_committed.

Failure at second 47 after checkpoint 12 completed at second 30:

  1. TaskManager dies. Flink restarts tasks and restores state and source offsets from checkpoint 12.
  2. Window counts revert to their value at second 30. Purchases from seconds 30 to 47 are re-read from Kafka and counted again, from the restored state, so each counts once.
  3. The sink's open transaction (holding output written after checkpoint 12) is aborted. Its records never become visible to read_committed readers.
  4. Output is recomputed and written into a new transaction, which commits when checkpoint 13 completes.

The dashboard sees each per-minute count once, about 30 seconds later than it would under at-least-once. If the dashboard consumer used read_uncommitted, it would have seen the aborted partial output too, and double-counted.


Failure Scenarios Catalogue

#ScenarioSymptomRoot causeFix
1Producer retry after lost ackDuplicate records in topicNon-idempotent producerenable.idempotence=true (default since Kafka 3.0)
2Zombie instance after GC pauseTwo instances writing the same outputNo fencingStable transactional.id. The newer instance's initTransactions() fences the old one
3Consumer crashes after DB write, before offset commitMessage processed twiceAck and effect not atomicDedup table or offsets in the same DB transaction
4Offset committed before processingMessages silently lost on crashAuto-commit or ack-firstDisable auto-commit, commit after the effect
5Downstream reads read_uncommittedAborted output appears, counts doubleReader isolation defaultisolation.level=read_committed
6Long-running transactionAll read_committed consumers stallLSO pinned by open transactionShorter transactions, sensible transaction.timeout.ms
7Flink transaction aborted by broker during long outageOutput for a checkpoint lostTxn timeout below recovery timeRaise transaction.timeout.ms and broker cap
8Dedup TTL shorter than replay horizonDuplicates after an operator replays old dataRetention too shortSize the window to the real horizon, document it
9Side effect inside a stream processorExternal API called on every retryEOS covers Kafka state onlyMove side effects to a sink with its own idempotency
10Client mints a new idempotency key per retryDouble chargesKey generated at the wrong layerGenerate once per intent, persist before first send
11Outbox relay with high-water-mark pollingSome events never publishedSequence allocation order differs from commit orderPoll on published_at IS NULL, or use CDC
12Same idempotency key reused with different bodyWrong response replayedClient bugFingerprint requests, reject mismatches with an error

War Story (illustrative)

A ledger team moved its balance updates onto Kafka Streams with exactly_once_v2 and declared victory. Two weeks later, finance found a few hundred customers with notification emails sent twice and one partner bank that had received duplicate settlement instructions. The Streams topology was correct: every balance in the output topics was right. The problem was a mapValues step that called the partner bank's HTTP API directly "to keep latency low". Each time a task aborted a transaction (during rebalances after deploys), the batch was reprocessed and the HTTP call was made again. Kafka rolled back its own records. It had no way to roll back a bank's. The fix was to emit "settlement requested" records to a topic, and have a separate sink service call the bank with the record's event ID as the bank's idempotency reference, recording each acknowledged call in its own database.


Real Numbers

These are orders of magnitude to reason with, not benchmarks. Measure your own setup.

MechanismAdded cost per operationNotes
Idempotent producerNegligibleSequence tracking is per batch, not per record. It is on by default in current Kafka
Kafka transactionA few extra round trips per transaction (coordinator, markers)Amortised over the batch. Overhead shrinks as commit interval grows
Kafka Streams EOSLatency ≈ commit interval (100 ms default)Output becomes visible to read_committed readers only at commit
Flink 2PC sinkLatency ≈ checkpoint interval plus checkpoint duration10 s to a few minutes is typical
Dedup table insert (Postgres, indexed UUID)One extra index insert per message, in an existing transactionUsually sub-millisecond. Watch table size and bloat
Idempotency-key API lookupOne indexed read (and on first call, an insert)Small next to the work itself
Outbox via CDCWrite amplification of one row per event, plus replication-log readingRemoves the dual-write window entirely

The broad rule: per-record costs are cheap, and the expensive part of exactly-once is the commit cadence, which trades latency for overhead.


Decision Guide

flowchart TD
    Q1{"Is a duplicate effect harmful?"} -->|"No (metrics, cache hints)"| AL["At-least-once, no dedup"]
    Q1 -->|"Yes"| Q2{"Can the operation be made naturally idempotent?"}
    Q2 -->|"Yes (upsert, set state)"| NI["At-least-once + idempotent writes"]
    Q2 -->|"No"| Q3{"Is everything inside one system?"}
    Q3 -->|"Kafka to Kafka"| KS["Kafka transactions or Streams exactly_once_v2"]
    Q3 -->|"Kafka to one database"| DB["Dedup table or offsets in the same DB TX"]
    Q3 -->|"Stateful stream job, many sinks"| FL["Flink checkpoints + 2PC or idempotent sinks"]
    Q3 -->|"Crosses services or third parties"| XS["Outbox + inbox + idempotency keys at every hop"]
SituationRecommended approachAvoid
Public payments APIClient idempotency keys, stored atomically with the resultServer-generated keys
Service publishing events after DB changesTransactional outbox with CDCDual writes
Consumer updating its own DBInbox or dedup table in the same transaction, ack after commitAuto-commit offsets
Kafka-to-Kafka transformationKafka Streams exactly_once_v2Hand-rolled consumer plus producer without transactions
Windowed aggregations with external sinksFlink with transactional or idempotent sinksNon-idempotent JDBC inserts
Notifications, emails, SMSAt-least-once with provider-side dedup keys, accept rare duplicatesPretending it is exactly-once
Cross-service business processSagas with idempotent steps and compensationsDistributed 2PC across services

Related reading: idempotent consumers, the message-queue view of exactly-once, Kafka internals, and timeouts, retries and backoff for the retry side of the story.


Interview Questions

Q1. Is exactly-once delivery possible? Not over an unreliable network with failures: a sender cannot distinguish a lost message from a lost acknowledgement, so it must either risk loss (no retry) or risk duplication (retry). What is achievable is exactly-once processing: at-least-once delivery plus deduplication or idempotency, made atomic with the effect inside some boundary.

Q2. What does Kafka's idempotent producer guarantee, and what doesn't it? It removes duplicates caused by the producer's own retries within a session, per partition, using producer ID and sequence numbers. It does not dedupe application-level double sends, does not survive a restart without a transactional.id, and gives no cross-partition atomicity.

Q3. Explain how Kafka transactions make read-process-write exactly-once. The output records and the input offsets are committed in one transaction coordinated by the transaction coordinator, which uses a logged two-phase commit and writes commit markers to every involved partition. Readers with read_committed only see committed data. A stable transactional.id with epochs fences zombie instances.

Q4. Your Kafka Streams app is configured for exactly-once, yet a downstream service was called twice. Why? The call was a side effect inside the processor. EOS covers Kafka records and state stores only. An aborted transaction (from a rebalance or crash) replays the batch, and the side effect runs again. Move it to a sink that uses an idempotency key.

Q5. How does Flink achieve exactly-once state, and why is that not enough for exactly-once output? Periodic barrier-based snapshots of all operator state plus source offsets. On failure it restores and replays. Records are processed again, but state reflects them once. External outputs written since the last checkpoint would be emitted twice, so the sink must be idempotent or transactional (two-phase commit tied to checkpoint completion).

Q6. In the dedup-table pattern, in what order do you commit the DB transaction and the broker ack, and why? DB first, ack second. A crash between them causes a redelivery that the dedup table absorbs. The reverse order loses the message on a crash between them.

Q7. Why is the outbox pattern needed? Why not publish to Kafka inside the DB transaction? The broker is not part of the database transaction. If the publish succeeds and the commit fails, you have a phantom event. If you publish after commit, a crash loses the event. The outbox puts the event in the same local transaction and relays it at-least-once.

Q8. How long should a dedup table keep keys? Longer than the longest time a duplicate can appear: producer retries, rebalances, relay restarts, and any operational replay. Document the horizon, because duplicates beyond it are not caught.

Q9. Design a Flink job's Kafka sink for exactly-once. What configuration bites people? DeliveryGuarantee.EXACTLY_ONCE, a unique transactional ID prefix, read_committed downstream, and a transaction timeout larger than checkpoint interval plus recovery time but within the broker's transaction.max.timeout.ms. Too short, and the broker aborts transactions Flink still intends to commit.

Q10. What is the difference between idempotency and fencing? Idempotency makes applying the same operation twice harmless. Fencing makes operations from a stale owner (whose lease or epoch has expired) be rejected. A pipeline with leased partition ownership usually needs both: one for duplicates, the other for zombies.

Q11. When is "offsets stored in the sink database" better than a dedup table? When all effects go to one transactional database: it needs no per-message table and can never disagree with itself. It does not catch duplicates introduced upstream (the same event at two offsets), so it pairs well with an idempotent or transactional producer upstream.

Q12. A product manager asks for "exactly-once email delivery". What do you say? You can guarantee that the system requests each email exactly once in effect, and you can pass a dedup key to the provider if it supports one. If the provider has no idempotency support, a crash between the provider accepting the email and you recording it can produce a duplicate. You can make it rare. You cannot make it impossible. Usually a rare duplicate email is acceptable, and saying so explicitly is better than promising something no system can deliver.