backenddrills

Complete free practice · 9 minutes

Kafka duplicate messages: a free replay practice drill

A consumer restart repeats a side effect even though the producer is idempotent.

Inspect the supplied evidence, write your hypothesis, then compare your choices with the reasoning. This educational simulation is free without an account or payment. Read the exercise here, or use the interactive lab to record your decisions.

Start this interactive free drill →See the complete edition · $28 one-time

The incident brief

The shipping consumer commits a shipment to PostgreSQL and dies before committing its Kafka offset. On restart, the same event creates another shipment. Producer idempotence is enabled. Database and offset commits are separate.

Environment and evidence

Kafka consumer · PostgreSQL · at-least-once processing

offset 418: event_id=evt-208 order_id=ord-91
14:20:01 INSERT shipment → COMMIT
14:20:02 process terminated
14:20:05 restart, offset 418 delivered again
14:20:06 INSERT shipment → COMMIT
No unique constraint on event_id or business operation

Decision 1: Diagnose

Why did producer idempotence not prevent the duplicate shipment?

  1. Producer idempotence does not make the consumer database effect atomic with offset progress.
  2. Kafka offsets must be globally unique across all partitions.
  3. The consumer should ignore all events after a restart.
Reveal the reasoning after choosing

Option A: The duplicate is a replay after a database commit, not necessarily a duplicate produced record.

Option B: Offsets identify positions inside a partition. That does not close this crash window.

Option C: That loses valid business events.

Decision 2: Stabilize

How should processing handle redelivery?

  1. Commit the offset before the database write.
  2. Remember the last event in a Java HashSet.
  3. Enforce a durable event or business-operation uniqueness key in the same transaction as the shipment.
Reveal the reasoning after choosing

Option A: A crash after committing the offset can now lose the shipment entirely.

Option B: This can suppress some duplicates while the process runs, but state is lost on restart and not shared across consumers.

Option C: The duplicate check and effect must be atomic. Handle a duplicate as a known replay and commit the offset only after a durable result.

Decision 3: Prevent

What fault-injection test is most useful?

  1. Test only a clean shutdown.
  2. Kill the consumer after database commit but before offset commit; restart and assert one business effect.
  3. Increase the number of topic partitions.
Reveal the reasoning after choosing

Option A: The dangerous window occurs at an abrupt failure between commits.

Option B: This exercises the exact replay window, including uniqueness conflict handling.

Option C: Partition count does not make the database transaction atomic with offsets.

Read the debrief after all three decisions

The mechanism

At-least-once delivery repeats a committed external effect when progress is not committed before a crash.

Verify the fix

Explain it in your own words

I would identify the stable event and intended business operation. The consumer must tolerate reprocessing. A database uniqueness constraint and an atomic transaction around deduplication plus shipment creation close this local replay gap. I would commit the offset after a durable result, and separately address any remote carrier call with its own idempotency or reconciliation.

Continue with the same skill

Read the related free guide → · Choose a structured study path →

Is this practice format useful to you?

The full edition contains 1008 exercises and 3024 decisions across 18 topics. Practice testing, SQL, APIs, Java 8, Java 21, design patterns, system design, cybersecurity and cloud. Related exercises revisit mechanisms through different decision lenses. You receive authored reviews and five study paths; this is a decision workbook, not a code execution service.

Explore everything included for $28 →

One-time payment. No subscription. Review the purchase and refund terms before buying.

Primary documentation

Original educational examples, prepared with AI assistance. No employer question, customer incident or hiring outcome is claimed. Editorial approach.

Try another free drill