Webhook deduplication: test the crash between receipt and commit
Test the failure between webhook receipt and commit. An executed SQLite fixture checks rollback, retry and duplicate handling without live payments.
Recording a webhook event ID is useful, but the timing of that record matters. If a worker marks an event processed and crashes before changing business data, a retry may skip work that never happened. If it changes business data and crashes before recording the ID, a retry may apply the change twice.
This article adds a storage-level failure test to the usual webhook checklist. It does not replace signature verification, payment reconciliation or durable queueing. Its example uses synthetic event IDs and an in-memory database, with no calls to Stripe or any payment system.
Separate delivery from completed processing
Stripe documents that webhook events can be delivered more than once and that delivery order is not guaranteed. Its webhook guidance recommends tracking processed event IDs. Those facts motivate the example, but the database design and test below are our implementation exercise, not a complete Stripe reference integration.
There are at least two different milestones:
- The endpoint has durably accepted an authenticated event for later work.
- The worker has committed the business change represented by that work.
A durable inbox or queue can justify acknowledging delivery before expensive processing finishes. It does not mean the worker has completed fulfillment. Avoid using a single ambiguous processed=true flag for both meanings.
Make the local database change and receipt atomic
For an operation whose effects fit inside one transactional database, place the deduplication receipt and business mutation in the same transaction. A uniqueness rule on the receipt key prevents an ordinary repeat from applying the local mutation again.
The fixture uses a deliberately small schema: receipts(event_id TEXT PRIMARY KEY) and a counter row. The counter represents a local business mutation, not a payment balance.
def apply(db, event_id, *, fail_after_update=False):
with db:
inserted = db.execute(
"INSERT OR IGNORE INTO receipts VALUES (?)", (event_id,)
).rowcount
if not inserted:
return "duplicate"
db.execute("UPDATE counters SET processed=processed+1 WHERE id=1")
if fail_after_update:
raise RuntimeError("synthetic failure before commit")
return "applied"
In this tightly controlled schema, the ignored insert means the event ID already exists. Do not copy broad OR IGNORE handling into a larger schema without reviewing every constraint it can suppress. An application should distinguish the intended duplicate from malformed data and unrelated storage errors.
The Python connection context manager commits on normal exit and rolls back on an exception. The exception is intentionally raised after both writes, before the transaction finishes.
Assert rollback before retrying
A test that only sends the same event twice misses the partial-failure window. This fixture starts with zero receipts and a zero counter, then:
- Applies event A with the injected failure enabled.
- Requires the failure to occur.
- Reads the database and requires zero receipts and a zero counter.
- Retries A without failure and requires
applied. - Replays A and requires
duplicate. - Applies distinct event B and requires
applied. - Requires two receipt rows and a final counter of two.
The complete script was executed on Python 3.10.12 and SQLite 3.37.2. All assertions passed. It proves that this explicit exception path rolls both writes back in the tested transaction configuration. It does not simulate an operating-system kill, disk corruption, network partition or database recovery after a power failure.
For a deployed system, add crash/restart tests with the actual database, storage settings and worker framework. A successful in-memory SQLite test cannot establish another database engine's durability or concurrency properties.
A receipt key needs a defined scope
This fixture has one event namespace. A production consumer serving multiple provider accounts or environments needs keys scoped so unrelated events cannot collide. It also needs a documented retention policy: deleting receipts can reopen the possibility of old work being applied again.
An event ID identifies a delivery event, not every possible business operation. Stripe notes that separate Event objects can sometimes represent duplicates and suggests using object ID plus event type to identify those cases. That requires care: the business model must still distinguish legitimate later changes. Do not impose a universal object/type uniqueness rule that prevents valid future updates.
Similarly, deduplicating an old event does not establish which of two different events represents the newest state. Ordering, reconciliation and allowed state transitions need separate rules and tests.
External side effects are outside this transaction
A database rollback cannot unsend an email, reverse an HTTP request or recall a parcel. If the worker sends email inside the transaction and then rolls back, a retry may send the same message again.
A common design option is to commit the local business change and an outbox record together, then let another component deliver the outbox item. That moves the reliability boundary; it does not eliminate the need for delivery retries and downstream idempotency. This fixture does not implement or test an outbox.
Before launch, ask the team to list each side effect and identify which transaction or idempotency boundary protects it. Verify signature checks on raw request bytes and order/account/amount reconciliation separately in supported test mode. Never infer that an authenticated event alone authorizes a particular order to ship.
Turn the fixture into an acceptance check
Ask for evidence from the failure point, not only a success screenshot: the worker fails after the local mutation, the receipt and mutation are absent, the retry applies once and another replay leaves the result unchanged. Repeat at the durable-acceptance boundary and each external delivery boundary under the application's actual design.
The goal is a precise claim about what survives a crash and what may be retried, rather than a blanket promise of "exactly once" processing.
Sources and method
- Stripe webhook documentation: duplicate deliveries, distinct Event objects, unordered delivery and asynchronous handling.
- Python 3.10 sqlite3 documentation: connection transaction context management and affected-row counts.
- SQLite transaction documentation: transaction boundaries and commit/rollback behavior.
Written with AI assistance. The synthetic failure fixture was executed locally. No live webhook endpoint, real payment, customer record or production reliability claim is involved.