Webhook retries, idempotency and failure handling

Assume every webhook arrives late, twice or out of order

Webhook delivery is at least once. A sender that doesn't get a 2xx in time sends the event again, even if your app actually processed it and only the response got lost. Networks drop connections, deploys restart servers, and two events about the same object can race each other.

A reliable handler is built around three ideas: answer fast, process each event once, and keep failed events recoverable.

1. Answer fast, work later

Return a 2xx as soon as the event is stored, then do the slow work (emails, API calls, database updates) in a background job. Most providers time out after 5 to 30 seconds; a handler that sends an email inline will eventually time out, and the provider will send the event again.

2. Make handlers idempotent

An idempotent handler gives the same result whether an event arrives once or five times. The usual pattern:

1. Every event carries a unique id: webhook-id in Standard Webhooks, evt_… in Stripe, X-GitHub-Delivery in GitHub.
2. Before acting, insert that id into a table with a unique constraint.
3. If the insert fails because the id exists, the event was already handled: answer 2xx and stop.

CREATE TABLE processed_webhooks (
  webhook_id  text PRIMARY KEY,
  received_at timestamptz NOT NULL DEFAULT now()
);

-- In the same transaction as the side effect:
INSERT INTO processed_webhooks (webhook_id) VALUES ($1)
ON CONFLICT (webhook_id) DO NOTHING;   -- 0 rows inserted: already processed

Doing the insert and the side effect in one transaction matters: if the side effect fails, the id is rolled back too and the retry can succeed.

3. Handle out-of-order events

customer.subscription.updated can arrive before customer.subscription.created. Two safe approaches:

• Fetch the current state from the provider's API when an event arrives, and treat the event only as a "something changed" signal.
• Compare versions or timestamps stored on the object, and ignore events older than what you have.

4. Retry with exponential backoff and jitter

When a delivery fails, the sender should wait longer after each attempt: for example 10 seconds, 1 minute, 5 minutes, 30 minutes, 2 hours. Add random jitter, so thousands of retries don't hit a recovering server at the same second, and honor a Retry-After header when the receiver sends one.

5. Keep failures recoverable

After the last attempt, an event should not disappear. Keep it as a dead delivery with every attempt's status, response and timing, alert someone, and make it possible to replay it once the endpoint is fixed, one by one or in bulk.

How HookStead handles this for you

• Events are stored before the provider gets its 2xx, and deliveries are retried from the database, so a crashed worker or Redis outage loses nothing.
• Retries use exponential backoff with jitter, honor Retry-After, and stop at your plan's attempt limit (up to 15 on Team). See pricing.
• Every delivery carries a stable webhook-id for your idempotency table, the same on every retry and replay.
• Alerts email you when a destination starts failing and again when it recovers.
• Retry, replay and bulk retry from the dashboard or the API.

Read next: Webhook security and HMAC verification, or set up your first source with the user guide.