Four webhook patterns that survive production

Webhooks look simple until production. Here are the four patterns that separate integrations we never hear about from the ones that page us at 3am.

First, idempotency. Every delivery is at-least-once, which means you will receive the same event twice eventually. Store the event ID and make the handler a no-op if you have seen it. This one change removes most duplicate-fulfilment bugs.

Second, respond fast and process later. Acknowledge with a 200 in under a second and push the work onto a queue. Handlers that call three other APIs inline are the ones that time out and trigger retries, which triggers duplicates, which triggers the first problem again.

Third, do not assume ordering. `subscription.updated` can arrive before `subscription.created` under retry. Reconcile against current state via the API rather than replaying a sequence. Fourth, verify the signature on every request, including in staging — a webhook endpoint is a public URL, and treating it as trusted is how test data ends up in production.