Webhook Architecture

Webhook Architecture

Overview

Webhooks are HTTP callbacks used by commerce platforms (Shopify, Stripe) to push real-time event notifications to your application. Reliable webhook infrastructure requires: HMAC signature verification to prevent spoofed events, idempotent handlers that tolerate duplicate delivery, exponential backoff retry logic, and a dead-letter queue for events that exhaust all retries. This skill covers building a reliable webhook receiver and, for custom platforms, a webhook sender using the Outbox Pattern.

When to Use This Skill

• When receiving webhooks from Shopify, Stripe, Square, or other platforms
• When debugging missed events or duplicate processing caused by webhook delivery issues
• When building a commerce platform or app that needs to notify external systems of events
• When designing event-driven architecture between commerce microservices
• When setting up webhook fanout (single event delivered to multiple consumers)

Core Instructions

Step 1: Determine your platform and what webhooks you need to handle

Step 2: Platform-specific webhook setup

---

Shopify

Register webhooks via the Shopify admin:

1. Go to Settings → Notifications and scroll to Webhooks 2. Click Create webhook 3. Select the event topic (e.g., Order creation) and enter your endpoint URL 4. Choose JSON as the format

Get your webhook secret for HMAC verification:

The secret is shown when you create the webhook. Store it as an environment variable — Shopify signs each webhook with this secret using HMAC-SHA256.

For apps using the Admin API, register webhooks programmatically:

const res = await fetch(`https://${shopDomain}/admin/api/2025-01/webhooks.json`, {
  method: 'POST',
  headers: { 'X-Shopify-Access-Token': accessToken, 'Content-Type': 'application/json' },
  body: JSON.stringify({ webhook: {
    topic: 'orders/create',
    address: `${process.env.APP_URL}/api/webhooks/shopify/order-created`,
    format: 'json',
  }}),
});

---

WooCommerce

Use WooCommerce's built-in webhooks:

1. Go to WooCommerce → Settings → Advanced → Webhooks 2. Click Add webhook 3. Set Name, Status: Active, Topic (e.g., Order Created), and your Delivery URL 4. The Secret field generates an HMAC-SHA256 signature for each delivery — copy it for your endpoint's verification

Or use WP Webhooks plugin for more control:

1. Install WP Webhooks (free, wordpress.org) for advanced trigger conditions and payload customization 2. Go to Settings → WP Webhooks and configure triggers for WooCommerce order events 3. WP Webhooks supports retry logic and delivery logs out of the box

---

Custom / Headless

For custom storefronts, implement both reliable receiving (for incoming webhooks from Stripe, Shopify, etc.) and reliable sending (Outbox Pattern for notifying your own integrations).

HMAC signature verification (Shopify and generic):

// lib/webhooks/verify.ts
import { createHmac, timingSafeEqual } from 'node:crypto';

export function verifyShopifyWebhook(rawBody: Buffer, hmacHeader: string, secret: string): boolean {
  const expected = createHmac('sha256', secret).update(rawBody).digest('base64');
  const received = Buffer.from(hmacHeader);
  const expectedBuffer = Buffer.from(expected);
  if (received.length !== expectedBuffer.length) return false;
  return timingSafeEqual(received, expectedBuffer);
}

export function verifyStripeWebhook(rawBody: Buffer, signatureHeader: string, secret: string): boolean {
  const parts = signatureHeader.split(',');
  const timestamp = parts.find(p => p.startsWith('t='))?.replace('t=', '');
  const v1 = parts.find(p => p.startsWith('v1='))?.replace('v1=', '');
  if (!timestamp || !v1) return false;
  // Reject events older than 5 minutes (replay attack protection)
  if (Math.abs(Date.now() / 1000 - parseInt(timestamp)) > 300) return false;
  const expected = createHmac('sha256', secret)
    .update(`${timestamp}.${rawBody.toString('utf8')}`)
    .digest('hex');
  return timingSafeEqual(Buffer.from(v1), Buffer.from(expected));
}

Idempotent webhook receiver — deduplicate using the platform's event ID:

// app/api/webhooks/shopify/route.ts
export async function POST(req: NextRequest) {
  const rawBody = Buffer.from(await req.arrayBuffer());
  const hmac = req.headers.get('x-shopify-hmac-sha256') ?? '';
  const topic = req.headers.get('x-shopify-topic') ?? '';
  const eventId = req.headers.get('x-shopify-webhook-id') ?? '';

  // 1. Verify signature — reject invalid requests immediately
  if (!verifyShopifyWebhook(rawBody, hmac, process.env.SHOPIFY_WEBHOOK_SECRET!)) {
    return NextResponse.json({ error: 'Invalid signature' }, { status: 401 });
  }

  // 2. Idempotency check — deduplicate by event ID
  const alreadyProcessed = await db.processedWebhooks.exists(eventId);
  if (alreadyProcessed) return NextResponse.json({ received: true, status: 'already_processed' });

  // 3. Mark as received BEFORE processing (prevents duplicate on concurrent delivery)
  await db.processedWebhooks.insert({ id: eventId, topic, receivedAt: new Date(), status: 'processing' });

  // 4. Return 200 immediately, process asynchronously
  processWebhookAsync(topic, rawBody, eventId); // Don't await — return fast

  return NextResponse.json({ received: true });
}

async function processWebhookAsync(topic: string, rawBody: Buffer, eventId: string) {
  try {
    const payload = JSON.parse(rawBody.toString('utf8'));
    switch (topic) {
      case 'orders/create': await importOrder(payload); break;
      case 'orders/cancelled': await cancelOrder(payload.id); break;
      case 'inventory_levels/update': await syncInventory(payload); break;
    }
    await db.processedWebhooks.update(eventId, { status: 'processed', processedAt: new Date() });
  } catch (err: any) {
    await db.processedWebhooks.update(eventId, { status: 'failed', error: err.message });
  }
}

Outbox Pattern for reliable webhook sending — guarantees at-least-once delivery even if your sender crashes:

// lib/webhooks/outbox.ts
// Write to outbox in the SAME transaction as the business event
export async function publishEvent(trx: Transaction, eventType: string, payload: object) {
  await trx.webhookOutbox.insert({
    id: crypto.randomUUID(),
    eventType,
    payload: JSON.stringify(payload),
    status: 'pending',
    attempts: 0,
    nextRetryAt: new Date(),
  });
}

// Outbox poller — runs every 10 seconds, separate from your main app
export async function processOutbox() {
  const pending = await db.webhookOutbox.findPending({ status: ['pending', 'retrying'], nextRetryAt: { $lte: new Date() }, limit: 100 });
  for (const event of pending) await deliverEvent(event);
}

// Retry schedule: 1min, 5min, 30min, 2hr, 8hr → DLQ after 5 attempts
const RETRY_DELAYS_MS = [60_000, 300_000, 1_800_000, 7_200_000, 28_800_000];

async function handleDeliveryFailure(event: OutboxEvent, error: string) {
  const nextAttempts = event.attempts + 1;

  if (nextAttempts >= RETRY_DELAYS_MS.length) {
    await db.webhookOutbox.update(event.id, { status: 'dead_letter', lastError: error });
    await db.webhookDeadLetters.insert({ eventId: event.id, failedAt: new Date(), reason: error });
    await alertOpsTeam(`Webhook permanently failed after ${nextAttempts} attempts`, { eventType: event.eventType, error });
  } else {
    const nextRetryAt = new Date(Date.now() + RETRY_DELAYS_MS[nextAttempts - 1]);
    await db.webhookOutbox.update(event.id, { status: 'retrying', attempts: nextAttempts, nextRetryAt, lastError: error });
  }
}

Replay dead-letter events (after fixing the subscriber endpoint):

export async function replayDeadLetter(deadLetterId: string) {
  const deadLetter = await db.webhookDeadLetters.findById(deadLetterId);
  await db.webhookOutbox.update(deadLetter.eventId, {
    status: 'pending', attempts: 0, nextRetryAt: new Date(),
  });
  await db.webhookDeadLetters.update(deadLetterId, { replayedAt: new Date() });
}

Best Practices

• Always return 2xx immediately — a slow webhook handler blocks delivery and may cause the sender to time out and retry; enqueue events on receipt and process asynchronously
• Use the Outbox Pattern for reliable sending — writing to an outbox table in the same DB transaction as your domain event guarantees at-least-once delivery even if your webhook sender crashes
• Make handlers idempotent — use the event's unique ID to deduplicate; "at-least-once" delivery is the standard for all webhook platforms; you must tolerate receiving the same event twice
• Log every delivery attempt — store delivery attempts with response codes, timing, and errors; this is essential for debugging and provides audit evidence for compliance
• Implement a dead-letter queue with alerting — events that exhaust retries need human intervention; alert via Slack/PagerDuty and provide a replay mechanism

Common Pitfalls

Related Skills

• @analytics-integration
• @erp-integration
• @marketplace-connectors
• @monitoring-alerting-commerce

Reliable Webhook Delivery System for a SaaS Platform

Problem/Feature Description

You are building the webhook delivery infrastructure for a multi-tenant SaaS platform. Partners register endpoints to receive events (such as invoice.paid, subscription.cancelled, user.created) whenever relevant things happen in the platform. The previous implementation called partner endpoints directly during the request lifecycle — this caused partner timeouts to slow down the main application and meant events were silently lost whenever a partner's server was down during the moment an event fired.

The platform team wants a robust, at-least-once delivery system. Events must survive application crashes between generation and delivery, failed deliveries must be retried automatically with increasing delays, and events that ultimately cannot be delivered must be moved to a dead-letter queue where an alert is triggered and an operator can manually re-queue them later. The system must also correctly authenticate outbound requests so partners can verify that payloads come from your platform.

Output Specification

Produce the following files:

• src/webhooks/outbox.ts — The outbox writer (called from within business transactions) and the outbox poller that delivers pending events
• src/webhooks/delivery.ts — HTTP delivery logic including signing, headers, and timeout handling
• src/webhooks/retry.ts — Retry scheduling and dead-letter queue promotion logic
• src/webhooks/schema.sql — Database schema for all required tables
• src/webhooks/replay.ts — Admin function to re-queue a dead-letter event for delivery
• README.md — Architecture overview explaining the reliability guarantees and the retry schedule

The implementation should be TypeScript. You do not need a running server or real database — focus on the logic, structure, and correctness of the code. Stub out database calls as needed (e.g., db.webhookOutbox.findPending(...)) but implement the full business logic around them.

Stripe Webhook Handler for Payment Processing

Problem/Feature Description

Your company has integrated Stripe as its payment processor. The engineering team has been getting reports of duplicate order fulfillments and failed payments being silently ignored. Investigation revealed that the webhook handler doesn't guard against duplicate deliveries (Stripe retries for up to 72 hours), and the current HMAC verification implementation is vulnerable to timing-based attacks. Additionally, security review flagged that the system isn't rejecting replayed events — a credential that was briefly exposed could allow an attacker to replay captured webhook payloads indefinitely.

Your task is to implement a production-quality Stripe webhook receiver endpoint in TypeScript. The handler must correctly verify Stripe's HMAC signature format, protect against replay attacks, deduplicate repeated event deliveries, and respond correctly so that Stripe stops retrying events even when internal processing fails.

Output Specification

Produce the following files:

• src/webhooks/verify.ts — HMAC verification utilities
• src/webhooks/stripe-handler.ts — The webhook handler endpoint logic (framework-agnostic: export a function that accepts rawBody: Buffer, headers: Record<string, string>, and returns {status: number, body: object})
• src/webhooks/schema.sql — SQL schema for any tables needed to support the implementation
• README.md — Brief explanation of the security properties provided by your implementation and why each design decision was made

The implementation should be demonstrably correct TypeScript. You do not need a running server — focus on the handler logic and its correctness.

Partner Webhook Subscription API

Problem/Feature Description

Your company is launching a partner integration program that allows third-party developers to subscribe to platform events and receive real-time notifications at their own endpoints. The developer relations team has asked for a self-service API where partners can register, update, and manage their webhook endpoints without requiring manual intervention from your team.

The security team has a firm requirement: each webhook subscription must be independently secured with a unique credential that partners use to verify that deliveries originate from your platform. This credential must be generated securely and may never be retrieved after creation — if a partner loses it, they must rotate or recreate the subscription. The team also wants the system to be observable: the support team needs to be notified automatically if the dead-letter queue for any subscriber grows large enough to indicate a systematic integration failure, rather than finding out from a frustrated partner days later.

Your task is to implement the webhook subscription management API and the DLQ monitoring logic.

Output Specification

Produce the following files:

• src/webhooks/subscriptions.ts — The subscription creation endpoint logic (framework-agnostic: export a function accepting request data and returning a response object)
• src/webhooks/monitoring.ts — DLQ monitoring function that checks queue depth and sends an alert when a threshold is exceeded
• src/webhooks/schema.sql — Database schema for webhook subscriptions and any supporting tables
• README.md — Document the API contract, the security model for signing secrets, and the monitoring behaviour