Webhooks

Webhook Gateway Operations

Manage the joelclaw webhook gateway — add providers, debug delivery, register with external services.

Architecture

External Service → Tailscale Funnel :443 → Worker :3111 → /webhooks/:provider
  → verifySignature() → normalizePayload() → (queue pilot or direct Inngest event) → notify function → gateway

• ADR-0048: Webhook Gateway for External Service Integration
• Gateway skill: Use gateway push/gateway test patterns for delivery checks

Current Providers

Current ADR-0217 pilot note: when QUEUE_PILOTS=github, the webhook gateway enqueues normalized github/workflow_run.completed events into the shared Redis queue instead of posting them directly to Inngest. The Restate drainer then forwards the concrete event name github/workflow_run.completed. github/package.published still goes direct.

Adding a New Provider

See references/new-provider-checklist.md for the full 8-step checklist.

Quick summary: 1. Create providers/{name}.ts implementing WebhookProvider interface 2. Register in server.ts provider map 3. Create Inngest notify function(s) in functions/{name}-notify.ts 4. Export from functions/index.ts and add to functions/index.host.ts (or index.cluster.ts when cluster-owned) 5. Store webhook secret in agent-secrets → add lease to start.sh 6. Deploy: joelclaw inngest restart-worker --register 7. Register webhook URL with external service 8. Verify E2E with curl + real webhook

Key Files

Debugging Webhooks

Check if webhook is arriving

# Watch worker logs
joelclaw logs worker --follow --grep webhook

# Or directly
curl -s http://localhost:3111/ | jq .webhooks
# → { endpoint: "/webhooks/:provider", providers: ["todoist", "front", "vercel"] }

Signature verification failures

# Test with manual HMAC (SHA1 example for Vercel)
SECRET="your-webhook-secret"
BODY='{"type":"test-webhook","payload":{}}'
HMAC=$(echo -n "$BODY" | openssl dgst -sha1 -hmac "$SECRET" -binary | xxd -p)
curl -X POST http://localhost:3111/webhooks/vercel \
  -H "Content-Type: application/json" \
  -H "x-vercel-signature: $HMAC" \
  -d "$BODY"

Common failures:

• Wrong secret — Todoist uses client_secret (not "Verification token"), Vercel uses the secret from webhook creation, Front uses the rules-based secret
• Encoding mismatch — Todoist = base64, Vercel = hex, Front = base64 over compact JSON
• Body mutation — Caddy/proxy rewrites body. Use Tailscale Funnel → worker directly
• Rate limited — 20 auth failures per IP per minute. Wait or restart worker

Check Inngest received events

joelclaw runs --count 5
# Look for vercel-deploy-*, todoist-*, front-* function runs

Gateway not receiving notifications

joelclaw gateway status
joelclaw gateway events   # Peek pending events

Registering Webhooks with Services

Vercel (Pro/Enterprise required)

# Via Vercel dashboard: Settings → Webhooks → Create
# Or via API:
VERCEL_TOKEN="your-api-token"
curl -X POST "https://api.vercel.com/v1/webhooks" \
  -H "Authorization: Bearer $VERCEL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://panda.tail7af24.ts.net/webhooks/vercel",
    "events": ["deployment.created", "deployment.succeeded", "deployment.error", "deployment.canceled"]
  }'

The response includes a secret — store it: secrets add vercel_webhook_secret --value "..."

GitHub

Set up via repo Settings → Webhooks:

• URL: https://panda.tail7af24.ts.net/webhooks/github
• Content type: application/json
• Secret: generate one, store as github_webhook_secret
• Events: push, pull_request, deployment_status, or "Send me everything"

Todoist

Already configured via Todoist App Console → Webhooks tab. Uses client_secret as HMAC key (not the "Verification token").

Front

Already configured via Front Rules → "Trigger a webhook" action. Rules webhooks scope to specific inboxes at the rule layer.

Signature Algorithms by Provider

Gotchas

• Caddy drops Funnel POST bodies — Point Tailscale Funnel directly at worker :3111, not through Caddy
• `joelclaw inngest restart-worker --register` after deploy — ensures restart + registration in one step
• Vercel webhooks are Pro/Enterprise only — free plans cannot create account-level webhooks
• Front has TWO webhook types — App-level (SHA256, challenges) vs Rules-based (SHA1, no challenges). We use Rules-based
• agent-secrets v0.5.0+ — raw output is default, don't pass --raw flag
• Idempotency keys on all events — safe to receive duplicates from retry-happy providers

Adding a New Webhook Provider — Full Checklist

Step-by-step for wiring a new external service's webhooks into the joelclaw gateway.

1. Create Provider Adapter

packages/system-bus/src/webhooks/providers/{provider}.ts

Implement the WebhookProvider interface from ../types.ts:

import { createHmac, timingSafeEqual } from "node:crypto";
import type { WebhookProvider, NormalizedEvent } from "../types";

/** Map provider event types → normalized event names */
const EVENT_MAP: Record<string, string> = {
  "provider.event_type": "normalized.name",
};

function getSecret(): string {
  const secret = process.env.MY_PROVIDER_WEBHOOK_SECRET;
  if (!secret) throw new Error("MY_PROVIDER_WEBHOOK_SECRET env var required");
  return secret;
}

export const myProvider: WebhookProvider = {
  id: "my-provider",
  eventPrefix: "my-provider",

  verifySignature(rawBody: string, headers: Record<string, string>): boolean {
    const signature = headers["x-my-provider-signature"];
    if (!signature) return false;

    const secret = getSecret();
    // Adjust algorithm (sha1/sha256) and encoding (hex/base64) per provider docs
    const computed = createHmac("sha256", secret).update(rawBody).digest("hex");

    try {
      return timingSafeEqual(
        Buffer.from(signature, "hex"),
        Buffer.from(computed, "hex"),
      );
    } catch {
      return false;
    }
  },

  normalizePayload(
    body: Record<string, unknown>,
    _headers: Record<string, string>,
  ): NormalizedEvent[] {
    const type = body.type as string | undefined;
    if (!type) return [];

    const mappedName = EVENT_MAP[type];
    if (!mappedName) return [];

    return [{
      name: mappedName,
      data: {
        // Extract meaningful fields — don't just pass the raw body
        id: String(body.id ?? ""),
        // ... provider-specific fields
      },
      idempotencyKey: `my-provider-${type}-${body.id ?? Date.now()}`,
    }];
  },
};

Signature Algorithm Cheat Sheet

⚠️ Gotcha: Some providers sign a transformed body (Front signs compact JSON, Stripe prepends timestamp). Always read the provider's docs carefully.

2. Register Route

In src/webhooks/server.ts:

import { myProvider } from "./providers/my-provider";
providers.set(myProvider.id, myProvider);

Webhook URL becomes: POST /webhooks/my-provider

3. Create Notify Functions

src/inngest/functions/{provider}-notify.ts

Follow the pattern from todoist-notify.ts:

import { inngest } from "../client";
import type { GatewayContext } from "../middleware/gateway";

export const myProviderEventNotify = inngest.createFunction(
  { id: "my-provider-event-notify", name: "MyProvider → Gateway: Event" },
  { event: "my-provider/event.type" },
  async ({ event, step, ...rest }) => {
    const gateway = (rest as any).gateway as GatewayContext | undefined;

    // Optional: enrich with API calls
    const context = await step.run("enrich-context", async () => {
      // Fetch additional data from provider API if needed
      return { /* enriched fields */ };
    });

    const agentPrompt = await step.run("build-prompt", () => {
      return [
        `## 📌 Provider Event`,
        "",
        `**Summary**: ${event.data.summary}`,
        `Context and next-action guidance for the agent.`,
      ].join("\n");
    });

    const result = await step.run("notify-gateway", async () => {
      if (!gateway) return { pushed: false, reason: "no gateway context" };

      return await gateway.notify("my-provider.event.type", {
        prompt: agentPrompt,
        ...event.data,
        ...context,
      });
    });

    return {
      status: result.pushed ? "notified" : "skipped",
      result,
    };
  }
);

3-step pattern (enrich → build-prompt → notify)

All notify functions follow this pattern: 1. enrich-context — optional API call to get extra data the webhook didn't include 2. build-prompt — create human-readable markdown prompt for the gateway agent 3. notify-gateway — push to gateway with structured data + prompt

4. Register Functions

Export from src/inngest/functions/index.ts:

export { myProviderEventNotify } from "./my-provider-notify";

Add to the active role list in src/inngest/functions/index.host.ts (or index.cluster.ts for cluster-owned functions):

// in src/inngest/functions/index.host.ts
import { myProviderEventNotify } from "./my-provider-notify";

export const hostFunctionDefinitions = [
  // ...existing functions
  myProviderEventNotify,
];

Update the health endpoint's webhooks.providers and events sections in serve.ts when introducing a new provider/event family.

5. Store Secrets

# Add the webhook signing secret
secrets add my_provider_webhook_secret --value "the-actual-secret"

# Optional: API token for enrichment
secrets add my_provider_api_token --value "the-api-token"

Add to ~/Code/joelhooks/joelclaw/packages/system-bus/start.sh:

MY_SECRET=$(secrets lease my_provider_webhook_secret --ttl 24h 2>/dev/null)
if [ -n "$MY_SECRET" ]; then
  export MY_PROVIDER_WEBHOOK_SECRET="$MY_SECRET"
else
  echo "WARNING: Failed to lease my_provider_webhook_secret" >&2
fi

6. Deploy

# Restart + register in one command (reloads code + re-leases secrets)
joelclaw inngest restart-worker --register

7. Register Webhook URL with External Service

Public URL pattern: https://panda.tail7af24.ts.net/webhooks/{provider}

Tailscale Funnel must be configured:

# Already running for existing webhooks — verify:
tailscale serve status
# Should show :443 → http://localhost:3111

Each service has its own webhook registration process:

• Vercel: Dashboard → Settings → Webhooks → Create (or REST API)
• GitHub: Repo → Settings → Webhooks → Add webhook
• Todoist: App Console → Webhooks tab
• Front: Rules → "Trigger a webhook" action
• Stripe: Dashboard → Developers → Webhooks → Add endpoint

When creating the webhook, you'll get a signing secret — store it immediately with secrets add.

8. Verify E2E

# Manual test with correct HMAC
SECRET="your-webhook-secret"
BODY='{"type":"test","id":"test-123"}'

# SHA1 hex (Vercel):
HMAC=$(echo -n "$BODY" | openssl dgst -sha1 -hmac "$SECRET" -binary | xxd -p)
curl -X POST http://localhost:3111/webhooks/my-provider \
  -H "Content-Type: application/json" \
  -H "x-my-provider-signature: $HMAC" \
  -d "$BODY"

# SHA256 base64 (Todoist):
HMAC=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | base64)
curl -X POST http://localhost:3111/webhooks/my-provider \
  -H "Content-Type: application/json" \
  -H "x-my-provider-hmac-sha256: $HMAC" \
  -d "$BODY"

# Check Inngest received events
joelclaw runs --count 3

# Check gateway got the notification
joelclaw gateway events

Common Issues