Builder Smoke Test

Builder Smoke Test

End-to-end smoke testing of the Agent Builder feature set against a hermetic project the skill scaffolds at ~/mastra-builder-smoke-tests/builder-smoke (configurable). The project links to the current worktree via pnpm link: overrides, so changes to packages under packages/, stores/, auth/, channels/, observability/, browser/, and client-sdks/ take effect on the next mastra dev restart.

This skill is for branch QA — it complements the release-time mastra-smoke-test. It exercises the Builder EE surface (stored entities, RBAC, registry, infra, channels) using a minimal, predictable project rather than the kitchen-sink examples/agent.

⚠️ Mandatory Test Checklist

Use `task_write` to track progress. Run ALL sections unless --test or --scope narrows the run.

Do not skip sections unless you hit an actual blocker. "Seemed complex" or "I'll come back to it" are not valid reasons. Attempt every step — only stop when you literally cannot proceed. Report what you tried and what blocked you.

Execution flow

1. Confirm the project directory. Before scaffolding, ask the user where they want $PROJECT_DIR to live. Offer the default (~/mastra-builder-smoke-tests/builder-smoke) as a suggestion. Skip the question if they already passed --dir or have $BUILDER_SMOKE_TEST_DIR exported. See references/setup.md step 0. 2. Read the reference file for each section you're about to run. 3. Under `--auth on`, extract the session cookie before running any other section. The WorkOS cookie is httpOnly, so curl cannot mint it and document.cookie cannot read it. The scaffold ships a debug route at GET /smoke-test/cookie gated by SMOKE_TEST_COOKIE_LEAK=1. Follow the "Extracting the session cookie for curl (auth on)" section below before touching any auth-on endpoint. Do not pivot to UI-only testing because curl is "blocked" — the cookie route is the unblock path. 4. Seed non-owner data after the server has booted at least once. A fresh scaffold has no skills authored by anyone other than the test user, which makes non-owner / Library Copy / non-owner visibility / non-admin stars flows untestable. Run bash .claude/skills/builder-smoke-test/scripts/seed-multi-user.sh (or with --dir $PROJECT_DIR) before sections 6 (Skills), 7 (Registry), and 10 (Stars). The script is idempotent and bypasses RBAC by writing directly to libsql, so it works regardless of --auth mode or current role. Do not mark non-owner steps as "blocked" without running this first. 5. Execute the steps — use curl for API checks (with -H "Cookie: $COOKIE" under --auth on), whichever browser tool the harness has wired up (Stagehand, Chrome MCP, etc.) for UI checks. 6. Record results in the summary table. 7. Mark the section complete with task_write before moving to the next.

Partial testing (--test)

If --test is provided:

1. Always run Setup. 2. Run only the specified section(s). 3. Skip everything else.

Example: --test skills,registry,agents → Setup + Skills + Registry + Agents.

Scope shortcuts (--scope)

--scope runs a curated group of related sections. Setup is always implied.

--scope and --test can be combined; the union is run.

Usage

# Full smoke (interactive)
/builder-smoke-test

# Specific sections
/builder-smoke-test --test workspace,skills
/builder-smoke-test --test agents,favorites
/builder-smoke-test --test reconciliation
/builder-smoke-test --test ui

# Scope shortcuts
/builder-smoke-test --scope rbac
/builder-smoke-test --scope skills
/builder-smoke-test --scope quick

# Force auth on / off (otherwise auto-detected from WORKOS_* env vars)
/builder-smoke-test --auth on
/builder-smoke-test --auth off

# Run auth-on as a non-admin role (must match the logged-in user's actual role)
/builder-smoke-test --auth on --role viewer
/builder-smoke-test --auth on --role member

# Skip the browser pass (API-only run)
/builder-smoke-test --skip-browser

Parameters

If --auth auto and no WorkOS env vars are present, the Auth section is auto-skipped and reported as ⏭️ Skipped (no WORKOS_* env vars).

Canonical order

When running multiple sections, execute them in the order shown in the section table (1 → 15). The order is intentional:

• Setup must run first — preflight + readiness probe gate every later

section.

• Workspace / Reconciliation / Defaults / Model Policy establish that

the server's view of the project matches what the rest of the run assumes. Run them before any CRUD pass.

• Skills → Registry → Agents → Pickers → Stars is a build-up: agents

reference skills, pickers depend on the entities created above.

• Permissions / Infrastructure / Channels / UI are read-mostly

inspections that benefit from existing entities.

• Auth runs last because it requires restarting mastra dev with a

different .env.

If --test or --scope narrows the run, keep the relative order — just skip the sections that fall outside the selection.

Required vs optional reference tiers

References fall into three tiers; an agent should treat them accordingly:

• Required (every run): setup.md. Any failure here blocks the rest

of the run.

• Standard (default tiers for `full`, `quick`, scope shortcuts):

workspace.md, skills.md, agents.md, favorites.md, ui.md (core), auth.md when --auth on.

• **Extended (only when explicitly selected via --test/--scope or

the matching code surface changed):** reconciliation.md, defaults.md, model-policy.md, registry.md, picker-allowlist.md, permissions.md, infrastructure.md, channels.md, ui.md extended tier.

When skipping an extended section, mark it ⏭️ Skipped (not in scope) in the result table — don't silently omit it.

Cleanup

The scaffold is a self-contained throwaway directory at $PROJECT_DIR. All fixture state (workspaces, agents, skills, libsql DB, .mastra/workspace files) lives inside it. The smoke test never writes to anything outside $PROJECT_DIR (other than the dev server it runs).

At the end of every run:

1. Stop the dev server (kill $(lsof -i :4111 -sTCP:LISTEN -t) or foreground Ctrl-C). 2. Choose how to dispose of fixture state:

• Reuse: leave $PROJECT_DIR in place. The next run can pass

--reuse (or --skip-scaffold to preflight) and pick up where this one left off. Fastest for iterating.

• Reset: rm -rf "$PROJECT_DIR" (or re-run scripts/scaffold.sh

without --reuse). Cheapest way to get back to a known-clean state. Don't bother per-entity DELETE — the directory IS the state. 3. If a section bailed mid-flight (assertion failure, network error), record the partial state in the report's Issues section so the next run knows what to expect.

Per-entity DELETE calls are only needed when a specific section explicitly tests DELETE behavior (those sections include the DELETE step inline). Otherwise the throwaway-directory model handles cleanup.

Never leave the dev server running on :4111 after the report is filed — it blocks future runs.

Prerequisites

• Working tree on the agent-builder feature branch (or any branch you want to QA).
• pnpm (10.x) and node on $PATH. The scaffold uses pnpm install --ignore-workspace inside the project dir so the repo-level workspace doesn't interfere.
• An OPENAI_API_KEY. Supply via --openai-key, export OPENAI_API_KEY in the shell, or let the scaffold prompt for it.
• (Optional) WorkOS credentials for --auth on runs: --workos-api-key, --workos-client-id, --workos-organization-id.
• Whichever browser MCP/tool the harness has access to. If none is available, run with --skip-browser and report UI as ⏭️ Skipped (no browser tool).

Project layout (scaffolded for you)

$PROJECT_DIR/                                    ← see "Project dir resolution" below
├── package.json                                 ← pnpm overrides → link:<worktree>/packages/*
├── tsconfig.json
├── .env                                         ← OPENAI_API_KEY (+ AUTH_PROVIDER + WORKOS_* on auth-on)
└── src/mastra/
    ├── index.ts                                 ← single Mastra instance, reads exported bindings from auth.ts
    ├── auth.ts                                  ← top-level switch(process.env.AUTH_PROVIDER); no-op when unset
    ├── agents/index.ts                          ← weather-agent (gpt-4o-mini)
    ├── tools/index.ts                           ← weather-info tool
    └── workflows/index.ts                       ← greet-workflow

The .env is the only thing that flips auth on/off — the same src/mastra/index.ts runs in both modes. Re-run scripts/scaffold.sh with or without --workos-* to switch.

Project dir resolution

$PROJECT_DIR is determined by every script (scaffold, preflight, wait-for-server) using this order:

1. --dir <path> flag 2. BUILDER_SMOKE_TEST_DIR env var (e.g. export BUILDER_SMOKE_TEST_DIR=~/code/builder-smoke) 3. ~/mastra-builder-smoke-tests/builder-smoke (default)

For a long-lived setup, exporting BUILDER_SMOKE_TEST_DIR once in your shell rc is the lowest-friction option — every script picks it up automatically.

Running scripts (cwd matters)

All scripts under .claude/skills/builder-smoke-test/scripts/ resolve the worktree root from their own location. They can be invoked from anywhere, but conventionally the repo root.

Invoke them as bash .claude/skills/builder-smoke-test/scripts/<name>.sh. Don't cd into scripts/ first — relative path resolution will break.

pnpm mastra:dev must be run from $PROJECT_DIR (where the scaffolded package.json is).

How mastra dev reads env (important)

mastra dev loads $PROJECT_DIR/.env via dotenv and unconditionally overwrites `process.env` with whatever's there (packages/cli/src/commands/dev/dev.ts ~line 384). Practical consequences:

• `.env` is the source of truth for the running server. Inline overrides like AUTH_PROVIDER= pnpm mastra:dev are silently clobbered.
• Shell-only vars survive only if `.env` has no entry for the same key. Re-running scripts/scaffold.sh always overwrites .env, so to toggle modes, re-scaffold.
• The auth mode the server actually runs in is determined by `.env` alone. A globally exported AUTH_PROVIDER=workos in your shell does NOT enable WorkOS auth in the server if .env doesn't have it — but it WILL leak into anything else this process runs, which is its own kind of confusing. Preflight flags this case.

Auth modes

Two states matter:

• auth off — AUTH_PROVIDER is absent (or blank) in $PROJECT_DIR/.env. No WorkOS, no RBAC, no FGA. This is the state for the auth-off run.
• auth on — AUTH_PROVIDER=workos plus WORKOS_API_KEY, WORKOS_CLIENT_ID, WORKOS_ORGANIZATION_ID all present in $PROJECT_DIR/.env. WorkOS authentication + role-based access + per-resource FGA all engage. This is the state for the auth-on runs. FGA is wired through the WorkOS auth provider — it can't be disabled independently.

To switch modes, re-run the scaffold with or without the --workos-* flags; that's faster and safer than hand-editing .env.

Detection: run preflight before each section

# Scaffold (or refresh) the project and assert the auth-off baseline:
bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect off \
  --openai-key "$OPENAI_API_KEY"

# Scaffold an auth-on project (re-runs scaffold with WorkOS keys, asserts auth on):
bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect on \
  --openai-key "$OPENAI_API_KEY" \
  --workos-api-key "$WORKOS_API_KEY" \
  --workos-client-id "$WORKOS_CLIENT_ID" \
  --workos-organization-id "$WORKOS_ORGANIZATION_ID"

Preflight chains scaffold.sh followed by validation checks (project exists with node_modules/@mastra/core, $PROJECT_DIR/.env has OPENAI_API_KEY, optional WorkOS keys present when --expect on, and auth mode matches --expect). Each failure prints a stable error code; this table tells the agent what to do.

Resolving missing env vars

If scaffold.sh or preflight.sh reports a missing OPENAI_API_KEY or WORKOS_* var, the agent must not silently source any rc file. Instead, work down this list and stop at the first one that resolves:

1. Check whether the var is already in the process env you can see (echo "${OPENAI_API_KEY:-<unset>}"). If yes, re-run scaffold with --openai-key "$OPENAI_API_KEY" (and equivalent for WorkOS). 2. Check whether the var is in $PROJECT_DIR/.env from a prior run (grep -E "^(OPENAI_API_KEY|WORKOS_)" "$PROJECT_DIR/.env" 2>/dev/null). If yes, you can pass --reuse to the next scaffold call. 3. If neither, look for rc files that exist on disk. Common candidates: ~/.zshrc, ~/.bashrc, ~/.zshenv, ~/.profile, ~/.env.global, and any project-local .env you find. Use ls -1 (or test -f) to confirm before listing — don't fabricate paths. 4. Ask the user in one message: "Can you paste the value(s), or give me permission to source one of these files?" Include the list of files that actually exist. 5. Only after the user explicitly approves a specific file, source it in a subshell and rerun preflight with the inherited env. Pattern:

   # auth off
   zsh -c 'source <approved-file> && bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect off --reuse'

   # auth on (preflight auto-picks WORKOS_API_KEY / WORKOS_CLIENT_ID / WORKOS_ORGANIZATION_ID from the sourced env)
   zsh -c 'source <approved-file> && bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect on --reuse'

Use bash -c instead of zsh -c if the approved file is a bashrc.

6. Never write the secret value back into any rc file, never export it into the user's interactive shell, and never echo it back in chat in full. Refer to it as <your-openai-key> once you've used it.

`.env` policy: the scaffold owns $PROJECT_DIR/.env. Re-running scaffold overwrites it. Do not hand-edit the scaffolded .env; instead, re-run scaffold with different flags. (The skill never edits .env files outside $PROJECT_DIR.)

Extracting the session cookie for curl (auth on)

The WorkOS session cookie is httpOnly, so document.cookie and Stagehand's extract cannot read it from a normal page. To hit authenticated endpoints from curl after a browser SSO login, the scaffold exposes a tiny debug route gated by an env var:

1. Add SMOKE_TEST_COOKIE_LEAK=1 to $PROJECT_DIR/.env (single line append; the scaffold leaves this var alone on re-run as long as the file already exists). 2. Restart mastra dev so the new env is picked up. 3. Sign in once in the Stagehand browser (stagehand_navigate to http://localhost:4111, complete WorkOS SSO). 4. From the same browser tab, navigate to http://localhost:4111/smoke-test/cookie and use stagehand_extract to read the page body. The page is a single text/plain line containing the request's Cookie header verbatim (e.g. wos_session=…). 5. Export it once: export COOKIE='<the-string-from-step-4>'. From here on, every authenticated curl is curl -H "Cookie: $COOKIE" "$BASE/…".

The route is only registered when `SMOKE_TEST_COOKIE_LEAK=1` and is intentionally insecure — never enable it in a real project. The WORKOS_COOKIE_PASSWORD written by the scaffold is derived from $PROJECT_DIR, so the cookie value stays valid across mastra dev restarts within the same scaffold; you only need to repeat step 4 if you re-scaffold to a new directory.

`/smoke-test/cookie` returns 404? Always an env-ordering issue. The apiRoutes list is built once when mastra dev boots from process.env.SMOKE_TEST_COOKIE_LEAK. The flag has to be in .env before the boot — adding it after start has no effect until you restart. If you see a 404, run grep SMOKE_TEST_COOKIE_LEAK "$PROJECT_DIR/.env", then stop and restart mastra dev. Don't pivot to "UI only" because of this.

Seeding non-owner skills (Library Copy / non-owner flows)

A fresh scaffold has zero skills, and everything created through the API is owned by either the auth-off "no caller" (no authorId) or the currently signed-in user under auth-on. To exercise flows that require a skill owned by someone else (Library Copy, non-owner read-only view, private-skill visibility from a non-owner) without provisioning a second WorkOS account, run the seed script after the server has booted at least once:

# Start the server once so libsql initializes the skills tables.
cd $PROJECT_DIR
pnpm mastra:dev                # leave running, then in another shell:

bash .claude/skills/builder-smoke-test/scripts/seed-multi-user.sh
# → seeds smoke-seed-public-skill  (visibility=public,  status=published)
#         smoke-seed-private-skill (visibility=private, status=published)
#   both owned by authorId='user_seed_other'

The script writes directly to $PROJECT_DIR/src/mastra/public/mastra.db via the sqlite3 CLI (no Node deps). It's idempotent — re-running replaces the seeded rows. Use the seeded skills wherever a reference file asks for "a skill owned by another user"; clean them up with DELETE curls against /api/stored/skills/:id or by re-scaffolding.

Starting the dev server

If the server is not running on :4111, the Setup section starts it. The convenience helpers live under scripts/:

# Scaffold + preflight (writes .env, installs deps, detects auth mode)
bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect off

# Start the server from the scaffolded project
cd ~/mastra-builder-smoke-tests/builder-smoke
pnpm mastra:dev

# Poll /api/agents until 200 (60s budget). Detects mastra dev's port-bump.
bash .claude/skills/builder-smoke-test/scripts/wait-for-server.sh

wait-for-server.sh probes /api/agents — not / — because the SPA shell can return 200 before the API mounts. If it reports the server is up on :4112+ instead of :4111, mastra dev fell through to the next port; stop, free :4111, and restart. Continuing on a non-default port silently breaks every curl in every reference.

API base URL

Every reference assumes $BASE is exported. Set it once at the start of the run:

export BASE=http://localhost:4111/api

All curl examples in the references use $BASE and won't work in a shell that hasn't exported it.

Quick reference: key endpoints

This table lists the surfaces an agent will hit and where to look for the authoritative request/response shape. Don't copy curl blocks from here — run the per-section commands in references/<section>.md.

Builder Studio routes

Mobile renders a bottom-bar with the same primary entries.

Browser smoke

Use whichever browser tool the harness has wired up (Stagehand, Chrome MCP, etc.). Don't assume a specific provider — discover what's available, then drive the same checklist in references/ui.md.

The scaffolded project registers StagehandBrowser (matching examples/agent-builder). If BROWSERBASE_* keys aren't set in the shell, Stagehand falls back to local Playwright; that's fine for smoke. If neither Stagehand nor a local browser is reachable, mark UI as ⏭️ Skipped (no browser provider).

Result reporting

After testing, provide:

## Builder Smoke Test Results

**Date**: <date>
**Branch**: <branch>
**Commit**: <short sha>
**Server**: scaffolded project @ localhost:4111 (`$PROJECT_DIR`)
**Auth**: on / off / auto-skipped

| #   | Section            | Status   | Notes                           |
| --- | ------------------ | -------- | ------------------------------- |
| 1   | Setup              | ✅/❌    |                                 |
| 2   | Workspace          | ✅/❌    |                                 |
| 3   | Reconciliation     | ✅/❌/⏭️ |                                 |
| 4   | Defaults           | ✅/❌    |                                 |
| 5   | Model Policy       | ✅/❌    |                                 |
| 6   | Skills             | ✅/❌    |                                 |
| 7   | Registry           | ✅/❌    |                                 |
| 8   | Agents             | ✅/❌    |                                 |
| 9   | Pickers            | ✅/❌    |                                 |
| 10  | Stars              | ✅/❌    |                                 |
| 11  | Permissions / RBAC | ✅/❌    |                                 |
| 12  | Infrastructure     | ✅/❌    |                                 |
| 13  | Channels           | ✅/❌    |                                 |
| 14  | UI                 | ✅/❌/⏭️ |                                 |
| 15  | Auth               | ✅/❌/⏭️ | (skipped if no WORKOS\_\* vars) |

**Product issues**: (list any — server/UI behaved unexpectedly. For each: HTTP method + path or UI route, expected vs actual, one-sentence guess at the cause. Do not pre-decide "known bug" — log what the server actually did. Say "none" if empty.)
**Skill issues**: (list any — the skill itself was wrong, unclear, stale, or unreachable. For each: which file + step (e.g. `references/skills.md` step F2), and what was wrong. Doc drift, not product bugs. Say "none" if empty.)

**Verify before filing.** Before adding anything to either list, re-confirm against the live response in this run, not memory of an earlier call:

- For any **shape mismatch / missing field / wrong key name** claim, paste the actual JSON fragment (or the relevant keys) directly under the bullet so the claim is reproducible. If the skill says `features.agent.skills` and the response has `features.agent.skills`, that is not a skill issue — names that look similar in passing (`featSkills`, `agent.features.skill`, etc.) are easy to misread.
- For any **endpoint inconsistency** claim (e.g. "endpoint A returns X but B returns Y"), re-curl both endpoints fresh in the same run rather than reusing a stale response from earlier in the section.
- For any **RBAC / authz** claim (403 where you expected 200, or vice versa), check `references/permissions.md` for the matrix _and_ check the "Design decisions" list in this file. Several roles intentionally share `*:read`, which means infra/list/get endpoints look "ungated" but are working as intended. Also confirm the cookie you sent belongs to the role you think it does (`curl -H "Cookie: $(cat /tmp/cookie.txt)" $BASE/auth/me | jq '.role // .roles'`).
- For any **missing endpoint** claim (e.g. "agent avatar 404"), confirm the contract first — several flows are client-composed on top of generic CRUD (avatar = `PATCH metadata.avatarUrl`; Library Copy = `POST /stored/skills` with `metadata.origin`). The "Design decisions (don't file as bugs)" section enumerates the common ones.
- If a claim can't be reproduced on a fresh request, drop it.
  **Regressions**: (list any behavioral changes from a previous run)
  **Warnings**: (e.g., dev-server crash on `/auth/refresh` polling, OPENAI_API_KEY required at startup)
  **Skipped sections**: (list with reason)

Known rough edges

The branch has accumulated minor papercuts. Note these in your report only if you hit them; don't fail the run on them:

• Don't rm $PROJECT_DIR/mastra.db by hand while the server is up — stop the server first, then delete.
• Dev server can crash on hot-reload from /auth/refresh polling. Restart and continue.
• OPENAI_API_KEY is required at startup — server won't boot without it, even if you only test non-LLM surfaces.
• mastra dev overwrites process.env from .env at boot, so inline env overrides on the command line don't reach the server. Re-run scaffold to change .env.
• The scaffold links against the current worktree's packages via link: overrides. If you switch worktrees, re-run scaffold so the symlinks point at the right tree.

Design decisions (don't file as bugs)

These have come up across multiple runs and are intentional. If you observe one, note it in your report as "expected behavior" — do not open a product issue.

• `GET /auth/me` without a cookie returns `200` with a `null`-ish body. The route is mounted as a public route (createPublicRoute); the contract is "return the current user or null", not "401 if missing". A 401 here would break the public app shell.
• `/editor/builder/infrastructure` is readable by every default role (admin / member / viewer). The handler gates on infrastructure:read and every default role has *:read, which matches by resource-wildcard. The page only exposes deployment-shape data (provider names, registered flags, configured/unconfigured booleans) — no secrets.
• Flipping a skill's `visibility` from `private` to `public` does not auto-publish unless the skill has a registered `skillPath`. Visibility and publication are independent fields by design. A plain-create skill flipped public stays at activeVersionId: null until a real POST /publish runs against a source path.
• *Zod schema validation runs before the permission middleware on `/stored/` writes.** A malformed body from a viewer returns a 400, not a 403. This is standard request lifecycle; the response surface doesn't leak resource state.
• The role-impersonation picker only lists roles _different from the current one_. Logged in as admin, you'll see Member and Viewer and nothing else — there is no Admin self-item. This is intentional (admin is the baseline; you're already there).
• Impersonation is UI-only. The API still answers per the real logged-in role. A curl while impersonating viewer will still return the admin's response.
• `Favorites` sidebar entry links to `/agent-builder/favorite` (singular). The plural /favorites is not a registered route and renders the React Router 404. Use the sidebar link or the singular URL when scripting.
• Avatar upload uses agent `PATCH` with `metadata.avatarUrl`, not a dedicated `/avatar` endpoint. See references/agents.md.
• Copy is client-side. There is no POST /stored/skills/:id/copy. The UI fetches the source skill and POSTs a new row to /stored/skills with metadata.origin = "library-copy". See references/registry.md.

Out of smoke-test scope

Some flows are documented in references/ but are not driven by the smoke-test agent because they require server-lifecycle gymnastics that don't fit a single run:

• Reconciliation steps 2/3/4/6 (references/reconciliation.md) require editing $PROJECT_DIR/src/mastra/index.ts (changing basePath / workspaceId / config), restarting mastra dev multiple times, and observing drift detection or orphan archival across restarts. The smoke-test agent runs only Step 1 (fresh-startup persistence) and Step 5 (non-builder workspaces untouched). Run the rest by hand when changing reconciliation code.
• Real role-swap testing (logging in as multiple WorkOS users with different roles in the same run) is out of scope. The agent verifies whichever role the live --role user actually has, and additionally exercises the UI-only role impersonation flow under --role admin (see references/ui.md).

References

• references/setup.md — server health, builder settings sanity, baseline counts, builder workspace existence
• references/workspace.md — workspace CRUD via API
• references/reconciliation.md — config-driven workspace lifecycle (fresh, idempotent, drift, archival, backfill)
• references/defaults.md — builder defaults applied at agent create (memory, workspace, browser, model)
• references/model-policy.md — allowed list, default model, dropdown filtering, rejection
• references/skills.md — skill CRUD, visibility, publish, filesystem writes, files array
• references/registry.md — skills.sh browse/install, library Copy flow, origin badges, gating
• references/agents.md — stored agent CRUD, skill attachment, model swap, delete-from-edit, avatar upload
• references/picker-allowlist.md — tools/agents/workflows pickers respect allowlists
• references/favorites.md — favorite/unfavorite agents and skills, idempotency (formerly stars.md)
• references/permissions.md — viewer/member/admin/owner gating, role expectation matrix, UI impersonation, auth-off bypass
• references/infrastructure.md — /editor/builder/infrastructure payload + UI
• references/channels.md — Slack provider visibility, connectChannel tool
• references/ui.md — browser checklist across Builder routes
• references/auth.md — WorkOS on/off, 401 behavior, authorId, mode-toggle via .env
• scripts/scaffold.sh — scaffold or refresh the hermetic project at $PROJECT_DIR
• scripts/preflight.sh — wraps scaffold.sh + mode expectation (--expect off|on)
• scripts/wait-for-server.sh — poll :4111 until healthy

Stored Agent CRUD & Skill Attachment

Test stored agent create, read, update, delete, skill attachment, and model configuration.

The model values used below (openai/gpt-4o-mini, openai/gpt-4o) are valid under the scaffolded project's admin policy, which allows any openai model via wildcard plus exactly anthropic/claude-opus-4-7 (see the scaffolded project's src/mastra/index.ts → models.allowed). If you've changed the policy, swap these for something allowed.

Server schema reminder: model is { provider, name } — not { provider, modelId }. The schema lives in packages/server/src/server/schemas/stored-agents.ts. Posting modelId returns 400 model: Invalid input.

Pagination is 0-indexed. page=0 is the first page; page=1 is the second. Default perPage varies by endpoint; pass it explicitly if it matters.

Visibility is auth-on-only. With --auth off, the server has no caller to attribute ownership to, so it forces visibility: "public" regardless of what you send. Don't assert on visibility under auth off; verify it under --auth on in references/auth.md.

Schemas are in packages/server/src/server/schemas/stored-agents.ts. Treat that file as the source of truth for response shapes.

Capability gate. Create / PATCH / DELETE steps require stored-agents:write. The scaffold grants this to owner, admin, and member; viewer does not have it. Under --role viewer, mark write steps n/a — role lacks stored-agents:write and run only the read-side steps (GET list, GET by id).

Auth-on session header. Under --auth on, prepend -H "Cookie: $COOKIE" to every curl in this file (exported from references/auth.md step 0). Snippets below omit it for readability so they read cleanly under --auth off; an authenticated run without the cookie returns 401 before the intended assertion fires.

Steps

1. Create a stored agent

curl -s -X POST $BASE/stored/agents \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Smoke Test Agent",
    "instructions": "You are a helpful test agent created during smoke testing.",
    "model": {
      "provider": "openai",
      "name": "gpt-4o-mini"
    }
  }' | jq .

Verify:

• [ ] Returns 200 with the created agent
• [ ] name matches the request
• [ ] Response includes a workspace association referencing the builder workspace
• [ ] id is present; record it as AGENT_ID=<id>

Notes (don't assert under --auth off):

• visibility will be "public" regardless of request (see auth-on path).
• authorId will be null (no caller).
• favoriteCount will be 0 and isFavorited will be false (no caller-scoped favorite rows).

2. Get the agent

curl -s $BASE/stored/agents/$AGENT_ID | jq .

• [ ] Returns 200 with the agent
• [ ] model.provider == "openai" and model.name == "gpt-4o-mini"
• [ ] instructions matches
• [ ] createdAt and updatedAt are ISO timestamps

3. List agents

# Page 0 is the first page
curl -s "$BASE/stored/agents?page=0&perPage=50" | jq '{ total, page, perPage, count: (.agents | length) }'

• [ ] total >= 1
• [ ] agents array length matches total (assuming total <= perPage)
• [ ] The created $AGENT_ID appears in the array

4. Create a skill for attachment

SKILL_RESP=$(curl -s -X POST $BASE/stored/skills \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Agent Smoke Skill",
    "description": "Skill to attach to smoke test agent",
    "instructions": "Skill-level instructions for the smoke test."
  }')
echo "$SKILL_RESP" | jq .
SKILL_ID=$(echo "$SKILL_RESP" | jq -r '.id')
echo "SKILL_ID=$SKILL_ID"

• [ ] Response is 200 with an id

instructions is required by the schema today. Creating a skill without it returns 400.

5. Attach skill to agent

curl -s -X PATCH $BASE/stored/agents/$AGENT_ID \
  -H 'Content-Type: application/json' \
  -d "{\"skills\": {\"$SKILL_ID\": {}}}" | jq '.skills'

• [ ] PATCH returns 200
• [ ] skills object contains a key $SKILL_ID

6. Verify skill cross-reference

curl -s $BASE/stored/agents/$AGENT_ID | jq '.skills'

• [ ] Skills object includes $SKILL_ID

7. Update agent model

curl -s -X PATCH $BASE/stored/agents/$AGENT_ID \
  -H 'Content-Type: application/json' \
  -d '{
    "model": {
      "provider": "openai",
      "name": "gpt-4o"
    }
  }' | jq '.model'

• [ ] model.name == "gpt-4o"
• [ ] model.provider == "openai"

8. Update agent instructions

curl -s -X PATCH $BASE/stored/agents/$AGENT_ID \
  -H 'Content-Type: application/json' \
  -d '{"instructions": "Updated instructions for smoke testing."}' | jq '.instructions'

• [ ] Returns the new instructions string verbatim

9. Detach skill from agent

curl -s -X PATCH $BASE/stored/agents/$AGENT_ID \
  -H 'Content-Type: application/json' \
  -d '{"skills": {}}' | jq '.skills'

• [ ] skills is now an empty object

10. Delete agent and skill (cleanup)

curl -s -o /dev/null -w "%{http_code}\n" -X DELETE $BASE/stored/agents/$AGENT_ID  # → 200
curl -s -o /dev/null -w "%{http_code}\n" -X DELETE $BASE/stored/skills/$SKILL_ID  # → 200
curl -s -o /dev/null -w "%{http_code}\n" $BASE/stored/agents/$AGENT_ID            # → 404
curl -s -o /dev/null -w "%{http_code}\n" $BASE/stored/skills/$SKILL_ID            # → 404

Delete from view / edit (#16199)

From the agents list, clicking a row navigates to /agent-builder/agents/$AGENT_ID/view. Owners can switch to /edit from there.

• [ ] Delete agent affordance is reachable from the view or edit page (kebab menu, panel button, or similar). Log which surface exposes it.
• [ ] Clicking opens a confirm dialog
• [ ] Confirming deletes the agent and navigates back to the agents list
• [ ] Subsequent GET /stored/agents/$AGENT_ID returns 404

Avatar upload (owner-only, #15877 / #16264)

Owners may upload an avatar; non-owners (even admins) cannot. This step requires --auth on.

There is no dedicated `/avatar` endpoint. Avatars ride on the regular agent PATCH as metadata.avatarUrl (a data URL). The server validates size and shape via validateMetadataAvatarUrl in packages/server/src/server/handlers/validate-avatar.ts (current cap is 512 KB; accepted MIME types are image/png, image/jpeg, image/webp, image/gif).

# Construct a small PNG data URL (any tiny image works; example is 1x1 transparent)
SAMPLE_PNG="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkAAIAAAoAAv/lxKUAAAAASUVORK5CYII="

curl -s -X PATCH "$BASE/stored/agents/$AGENT_ID" \
  -H "Content-Type: application/json" \
  -H "$SESSION" \
  -d "{\"metadata\": {\"avatarUrl\": \"$SAMPLE_PNG\"}}" | jq '.metadata.avatarUrl'

• [ ] Owner: 200, response metadata.avatarUrl is the data URL you sent (or a server-rewritten URL)
• [ ] Non-owner authenticated user: 403 (ownership check inside assertWriteAccess)
• [ ] Auth off: behaves as owner (no caller → bypass)
• [ ] Oversized blob (>512 KB) is rejected with `413 Payload Too Large` from validateMetadataAvatarUrl (not 400)

Builder defaults at create

For full coverage of applyBuilderDefaults(), see references/defaults.md. Short version: when you POST /stored/agents with no workspace/memory/browser/model, the response should include the configured defaults.

Model dropdown verification

The builder config defines which models are allowed. Verify via the settings endpoint:

curl -s $BASE/editor/builder/settings | jq '{ models: .configuration.agent.models, modelPolicy }'

• [ ] configuration.agent.models.allowed and modelPolicy.allowed agree on the same allow-list
• [ ] Both include the entry you used in step 1 ({ "provider": "openai" } wildcard covers gpt-4o-mini)

Checklist

• [ ] Create stored agent (model.name, not modelId) with auto-workspace assignment
• [ ] Get agent by ID returns 200 with matching fields
• [ ] List agents with page=0 includes the new agent
• [ ] Create + attach skill (skill instructions required)
• [ ] Skill cross-reference visible on GET
• [ ] Update model (provider + name)
• [ ] Update instructions
• [ ] Detach skill
• [ ] Delete agent + skill, follow-up GET returns 404
• [ ] Model policy in settings matches what was accepted on create

Auth Toggle

Test auth on/off behavior end-to-end. This section is the only place that requires WorkOS env vars; every other section runs auth-off.

Mode toggle

The skill defines two states and switches between them by editing $PROJECT_DIR/.env (the scaffolded project). There is no global on/off flag in code — auth on is "AUTH_PROVIDER plus WorkOS creds present in .env," auth off is "those lines absent or commented." mastra dev reads .env once at boot, so any change requires a server restart.

auth off (Prompt 1 default)

$PROJECT_DIR/.env must have AUTH_PROVIDER commented or absent. The three WORKOS_* vars may stay in .env — they're inert without AUTH_PROVIDER. Confirm with:

bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect off

auth on (Prompts 2–3)

$PROJECT_DIR/.env must have all four:

AUTH_PROVIDER=workos
WORKOS_API_KEY=<key>
WORKOS_CLIENT_ID=<id>
WORKOS_ORGANIZATION_ID=<org-id>

Optional but commonly set: WORKOS_REDIRECT_URI (defaults to $BASE/auth/callback), WORKOS_COOKIE_PASSWORD.

Confirm with:

bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect on

If preflight reports missing vars, surface that to the user — don't edit .env without explicit consent. The user can either add the lines themselves or dictate values for you to write.

Steps

0. Auth ON — extract the session cookie for curl

The WorkOS session cookie is httpOnly, so curl cannot mint it via /sign-in and document.cookie cannot read it in the browser. Do this before any other auth-on step that uses `curl`. Don't fall back to "UI only" — every snippet below uses -H "Cookie: $COOKIE" and this is how you get the value.

The scaffold ships a debug route gated by SMOKE_TEST_COOKIE_LEAK=1 that echoes back the request's Cookie header. scaffold.sh writes this line into .env automatically whenever the --workos-* flags trigger auth-on mode, so you should not normally need to add it by hand.

Ordering gotcha. The apiRoutes array is built once when mastra dev boots, from process.env.SMOKE_TEST_COOKIE_LEAK. If the flag isn't set in .env before that boot, GET /smoke-test/cookie returns 404 even though it appears in this doc. If you see a 404 here, the fix is always "stop mastra dev, confirm the line is in $PROJECT_DIR/.env, restart" — never try to enable it at runtime via curl or browser. Verify with grep SMOKE_TEST_COOKIE_LEAK "$PROJECT_DIR/.env" before restarting.

Recipe:

# 1. Confirm the leak flag is in $PROJECT_DIR/.env (scaffold.sh writes it for auth-on).
grep -q '^SMOKE_TEST_COOKIE_LEAK=1' "$PROJECT_DIR/.env" \
  || echo 'SMOKE_TEST_COOKIE_LEAK=1' >> "$PROJECT_DIR/.env"
# 2. Restart `mastra dev` so it picks up the env change (only needed if step 1 wrote anything).
# 3. In the browser, sign in via WorkOS (navigate to $BASE/agent-builder).
# 4. In the same browser session, navigate to $BASE/smoke-test/cookie
#    and copy the response body (it's the raw Cookie header).
# 5. Export it for the rest of the run:
export COOKIE='wos-session=...; Path=/; ...'

After that, every -H 'Cookie: <session-cookie>' placeholder below becomes -H "Cookie: $COOKIE". Re-run steps 3-5 if the dev server is restarted with a different WORKOS_COOKIE_PASSWORD (the scaffold derives a stable one, so this is usually a one-shot operation per scaffold).

1. Auth ON — verify login required

Ensure --expect on passes, restart mastra dev if you just edited .env.

curl -s -o /dev/null -w '%{http_code}' $BASE/stored/agents

• [ ] Returns 401 (not 200)
• [ ] Response body is JSON, not HTML or a stack trace

In the browser:

• [ ] Navigate to http://localhost:4111/agent-builder
• [ ] Redirected to WorkOS login
• [ ] After login, builder loads normally

1b. Auth ON — assert the logged-in role matches --role

--role defaults to admin. After login, ask the server who you are:

curl -s -H "Cookie: <session-cookie>" "$BASE/auth/me" | jq '{id, email, roles, permissions}'

• [ ] HTTP 200 + JSON body with id, email, roles, permissions (the user identifier field is id, not userId)
• [ ] roles includes the value passed via --role (e.g. --role viewer → roles contains "viewer")

If roles does not contain the --role value, stop the run and tell the user:

The logged-in user's roles are <actual roles> but --role is <expected>. Either change your WorkOS role to <expected> and restart the server, or re-run the smoke test with --role <one of your actual roles>.

Do not try to "simulate" a different role by setting headers — there is no server-side role-override header in this build. The only way to test a different role is to log in as a user who actually has it.

2. Auth ON — verify authorId is set

After logging in, create an entity (use the browser session or copy the session cookie into curl):

curl -s -X POST $BASE/stored/skills \
  -H 'Content-Type: application/json' \
  -H 'Cookie: <session-cookie>' \
  -d '{
    "name": "Auth Test Skill",
    "description": "Created to verify authorId is set under auth-on",
    "instructions": "Auth-on smoke test placeholder."
  }' | jq '.authorId'

• [ ] authorId is a non-empty string matching the logged-in WorkOS user ID (typically prefixed user_…); it must not be null, undefined, or omitted

3. Auth ON → Auth OFF — switch mode

1. Comment out the AUTH_PROVIDER=workos line in $PROJECT_DIR/.env (one # at the start of the line). 2. Restart the dev server (kill the existing mastra dev process, then re-run from $PROJECT_DIR). 3. Re-run preflight with the new expectation:

   bash .claude/skills/builder-smoke-test/scripts/preflight.sh --expect off

• [ ] Preflight reports detected mode off
• [ ] API returns 200 without a session:

  curl -s -o /dev/null -w '%{http_code}' $BASE/stored/agents

• [ ] In the browser, /agent-builder loads without a login prompt

4. Auth OFF — data persists

• [ ] Entities created during the auth-on phase still appear in the auth-off

listings

• [ ] authorId on those entities is preserved (records from auth-on don't

get rewritten)

5. Auth-not-configured bypass (#16107)

With AUTH_PROVIDER absent, ownership/role checks at the route layer should be bypassed cleanly:

• [ ] Creating entities returns 200/201 with no authorId in the response (the server resolves getCallerAuthorId → null and writes the row without an author)
• [ ] Reads / writes succeed without any auth header
• [ ] Library page still surfaces public skills

6. Error handling

Re-enable auth (uncomment AUTH_PROVIDER=workos, restart). Make an unauthenticated request:

curl -s $BASE/stored/agents | jq .

• [ ] Clear JSON error (401/403), not a server crash
• [ ] Error body is JSON-shaped, not HTML

Notes

• Auth changes require a server restart — mastra dev only reads .env at

boot.

• The WorkOS session cookie is httpOnly, so a Stagehand-style browser

automation picks it up automatically.

• authorId on entities created while auth is off will be missing/null (the handler resolves it from request context, which has no caller). Records created while auth was on keep their original authorId after a mode flip — they are never rewritten.

Checklist

• [ ] Preflight reports the expected mode before each phase of this section
• [ ] Auth ON: API returns 401 without session
• [ ] Auth ON: browser redirects to login
• [ ] Auth ON: authorId set on created entities
• [ ] Auth OFF: API accessible without auth
• [ ] Auth OFF: browser loads without login
• [ ] Auth ON → OFF: data persists, authorId preserved
• [ ] Unauthenticated requests return clean JSON errors

Appendix: FGA in this example

Background on the fine-grained authorization layer — only relevant if an auth-on run surfaces an FGADeniedError.

• MastraFGAWorkos is the WorkOS-backed FGA provider. It's constructed

in the scaffolded project's src/mastra/auth.ts (via initWorkOS() from @mastra/auth-workos) and resolves per-resource permissions ("can user X :read agent Y") against the WorkOS organization named by WORKOS_ORGANIZATION_ID.

• FGA fires only when (a) a route declares an fga block in its metadata

AND (b) the server has an FGA provider configured (which, in this example, means AUTH_PROVIDER=workos). There is no separate enable/ disable env var.

• If FGA denies a request during an auth-on run, the most likely causes

are: WORKOS_ORGANIZATION_ID doesn't match the org the FGA tuples are stored under, or the logged-in user has no matching tuple. Report the denial along with the org/user combo — don't try to disable FGA independently, it's coupled to WorkOS auth here.

Channels

mastra.channels providers (Slack here) surface on the Infrastructure page and via the connectChannel client tool inside Builder chats. PR #16161 ships the Slack frontend; #16170 ships the tool.

Source-of-truth

In the scaffolded project's src/mastra/index.ts:

channels: {
  slack: new SlackProvider({ baseUrl: process.env.MASTRA_BASE_URL }),
}

SlackProvider.isConfigured controls whether the provider appears in /editor/builder/infrastructure's channels list.

Steps

1. Slack with no env vars _(auth-off-friendly negative path)_

If SLACK_CLIENT_ID / SLACK_CLIENT_SECRET / etc. are unset (the default scaffold state):

curl -s "$BASE/editor/builder/infrastructure" | jq '.channels'

• [ ] Response shape is { providers: [...] }
• [ ] providers is [] (empty array) — slack is filtered out by isConfigured
• [ ] Section status: ✅ (negative path verified). Steps 2–5 (positive Slack flow) defer to Run 2 only when SLACK_* env vars are set.

2. Slack with env vars

Set SLACK_* env vars in $PROJECT_DIR/.env, restart the dev server.

curl -s "$BASE/editor/builder/infrastructure" | jq '.channels.providers[] | select(.id==\"slack\")'

• [ ] Slack entry present
• [ ] Config entries reflect set vars (non-null)
• [ ] No raw secret values exposed

3. UI: Infrastructure page shows Slack

Navigate to /agent-builder/infrastructure.

• [ ] Channels section lists Slack
• [ ] Shows base URL (or "Provider default")
• [ ] Provider name reads as expected ("Slack", not the class name)

4. connectChannel tool in Builder chat

In a Builder chat (e.g., the builder agent in /agent-builder), prompt:

Connect Slack so my agent can post to #general.

• [ ] Agent uses the connectChannel tool
• [ ] Tool returns a connection URL or success state
• [ ] No raw OAuth secrets leaked into chat
• [ ] If Slack isn't configured: tool returns an actionable error pointing to env vars

5. Negative path

With Slack unset, ask the same question.

• [ ] Tool/agent reports Slack is unavailable
• [ ] Suggests configuring SLACK_* env vars

Checklist

• [ ] Slack hidden when not configured
• [ ] Slack shown when configured; no secrets leaked
• [ ] Provider name renders cleanly
• [ ] connectChannel tool callable from Builder chat
• [ ] Unconfigured channel returns actionable error

Builder Defaults on Agent Create

When the Agent Builder is enabled, applyBuilderDefaults() fills in workspace, memory, browser, and model on stored-agent create — but only for fields the caller did not explicitly set. Explicit null is preserved as "no default" (opt-out).

Reference: packages/core/src/agent-builder/ee/apply-builder-defaults.ts and the scaffolded project's src/mastra/index.ts (the builder.configuration.agent block).

Visibility under `--auth off`. Some examples below pass "visibility": "private". With --auth off the server has no caller to attribute ownership to and forces visibility: "public" (and authorId: null) regardless of what you send. That is expected; don't fail the step on it. Under --auth on, "private" is preserved.

Source-of-truth: builder config in the scaffolded project

builder: {
  configuration: {
    agent: {
      workspace: { type: 'id', workspaceId: 'builder-workspace' },
      memory:    { options: { lastMessages: 10 } },
      browser:   { type: 'inline', config: { provider: 'stagehand' } },
      models: {
        allowed: [{ provider: 'openai' }, { provider: 'anthropic', modelId: 'claude-opus-4-7' }],
        default: { provider: 'openai', modelId: 'gpt-5.4' },
      },
    },
  },
}

Steps

1. Create an agent with no overrides

RESP=$(curl -s -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "Defaults Smoke Agent", "instructions": "Smoke test for builder defaults.", "visibility": "private" }')
echo "$RESP" | jq .
AGENT_ID=$(echo "$RESP" | jq -r '.id // .agent.id')

Verify the response (or a follow-up GET /stored/agents/$AGENT_ID):

• [ ] workspace.workspaceId is "builder-workspace" (nested under workspace, type="id")
• [ ] model.provider is "openai" and model.name is "gpt-5.4" (default model — API persists the config's modelId under the name field)
• [ ] memory.options.lastMessages is 10
• [ ] browser.config.provider is "stagehand" (inline provider)
• [ ] authorId set (or null if auth is off)

2. Create an agent with explicit overrides

curl -s -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Overrides Smoke Agent",
    "instructions": "Smoke test for explicit overrides.",
    "model": { "provider": "anthropic", "name": "claude-opus-4-7" },
    "memory": { "options": { "lastMessages": 3 } },
    "visibility": "private"
  }' | jq .

• [ ] model matches the override (anthropic / claude-opus-4-7)
• [ ] memory.options.lastMessages is 3, not 10
• [ ] workspace.workspaceId is still the default builder workspace (not overridden)
• [ ] browser is still the default

3. Create an agent with explicit null to opt out (browser only)

browser accepts null (or false) to opt out of the default. memory and workspace schemas do NOT accept null — omit the field entirely to "opt out" (the default just won't apply for omitted fields anyway). Sending memory: null returns HTTP 400.

curl -s -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Null Opt-out Smoke Agent",
    "instructions": "Smoke test that explicit null preserves opt-out.",
    "browser": null,
    "visibility": "private"
  }' | jq .

• [ ] browser is null (default was NOT applied because caller set null)
• [ ] memory still got the default (caller omitted the field)
• [ ] model still got the default
• [ ] workspace.workspaceId still got the default

Verify the negative path:

curl -s -o /dev/null -w "%{http_code}\n" -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "Memory Null Smoke", "instructions": "x", "memory": null }'

• [ ] Status code is 400 (schema rejects memory: null)

4. Verify defaults expose via settings

curl -s "$BASE/editor/builder/settings" | jq '.configuration.agent'

• [ ] workspace, memory, browser, models.default, models.allowed all appear

Cleanup

curl -s -X DELETE "$BASE/stored/agents/$AGENT_ID" | jq .
# repeat for other agents created above

Checklist

• [ ] Default workspace applied when caller omits
• [ ] Default model applied when caller omits
• [ ] Default memory applied when caller omits
• [ ] Default browser applied when caller omits
• [ ] Explicit fields are preserved (not overwritten)
• [ ] Explicit null on browser preserves opt-out (default NOT applied)
• [ ] Explicit null on memory returns HTTP 400 (schema does not allow null)
• [ ] Settings endpoint exposes the configured defaults

Favorites

Test favorite/unfavorite functionality for stored agents and skills.

Favorite endpoints are documented in packages/server/src/server/schemas/favorites.ts (toggle response) and the agent/skill response schemas in the same directory (GET response). Refer to those for exact field names and types; assert against the schema, not against fields baked into this doc.

Both favorite and unfavorite are idempotent — calling them twice returns the same body the second time. Favorites are gated by the favorites builder feature (404 if disabled) and require auth (401 under --auth off). (This file was formerly named stars.md; the feature was renamed stars → favorites across the stack. See PR #16749 — STACK-3.)

Field-name asymmetry. The toggle endpoint (PUT|DELETE /stored/{type}/:id/favorite) returns { favorited, favoriteCount }. The GET endpoint (/stored/{type}/:id) exposes the caller's favorite state as isFavorited (alongside favoriteCount). When asserting "is favorited" on GET, check isFavorited, not favorited.

Auth requirement

This section requires `--auth on`. Favorites are scoped per caller (the row in stored_favorites is keyed on (entityId, authorId)). With --auth off, there is no caller to attach the favorite to and the route returns either 401 Authentication required or 404 Not Found depending on how the route is registered for the current build — both mean "unreachable under auth-off". Treat any non-2xx as the expected outcome.

Running with --auth off

Favorites are fully unreachable under `--auth off`. The PUT/DELETE endpoints return a non-2xx (typically 401 Authentication required, sometimes 404 Not Found depending on build), and the Studio + Agent Builder favorite buttons render with a "Sign in to favorite this agent/skill" tooltip. Do the sanity check below, mark this section as Skipped (requires --auth on), and move on. Do not try to create agents and favorite them — it will not work and is not expected to work.

# Sanity: confirm favorites are gated by auth
curl -s -o /dev/null -w "%{http_code}\n" -X PUT $BASE/stored/agents/$AGENT_ID/favorite
# → 401 or 404 (unreachable under auth-off)
curl -s -o /dev/null -w "%{http_code}\n" -X DELETE $BASE/stored/agents/$AGENT_ID/favorite
# → 404

• [ ] Both calls return a non-2xx status, either 401 or 404 (both mean "unreachable under auth-off")
• [ ] Skip the rest of this file; report the section as Skipped (requires --auth on)

Prerequisites (auth-on)

You need a logged-in session ($SESSION should be a Cookie: header) and a stored entity to target.

If you have `stored-agents:write` / `stored-skills:write` (owner, admin, member), create test entities:

# Test agent
AGENT_RESP=$(curl -s -X POST $BASE/stored/agents \
  -H "$SESSION" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Favorite Test Agent",
    "instructions": "Test agent for favorite testing",
    "model": {"provider": "openai", "name": "gpt-4o-mini"}
  }')
AGENT_ID=$(echo "$AGENT_RESP" | jq -r '.id')

# Test skill
SKILL_RESP=$(curl -s -X POST $BASE/stored/skills \
  -H "$SESSION" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Favorite Test Skill",
    "description": "Test skill for favorite testing",
    "instructions": "Favorite test instructions."
  }')
SKILL_ID=$(echo "$SKILL_RESP" | jq -r '.id')

If you don't have write perms (viewer), use the rows from seed-multi-user.sh (run it from SKILL.md execution flow step 4 if you haven't):

SKILL_ID=smoke-seed-public-skill   # public, owned by user_seed_other
# For agents, skip steps 1–3 and run the skill steps only — the seed script does not
# seed stored agents. Note "agent favorite CRUD: not exercised in non-admin runs" in the report.

Steps

1. Favorite an agent

curl -s -X PUT $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .

• [ ] HTTP 200
• [ ] Body is { "favorited": true, "favoriteCount": <n> } with n >= 1

2. Verify the agent is favorited

curl -s $BASE/stored/agents/$AGENT_ID -H "$SESSION" | jq .

• [ ] isFavorited is true on the GET response
• [ ] favoriteCount matches the value from step 1

3. Unfavorite the agent

curl -s -X DELETE $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .

• [ ] HTTP 200
• [ ] Body shows the agent is no longer favorited for the caller and favoriteCount decreased by 1
• [ ] Re-fetching the agent reflects the unfavorited state

4. Favorite a skill

curl -s -X PUT $BASE/stored/skills/$SKILL_ID/favorite -H "$SESSION" | jq .

• [ ] HTTP 200
• [ ] Body is { "favorited": true, "favoriteCount": <n> }

5. Verify the skill is favorited

curl -s $BASE/stored/skills/$SKILL_ID -H "$SESSION" | jq .

• [ ] isFavorited is true on the GET response
• [ ] favoriteCount matches step 4

6. Unfavorite the skill

curl -s -X DELETE $BASE/stored/skills/$SKILL_ID/favorite -H "$SESSION" | jq .

• [ ] HTTP 200
• [ ] Body is { "favorited": false, "favoriteCount": <previous - 1> }

7. Idempotent favorite (favorite twice)

curl -s -X PUT $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .
curl -s -X PUT $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .

• [ ] Both calls return 200
• [ ] Both bodies are identical (favoriteCount does not increment on the second call)

8. Idempotent unfavorite (unfavorite twice)

curl -s -X DELETE $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .
curl -s -X DELETE $BASE/stored/agents/$AGENT_ID/favorite -H "$SESSION" | jq .

• [ ] Both calls return 200
• [ ] Both bodies are identical (favorited: false, favoriteCount unchanged on the second call)

Cleanup

curl -s -X DELETE $BASE/stored/agents/$AGENT_ID -H "$SESSION" -o /dev/null -w "%{http_code}\n"  # → 200
curl -s -X DELETE $BASE/stored/skills/$SKILL_ID -H "$SESSION" -o /dev/null -w "%{http_code}\n"  # → 200

Checklist

• [ ] Auth-off path: PUT/DELETE favorite return non-2xx (401 or 404); no other assertions
• [ ] Auth-on: favorite agent (200 + favorited: true)
• [ ] Verify agent favorited on GET
• [ ] Unfavorite agent (200 + favorited: false)
• [ ] Favorite skill (200 + favorited: true)
• [ ] Verify skill favorited on GET
• [ ] Unfavorite skill (200 + favorited: false)
• [ ] Idempotent favorite (second body identical)
• [ ] Idempotent unfavorite (second body identical)

Infrastructure

The /editor/builder/infrastructure endpoint reports configured channels, browser, workspace, and registries. The Agent Builder Infrastructure page renders this. The payload is deployment-shape only — provider names and enabled/disabled flags, no secrets — so all default roles can read it (*:read matches infrastructure:read).

Source-of-truth

Endpoint: GET /editor/builder/infrastructure (requires infrastructure:read).

Schema (packages/server/src/server/schemas/editor-builder.ts → infrastructureStatusResponseSchema):

{
  channels: { providers: Array<{ id, name, isConfigured, routeCount }> },
  browser:  { type, provider, env, registered, availableProviders, config: [{key,value}] },
  workspace:{ type, workspaceId, name, source, registered, hasFilesystem, hasSandbox,
              filesystemProvider, sandboxProvider, config: [{key,value}] },
  registries: { skillsSh: { enabled } },
}

Notes:

• channels.providers is filtered server-side to providers that report isConfigured: true.
• browser.config and workspace.config are arrays of {key, value} pairs. Unset values are emitted as null (not omitted), so the UI can render "Provider default" / "Not set".
• registries is an object keyed by registry id (currently only skillsSh), not an array. Each value is { enabled }.

Steps

1. Any signed-in role can read

curl -s -H "$SESSION" "$BASE/editor/builder/infrastructure" | jq .

• [ ] HTTP 200
• [ ] Top-level keys present: channels, browser, workspace, registries
• [ ] Works for admin, member, and viewer — the gate is infrastructure:read, and every default role has *:read

2. Unauthenticated cannot read

With auth on and no session cookie:

curl -s -o /dev/null -w '%{http_code}\n' "$BASE/editor/builder/infrastructure"

• [ ] HTTP 401 (no session)

3. Browser block

curl -s -H "$SESSION" "$BASE/editor/builder/infrastructure" | jq '.browser'

• [ ] provider is "stagehand" (matches inline config in the scaffolded project's src/mastra/index.ts)
• [ ] type is "inline" (matches browser: { type: 'inline', config: ... })
• [ ] registered is true
• [ ] availableProviders is a non-empty array
• [ ] config is an array of {key, value} pairs; unset values appear as null

4. Workspace block

curl -s -H "$SESSION" "$BASE/editor/builder/infrastructure" | jq '.workspace'

• [ ] type is "id" (matches workspace: { type: 'id', workspaceId: 'builder-workspace' })
• [ ] workspaceId is "builder-workspace"
• [ ] name is present (e.g. "Builder Workspace") and source reflects builder metadata
• [ ] registered is true; hasFilesystem is true; hasSandbox reflects scaffold config
• [ ] filesystemProvider populated (e.g. "local"); sandboxProvider may be null when no sandbox is configured
• [ ] config is an array of {key, value} pairs — may be empty [] when the workspace block doesn't expose tunables

If you change the inline workspace block in the scaffolded project's src/mastra/index.ts and restart:

• [ ] type flips to "inline"

5. Channels block

curl -s -H "$SESSION" "$BASE/editor/builder/infrastructure" | jq '.channels'

• [ ] Shape is { providers: [...] } (object with providers array, not a bare array)
• [ ] Only providers with isConfigured: true are present (Slack appears only if SLACK_* env vars are set)
• [ ] Each entry has id, name, isConfigured, routeCount

6. Registries block

curl -s -H "$SESSION" "$BASE/editor/builder/infrastructure" | jq '.registries'

• [ ] Shape is { skillsSh: { enabled: <boolean> } } (object keyed by registry id, not an array)
• [ ] skillsSh.enabled matches builder.registries.skillsSh.enabled from config (default false)
• [ ] Flipping enabled in config + restart → reflected here

7. UI: Agent Builder Infrastructure page

Navigate to http://localhost:4111/agent-builder/infrastructure.

• [ ] Page loads for admin
• [ ] Sidebar shows "Infra" link below a divider, matching Studio's style
• [ ] Browser, Workspace, and Channels sections render
• [ ] Unset values show "Provider default" / "Not set" rather than empty strings
• [ ] Mobile bottom-bar also exposes the "Infra" link
• [ ] Viewer/member: link visible (every default role has *:read, which matches infrastructure:read); direct navigation also resolves

Checklist

• [ ] Every default role with *:read (admin, member, viewer) can GET — the gate is infrastructure:read
• [ ] Browser block: type, provider, registered, availableProviders, config all present
• [ ] Workspace block: type, workspaceId, filesystemProvider, sandboxProvider, config
• [ ] Channels block: { providers: [...] } shape; only configured providers listed
• [ ] Registries block: { skillsSh: { enabled } } object shape (not array)
• [ ] UI page renders Browser / Workspace / Channels / Registries sections; sidebar + mobile bottom-bar link gated by infrastructure:read (visible to every default role under auth-on)

Model Policy

The builder's configuration.agent.models.allowed array constrains which models can be used. The models.default entry seeds new agents. Studio + Agent Builder dropdowns must respect the allowlist.

Two shapes, same concept. Builder _config_ (TypeScript, the scaffolded project's src/mastra/index.ts) uses { provider, modelId }. Stored-agents _API_ (POST /stored/agents, schema in packages/server/src/server/schemas/stored-agents.ts) uses { provider, name }. When you POST to create an agent, use name. When you read the policy from the settings endpoint or the TS source, you'll see modelId.

Non-admin runs. The scaffold grants member write on stored agents, so the create-time policy checks (steps 2–4) are reachable from --role member. viewer gets 403 before policy validation runs — for viewer, only step 1 (read-side) is meaningful; the UI dropdown gating in references/ui.md is the alternative path for viewer model-policy verification.

Source-of-truth

In the scaffolded project's src/mastra/index.ts:

models: {
  allowed: [
    { provider: 'openai' },                                  // wildcard: any openai model
    { provider: 'anthropic', modelId: 'claude-opus-4-7' },   // exact: only this anthropic model
  ],
  default: { provider: 'openai', modelId: 'gpt-5.4' },
}

Steps

Capability gate: Steps 2–4 require stored-agents:write. For --role viewer, skip them and rely on the UI dropdown check in references/ui.md. For --role admin and --role member, run them all (the scaffold grants member that perm).

1. Settings exposes the policy

curl -s "$BASE/editor/builder/settings" | jq '.configuration.agent.models'

• [ ] allowed is an array with the two entries above
• [ ] default matches { provider: 'openai', modelId: 'gpt-5.4' }

2. Create with an allowed wildcard model

curl -s -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Policy OK Agent",
    "instructions": "test",
    "model": { "provider": "openai", "name": "gpt-4o-mini" }
  }' | jq .

• [ ] 200; model.name is gpt-4o-mini (allowed via wildcard)

3. Create with the allowed exact model

curl -s -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Policy Exact Agent",
    "instructions": "test",
    "model": { "provider": "anthropic", "name": "claude-opus-4-7" }
  }' | jq .

• [ ] 200; model accepted

4. Create with a disallowed model

curl -s -o /tmp/policy-err.json -w '%{http_code}\n' \
  -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Policy Reject Agent",
    "instructions": "test",
    "model": { "provider": "anthropic", "name": "claude-haiku-3" }
  }'
cat /tmp/policy-err.json | jq .

• [ ] Returns 422 (semantic validation error) with a clear "model not in allowed list" message
• [ ] No agent was created (list count unchanged)

5. Browser dropdown respects the policy

In Studio (/agents) and Agent Builder (/agent-builder/agents/:id) model dropdowns:

• [ ] All OpenAI models appear (wildcard expansion)
• [ ] Only claude-opus-4-7 appears under Anthropic
• [ ] No other Anthropic models appear
• [ ] No other providers (Google, Mistral, etc.) appear

6. Default model is pre-selected on create

In the agent-create form:

• [ ] Model dropdown shows openai / gpt-5.4 selected by default
• [ ] User can change to any allowed alternative

Cleanup

# remove any agents created above

Checklist

• [ ] Settings exposes allowed + default
• [ ] Wildcard provider allows any model
• [ ] Exact (provider, modelId) policy entry restricts to that model on create
• [ ] Disallowed model rejected at create
• [ ] UI dropdown reflects allowlist exactly
• [ ] Default model pre-selected on agent create

Permissions / RBAC

Verify role-based gating across Studio and Agent Builder. Covers route-level RBAC, component-level gating (#16271), the UI-only role impersonation feature (#15864), and the auth-off bypass (#16107).

Default roles

The scaffolded project configures its own WorkOS roleMapping in src/mastra/auth.ts (so it doesn't depend on DEFAULT_ROLES in core):

Public stored skills/agents short-circuit read checks (see authorship.ts). Auth disabled bypasses role checks entirely.

Member write grants are narrow. Members can create/PATCH their own stored agents/skills/workspaces (so Library Copy, Stars, and edit flows are exercisable under a non-admin role), but cannot :publish, :delete, or :share. Ownership rules still apply: a member PATCHing another author's record gets 403. This mapping lives in the scaffolded project's auth.ts — DEFAULT_ROLES in core is unchanged.

WorkOS admin → owner-equivalent. The scaffold maps admin → ['*'], so a WorkOS-provisioned admin in this project carries permissions: ["*"] and passes DELETE checks. Treat admin as owner-equivalent for matrix purposes.

Picking the role to test

Under --auth on, the smoke test runs as whichever role the logged-in WorkOS user actually has. The --role flag (default admin) is the agent's expectation; setup asserts it matches /api/auth/me's roles field and stops if it doesn't.

There is no server-side "preview as role" header in this build. The "View as role" feature in the UI is purely frontend state (see references/ui.md — Impersonation UI). It does not change what the API returns. To exercise role gating at the API layer, the logged-in user must actually have that role.

If the user is logged in as admin and wants to test viewer behavior, they have two options:

1. Change their WorkOS role to viewer, restart mastra dev, re-run with --role viewer. 2. Run with --role admin and exercise the UI-only impersonation flow (covered in references/ui.md).

Role expectation matrix

Pass criteria per role for representative endpoints. The agent uses this to set expected status codes per section when --role is non-admin.

Member create/PATCH on own works because the scaffold grants stored-{agents,skills,workspaces}:write. Publish/delete/share remain admin-only. Member PATCH on another author's record returns `404 Not Found` — the visibility/ownership filter hides the row before the handler runs, so non-owners can't tell the difference between "doesn't exist" and "forbidden". This is the standard REST "don't reveal existence" pattern.

Steps

1. Confirm the logged-in role

curl -s -H "$SESSION" "$BASE/auth/me" | jq '{roles, permissions}'

• [ ] roles includes the value passed via --role
• [ ] permissions matches the grants for that role (see Default roles table)

If mismatch, halt — see references/auth.md step 1b.

2. Read works for every role

curl -s -o /dev/null -w '%{http_code}\n' -H "$SESSION" "$BASE/stored/agents"
curl -s -o /dev/null -w '%{http_code}\n' -H "$SESSION" "$BASE/stored/skills"

• [ ] Both 200 regardless of role
• [ ] Body is JSON; not HTML, not a stack trace
• [ ] Private agents/skills owned by _other_ users are absent from the list (unless caller is admin/owner)

3. Write gated by role

Try the matrix's POST row for the current --role. Expected codes per role are in the matrix above.

curl -s -o /dev/null -w '%{http_code}\n' -H "$SESSION" \
  -X POST "$BASE/stored/agents" \
  -H 'Content-Type: application/json' \
  -d '{ "name": "Role Gating Test", "instructions": "x", "model": { "provider": "openai", "name": "gpt-4o-mini" } }'

• [ ] Status matches the matrix for --role
• [ ] 403 bodies are JSON with an error message (no stack trace, no HTML)

4. Execute and write differ by resource (member case)

If --role member:

• [ ] POST /stored/agents → 200 (has stored-agents:write)
• [ ] POST /agents/:id/chat against an existing public agent → 200 (has :execute)
• [ ] POST /stored/skills/:id/publish against own draft → 403 (no :publish)
• [ ] PATCH /stored/skills/:id on another author's row → 403 (ownership check, not perm check)

If --role viewer:

• [ ] POST /agents/:id/chat → 403 (no :execute)
• [ ] POST /stored/agents → 403 (no :write)

5. Delete is owner-only

The scaffold maps admin → ['*'], so an admin in this project passes DELETE checks (owner-equivalent). Only viewer and member see 403 on DELETE.

• [ ] --role owner or --role admin: DELETE /stored/agents/:id on own → 200
• [ ] --role member: same DELETE → 403
• [ ] --role viewer: same DELETE → 403

If the matrix and the live response disagree, that is the finding — file it, don't "correct" the matrix without checking the scaffold's roleMapping in auth.ts first.

6. Visibility flip + publish semantics

The :share and :publish actions on stored-skills / stored-agents aren't wired into route requiresPermission. Instead, the handler calls assertShareAccess(ctx, record) (and equivalent for publish) inside the PATCH/POST handler. That helper allows the action when any of these hold:

1. The record has no owner (legacy/unowned). 2. The caller is the record's authorId. 3. The caller has the admin-bypass permission for the resource (stored-skills:write with no record filter). 4. The caller explicitly holds <resource>:share or <resource>:publish in their role grants.

Verify via the API:

• [ ] Owner can flip visibility on their own skill: PATCH /stored/skills/:id with {"visibility":"public"} returns 200 and the response has visibility: "public".
• [ ] Admin can flip visibility on a skill they don't own: same PATCH against another user's skill returns 200.
• [ ] Viewer / member can't flip visibility on a skill they don't own: same PATCH returns 403 with a JSON error body.
• [ ] Auth-off mode bypasses these checks (handler short-circuits when getCallerAuthorId(ctx) is null); record this as "auth-off bypass" rather than testing the matrix.

7. Auth-off bypass

Disable auth (comment out AUTH_PROVIDER in .env), restart. Everything should be reachable without role checks (#16107).

curl -s -o /dev/null -w '%{http_code}\n' "$BASE/stored/agents"
curl -s "$BASE/auth/me"

• [ ] /stored/agents returns 200
• [ ] /auth/me returns 200 with body null (not 401, not a user object) — the route resolves a missing caller to null rather than rejecting
• [ ] UI loads without login
• [ ] All affordances visible
• [ ] Created records have authorId: null

8. UI gating (per-role sidebar / affordances)

In the browser, while logged in as the --role user:

• [ ] Sidebar items matching the role's grants are visible; ungranted items are hidden
• [ ] Create/Edit/Delete buttons match the role's grants
• [ ] Direct nav to a gated route (/agent-builder/agents/:id/edit for viewer) redirects to the read-only view or denies

If --role admin (or owner), also run the UI Impersonation subset in references/ui.md — that's the only honest way to exercise viewer/member UI gating without re-authenticating.

Checklist

• [ ] /auth/me roles match --role
• [ ] Reads pass for all roles
• [ ] Writes pass/fail per the matrix
• [ ] Execute pass/fail per the matrix (member ≠ viewer)
• [ ] Delete is owner-only
• [ ] Visibility flips gated by ownership / admin-bypass / explicit :share
• [ ] Auth off bypasses all role checks
• [ ] UI affordances narrow with role

Picker Allowlists (Tools / Agents / Workflows)

The builder's features.agent.tools|agents|workflows flags + per-feature allowlists determine which entities appear in agent picker dropdowns. See PR #16025.

Source-of-truth

features.agent.tools = true means tools can be attached. If a pickerAllowlist is configured per surface, only entries on the list appear. With no allowlist, all registered tools/agents/workflows are visible (subject to feature flag).

Settings payload exposes resolved allowlists nested under .picker as picker.visibleTools / picker.visibleAgents / picker.visibleWorkflows (each is string[] | null; null = unrestricted). They are NOT at the top level — jq '.visibleTools' returns null because the field doesn't exist there. Stored-agent records use different keys (tools / agents / workflows), so cross-check both shapes when validating.

Casing note. The allowlist in configuration.agent.{tools,agents,workflows}.allowed accepts both registration-key form (weatherInfo) and entity-.id form (weather-info if the underlying tool sets id: 'weather-info'). The server resolves either to the canonical response key. So allowed: ['weather-info'] and allowed: ['weatherInfo'] are functionally equivalent — visibleTools will emit the registration key (weatherInfo for tools/workflows, .id for agents). This is not a bug; it's deliberate alias matching. See packages/server/src/server/handlers/editor-builder.ts collectAliases.

Steps

1. Confirm features enabled

curl -s "$BASE/editor/builder/settings" | jq '.features.agent'

• [ ] tools, agents, workflows, skills, model, browser, favorites, scorers, variables, avatarUpload, memory all true

(matches the scaffolded project's builder config — favorites replaces the legacy stars key after the rename)

1b. Confirm picker allowlists shape

curl -s "$BASE/editor/builder/settings" | jq '.picker'

• [ ] visibleTools is string[] (or null if unrestricted)
• [ ] visibleAgents is string[] and excludes builder-agent
• [ ] visibleWorkflows is string[]
• [ ] Tool/workflow keys use the registration form (camelCase, e.g., weatherInfo); agent keys use entity .id form (e.g., weather-agent)

2. Tool picker reflects registered tools

Open an agent in Builder (/agent-builder/agents/<id>/view or /edit), open the Tools picker.

• [ ] All non-internal tools registered on mastra.tools are visible
• [ ] Internal/system tools (e.g., _internal_*) are hidden

3. Agent picker reflects registered agents

Same agent, open the Sub-agents/Network picker.

• [ ] Stored agents from mastra.agents are visible
• [ ] The current agent itself is hidden (no self-reference)

4. Workflow picker reflects registered workflows

Same agent, open the Workflows picker.

• [ ] All workflows in mastra.workflows are visible
• [ ] Workflows without inputSchema either appear with a warning or are hidden — note which

5. Feature flag off hides picker entirely

If you have shell access to flip a feature flag temporarily, set features.agent.tools = false, restart, reload the agent. (Skip if you don't want to restart.)

• [ ] Tools picker is hidden / disabled
• [ ] Attempting to PATCH { tools: ["x"] } returns 4xx

6. Allowlist (if configured)

If a pickerAllowlist is defined for a surface, only those entries appear.

• [ ] Visible entries are exactly the allowlist
• [ ] Removing an entry from the allowlist on restart removes it from the picker

(Skip if no allowlist is configured in the running example.)

Checklist

• [ ] Features flags reflected in settings
• [ ] Tools picker shows registered tools, hides internals
• [ ] Agent picker shows registered agents, hides self
• [ ] Workflow picker shows registered workflows
• [ ] Feature flag off hides picker
• [ ] Allowlist (if any) is respected

Workspace Reconciliation

Test the config-driven workspace lifecycle managed by ensureBuilderWorkspaces().

Smoke-test scope

The smoke-test agent only runs Step 1 (fresh-startup persistence — already covered by Setup) and Step 5 (non-builder workspaces untouched after restart). Steps 2, 3, 4, and 6 require editing the scaffolded project's src/mastra/index.ts, restarting the dev server multiple times, and reverting changes — they don't fit a single smoke-test run and are kept here as a manual checklist for code changes to ensureBuilderWorkspaces() (packages/editor/src/editor-workspace-reconciliation.test.ts covers most of this at unit level).

When the agent is invoked without --test reconciliation, mark steps 2/3/4/6 as ⏭️ out-of-scope (manual) in the report. Run them by hand if you touched reconciliation code.

Background

On startup, ensureBuilderWorkspaces():

1. Reads the builder config's workspace field 2. Resolves the referenced Workspace instance from the Mastra constructor 3. Snapshots the workspace config (name, filesystem, sandbox) 4. Creates or updates the stored workspace in the DB 5. Archives orphaned workspaces (ones tagged metadata.source: "builder" that are no longer in the config)

Prerequisites

Auth-on session header. Under --auth on, prepend -H "Cookie: $COOKIE" to every curl below (exported from references/auth.md step 0). Snippets omit it for readability; an authenticated run without the cookie returns 401 before any reconciliation assertion can fire.

Resolve the builder workspace ID (the rest of this file assumes $WORKSPACE_ID is set):

WORKSPACE_ID=$(curl -s $BASE/stored/workspaces | jq -r '.workspaces[] | select(.metadata.source == "builder") | .id' | head -1)
echo "WORKSPACE_ID=$WORKSPACE_ID"

Steps

1. Fresh Startup Persistence

This is verified in Setup (section 1). After a fresh server start:

curl -s $BASE/stored/workspaces/$WORKSPACE_ID | jq .

• [ ] Workspace exists with metadata.source: "builder"
• [ ] Filesystem and sandbox config match what's in the scaffolded project's src/mastra/index.ts

To confirm the workspace is also registered at runtime (the runtime registry is separate from the stored projection), check either the list endpoint or the infrastructure endpoint:

# Option A: list response annotates each workspace with runtimeRegistered.
curl -s $BASE/stored/workspaces | jq '.workspaces[] | select(.id == "'"$WORKSPACE_ID"'") | .runtimeRegistered'
# → true

# Option B: infrastructure endpoint reports it as workspace.registered.
curl -s $BASE/editor/builder/infrastructure | jq '.workspace.registered'
# → true

Note: the detail GET (/stored/workspaces/:id) does not include runtimeRegistered; the field is only computed on the list response.

2. Idempotent Restart (No-Op) — manual only

Restart the server without changing any config. Record the workspace's updatedAt and resolvedVersionId before and after.

# Before restart
curl -s $BASE/stored/workspaces/$WORKSPACE_ID | jq '{updatedAt, resolvedVersionId}'

# Restart server (Ctrl+C, then re-run mastra dev from $PROJECT_DIR)
# After restart
curl -s $BASE/stored/workspaces/$WORKSPACE_ID | jq '{updatedAt, resolvedVersionId}'

• [ ] updatedAt is unchanged (no write on no-op)
• [ ] resolvedVersionId is unchanged
• [ ] No duplicate workspace records

3. Config Drift Detection — manual only

Change the workspace config in the scaffolded project's src/mastra/index.ts. For example, change the basePath:

// Before
filesystem: new LocalFilesystem({ basePath: '.mastra/workspace' }),

// After (temporary change for testing)
filesystem: new LocalFilesystem({ basePath: '.mastra/workspace-v2' }),

Restart the server, then check:

curl -s $BASE/stored/workspaces/$WORKSPACE_ID | jq .

• [ ] filesystem.config.basePath is now .mastra/workspace-v2
• [ ] updatedAt changed (DB was updated)
• [ ] resolvedVersionId changed (new version created)
• [ ] id and createdAt unchanged (same workspace, just updated)
• [ ] Still only 1 workspace with this ID

⚠️ Revert the config change after verifying:

filesystem: new LocalFilesystem({ basePath: '.mastra/workspace' }),

4. Orphan Archival — manual only

Change the workspace ID in the builder config:

// In the Workspace constructor
id: 'builder-workspace-v2',

// In the MastraEditor config
workspace: { type: 'id', workspaceId: 'builder-workspace-v2' },

Restart the server, then check both workspaces:

# Old workspace should be archived
curl -s $BASE/stored/workspaces/$WORKSPACE_ID | jq '{status, metadata}'

# New workspace should be created
curl -s $BASE/stored/workspaces/builder-workspace-v2 | jq '{status, metadata}'

• [ ] Old workspace: status is "archived"
• [ ] Old workspace: metadata.source still "builder"
• [ ] New workspace: status is "draft", metadata.source is "builder"
• [ ] Old workspace NOT deleted (preserved for data safety)

⚠️ Revert the config change after verifying. On next restart, the old workspace should un-archive and the v2 workspace should be archived.

5. Non-Builder Workspaces Untouched

Create a user workspace, then restart the server:

# Create user workspace (under --auth on, also pass -H "Cookie: $COOKIE")
curl -s -X POST $BASE/stored/workspaces \
  -H 'Content-Type: application/json' \
  ${COOKIE:+-H "Cookie: $COOKIE"} \
  -d '{"id": "user-workspace", "name": "User Workspace"}' | jq .

# Record state
curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/user-workspace | jq '{status, metadata, updatedAt}'

# Restart server, then check again
curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/user-workspace | jq '{status, metadata, updatedAt}'

• [ ] User workspace is unchanged after restart
• [ ] metadata.source is NOT "builder" (or absent)
• [ ] status unchanged (not archived by reconciliation)

Restart noise: immediately after a dev-server restart, the server log may briefly emit Stored workspace with id ... not found for in-flight requests that started before the restart (e.g. a browser tab polling /stored/workspaces/:id). This is not a reconciliation failure — it's just a stale request landing after the server came back up. Don't include it in product issues unless you can reproduce it with no client connected.

Clean up:

curl -s -X DELETE ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/user-workspace

6. Metadata Backfill — unit-test only

This scenario covers old workspace records that were created before metadata.source was added. It's difficult to test manually without direct DB access. Covered by unit tests in editor-workspace-reconciliation.test.ts.

• [ ] Verified by unit test: it('backfills metadata.source on existing workspace without metadata')

Checklist

• [ ] Fresh startup creates workspace with builder metadata
• [ ] Idempotent restart doesn't modify workspace
• [ ] Config drift updates workspace in DB
• [ ] Orphan archival archives removed workspaces
• [ ] Non-builder workspaces untouched
• [ ] Metadata backfill covered by unit tests

Skill Registry (skills.sh + Library Copy)

Two paths to acquire skills from outside your own authored set:

1. External registry — currently only skills.sh, opt-in via builder.registries.skillsSh.enabled. Browse + install proxies to skills.sh and persists a stored skill with metadata.origin = { type: 'skills-sh', ... }. 2. Library Copy — any authenticated user can copy a public stored skill they don't own. The copy is a fresh private stored skill with metadata.origin = { type: 'library-copy', sourceSkillId, sourceAuthorId, copiedAt }.

Source-of-truth

• Routes: GET /editor/builder/registries, GET /editor/builder/registries/:id/search|popular|preview, POST /editor/builder/registries/:id/install.
• skills.sh enabled flag: builder.registries.skillsSh.enabled (default false).
• Origin schema: packages/server/src/server/schemas/stored-skills.ts (skillOriginSchema discriminated union, types skills-sh and library-copy).
• Library page: /agent-builder/library.

Steps — skills.sh registry

These steps assume skillsSh.enabled = true. The scaffolded project ships with `registries` unset (skills.sh disabled by default). Under that config, registry surfaces are hidden in UI and /editor/builder/registries/skills-sh/* returns 404. Run the registries-list step (step 1) to confirm the disabled-path behavior, then mark steps 2–5 as ⏭️ skills.sh disabled and continue. Only run the full registry walk if you've explicitly enabled builder.registries.skillsSh.enabled in src/mastra/index.ts and restarted.

1. Registries list

curl -s "$BASE/editor/builder/registries" | jq .

• [ ] If enabled: [{ id: 'skills-sh', enabled: true, label: '...' }]
• [ ] If disabled: the entry is still present with enabled: false (e.g. [{ id: 'skills-sh', enabled: false }]) — the registry is listed but not callable. Search/popular/preview/install on a disabled registry return 404 (Registry not found).

2. Search

curl -s "$BASE/editor/builder/registries/skills-sh/search?q=react" | jq '.skills | length'

• [ ] Returns 200 with at least one result for a common term
• [ ] Each result has id, name, installs, topSource

3. Popular

curl -s "$BASE/editor/builder/registries/skills-sh/popular" | jq '.skills | length'

• [ ] Returns 200 with the popular list

4. Preview

Pick a skill from search/popular. Preview takes the GitHub coordinates as query params (owner, repo, path — where path is the skill name within the repo). Source: packages/server/src/server/schemas/builder-registry.ts builderRegistryPreviewQuerySchema.

curl -s "$BASE/editor/builder/registries/skills-sh/preview?owner=OWNER&repo=REPO&path=SKILLNAME" | jq .

• [ ] Returns name, description, instructions (frontmatter stripped), files tree
• [ ] Instructions do NOT start with --- (frontmatter has been stripped)

5. Install

Install body is { owner, repo, skillName, visibility? }. Source: packages/server/src/server/schemas/builder-registry.ts builderRegistryInstallBodySchema.

curl -s -X POST "$BASE/editor/builder/registries/skills-sh/install" \
  -H 'Content-Type: application/json' \
  -d '{ "owner": "OWNER", "repo": "REPO", "skillName": "SKILLNAME" }' | jq .

• [ ] Returns 200/201 with { storedSkillId, name, filesWritten }
• [ ] Subsequent GET /stored/skills/<storedSkillId> shows metadata.origin.type = "skills-sh"
• [ ] metadata.origin.owner, repo, skillName, installedAt present
• [ ] instructions does NOT start with ---

Record INSTALLED_SKILL_ID = storedSkillId.

6. Collision

Re-run the same install:

curl -s -o /tmp/install-err.json -w '%{http_code}\n' \
  -X POST "$BASE/editor/builder/registries/skills-sh/install" \
  -H 'Content-Type: application/json' \
  -d '{ "owner": "OWNER", "repo": "REPO", "skillName": "SKILLNAME" }'
cat /tmp/install-err.json | jq .

• [ ] 409 Conflict
• [ ] Error payload includes existingSkillId (used by UI for "Open existing")

7. UI: Browse dialog

Navigate to /agent-builder/skills.

• [ ] "Browse registry" button visible only when registries are enabled (gated by useBuilderRegistries)
• [ ] Clicking opens dialog with search + popular tabs
• [ ] Selecting a skill shows preview pane (markdown rendered)
• [ ] "Install" creates a new stored skill, dialog closes, list refreshes
• [ ] On collision, toast offers "Open existing" → navigates to the stored skill

8. Origin badge on skill list

In /agent-builder/skills:

• [ ] Installed skill shows an origin badge (skills.sh logo or "skills.sh")
• [ ] Badge links to the source on hover/click

Steps — Library Copy flow

There is no dedicated copy endpoint. Library "Copy" is implemented client-side as a regular POST /stored/skills with the source's fields plus metadata.origin = { type: 'library-copy', sourceSkillId, sourceAuthorId, copiedAt }. Don't grep the server for /copy; verify origin metadata on the resulting record instead. The action goes through the normal create path and is gated by stored-skills:write; the view-page Copy button uses the same check (canCopy = !rbacEnabled || hasPermission('stored-skills:write') in pages/agent-builder/skills/view.tsx). The scaffold grants member that perm, so admin and member both see the button and exercise the copy. Viewer doesn't see the button — mark Copy steps n/a — role lacks stored-skills:write under viewer.

Setup note — requires multi-user data: The Library Copy flow requires at least one public skill owned by a different user so the current user can "Copy" it. A fresh scaffold has none. Options:

>

1. Recommended: run bash .claude/skills/builder-smoke-test/scripts/seed-multi-user.sh after the server has booted at least once. This inserts smoke-seed-public-skill (public) and smoke-seed-private-skill (private) owned by a fake user_seed_other, so all checklist items below become exercisable without a second WorkOS account.

2. Auth-on with a second account: sign in as a different WorkOS user, publish a skill as public, sign back in as the test user and Copy it.

3. Skip: under --auth off, every API-created skill resolves to the same null author, so the Copy affordance won't appear naturally. If you didn't seed and don't have a second account, mark these steps n/a — multi-user data not available and move on; do not flag them as failures.

9. Library page lists public skills

Navigate to /agent-builder/library.

• [ ] Shows at least one public skill authored by someone other than current user (skip with a note if none exist — see setup note above)
• [ ] All shown skills are public and authored by someone other than current user

10. Copy a public skill

Click a skill not authored by you.

• [ ] Detail dialog opens in read-only mode
• [ ] "Copy" button visible (replaces Edit for non-owners on public skills)
• [ ] Click "Copy" → name prompt dialog appears
• [ ] Default name is <source-name>-copy
• [ ] Submit → new private stored skill created
• [ ] Toast confirms; clicking it navigates to the new skill

11. Verify origin metadata via API

curl -s "$BASE/stored/skills/<copiedSkillId>" | jq '.metadata.origin'

• [ ] type is "library-copy"
• [ ] sourceSkillId matches the original
• [ ] sourceAuthorId matches the original author
• [ ] copiedAt is an ISO timestamp

12. Origin badge for copies

In /agent-builder/skills:

• [ ] Copied skill shows a "copied" badge with tooltip "Copied from <original name>"

13. Collision on name

Try to copy the same source skill twice without renaming:

• [ ] Second copy with same name returns 409
• [ ] UI offers to pick a new name

Cleanup

curl -s -X DELETE "$BASE/stored/skills/$INSTALLED_SKILL_ID" | jq .
# delete any library copies created above

Checklist

skills.sh

• [ ] Registries list reflects enabled flag
• [ ] Search returns results
• [ ] Popular returns results
• [ ] Preview strips frontmatter
• [ ] Install persists with metadata.origin.type = 'skills-sh'
• [ ] Re-install returns 409 with existingSkillId
• [ ] UI Browse button gated by enabled registries
• [ ] Origin badge renders on installed skill

Library Copy

• [ ] Library page shows non-owned public skills
• [ ] Copy button visible for non-owners
• [ ] Copy produces a private skill with library-copy origin
• [ ] Origin badge renders on copied skill
• [ ] Name collision returns 409

Workspace CRUD

Test stored workspace create, read, update, and delete via API.

Prerequisites

Auth-on session header. Under --auth on, prepend -H "Cookie: $COOKIE" to every curl in this file (exported from references/auth.md step 0). Snippets below omit it for readability so they read cleanly under --auth off; an authenticated run without the cookie returns 401 before the intended assertion fires.

Visibility asymmetry. Stored workspaces (and the other stored entities) have a visibility field that differs between modes. Under --auth off, the server coerces visibility to "public" on every create — there is no caller to scope ownership against, so private wouldn't be meaningful. Under --auth on, the field is left as whatever was sent (or null when omitted, not "public"). Don't assert visibility === "public" on freshly created workspaces under auth on — assert null or whatever you explicitly sent. Visibility behavior is covered in detail by references/auth.md.

Resolve the builder workspace ID (used in steps 2 and 6):

WORKSPACE_ID=$(curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces | jq -r '.workspaces[] | select(.metadata.source == "builder") | .id' | head -1)

Steps

1. List Workspaces

curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces | jq .

Verify:

• [ ] Response is JSON with workspaces array
• [ ] Builder workspace appears in the list

2. Get Single Workspace

curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/$WORKSPACE_ID | jq .

Verify:

• [ ] Returns the workspace object (not 404)
• [ ] Has id, name, status, filesystem fields
• [ ] metadata.source is "builder" for config-sourced workspaces (metadata may be absent on user-created workspaces — select(.metadata.source == "builder") jq pattern handles both)

3. Create a Test Workspace

curl -s -X POST $BASE/stored/workspaces \
  -H 'Content-Type: application/json' \
  ${COOKIE:+-H "Cookie: $COOKIE"} \
  -d '{
    "id": "smoke-test-workspace",
    "name": "Smoke Test Workspace",
    "filesystem": {
      "provider": "local",
      "config": { "basePath": ".mastra/smoke-test-workspace" }
    }
  }' | jq .

Verify:

• [ ] Returns 200/201 with the created workspace
• [ ] id matches "smoke-test-workspace"
• [ ] metadata.source is NOT "builder" (user-created)

4. Update the Test Workspace

curl -s -X PATCH $BASE/stored/workspaces/smoke-test-workspace \
  -H 'Content-Type: application/json' \
  ${COOKIE:+-H "Cookie: $COOKIE"} \
  -d '{
    "name": "Updated Smoke Test Workspace",
    "description": "Updated during smoke test"
  }' | jq .

Verify:

• [ ] Returns updated workspace
• [ ] name is now "Updated Smoke Test Workspace"
• [ ] updatedAt changed

5. Delete the Test Workspace

curl -s -X DELETE ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/smoke-test-workspace -o /dev/null -w "%{http_code}\n"

Verify:

• [ ] HTTP 200 or 204 for owner/admin; HTTP 403 for member/viewer (stored-workspaces:delete is admin-only — see references/permissions.md). On 403, mark "delete blocked by RBAC (expected)" and leave the test workspace in place; subsequent steps should accept its presence.
• [ ] When delete succeeds, GET /stored/workspaces/smoke-test-workspace returns 404

6. Verify Builder Workspace is Untouched

curl -s ${COOKIE:+-H "Cookie: $COOKIE"} $BASE/stored/workspaces/$WORKSPACE_ID | jq .

• [ ] Builder workspace still exists, unchanged

Checklist

• [ ] List returns workspaces array
• [ ] Get returns single workspace by ID
• [ ] Create works for user-created workspace
• [ ] Update modifies name/description
• [ ] Delete removes workspace
• [ ] Builder workspace unaffected by test CRUD