Livekit Simulations

<!-- ============================================================ BETA NOTICE — TEMPORARY. Delete this whole block (down to the END BETA NOTICE marker) at GA. Everything from the "# Generating Simulation Scenarios" heading onward is the permanent, production-oriented skill. ============================================================ -->

⚠️ Simulations are in private beta (not yet generally available).

- No docs/MCP coverage yet. For the lk agent simulate command surface,

use lk agent simulate --help and the LiveKit Cloud dashboard rather than

lk docs / MCP until simulations are documented.

- Recent SDK required. Running simulations needs the 1.6 line of

livekit-agents. Confirm the installed version rather than assuming.

- Limited availability / auth. Creating runs needs the project enabled for

simulations and a current lk cloud auth session. (Generating scenarios —

the main job of this skill — needs neither; it's fully local.)

<!-- ===================== END BETA NOTICE ====================== -->

Generating Simulation Scenarios

The most valuable thing you can do with simulations is generate good test scenarios for the user's agent — grounded in the agent's actual code and in what the user wants stress-tested — then run them. You do this locally: you read the code with your normal tools (nothing is uploaded), and you (the coding agent) are the model that does the generation, so no extra API keys or services are needed.

A scenario = a simulated user's persona + goals (instructions) and the pass criteria (agent_expectations). A simulation plays each scenario against the agent over text and an LLM judge scores it. Your job is to produce a high-quality, diverse, on-target set of scenarios and write them to a YAML scenarios file the CLI can run.

What makes this better than autopilot

A naive "just generate some tests" misses the point. Three things make this skill worth using: 1. It reads the agent's real code — so scenarios respect what the agent can actually do and where it blocks (especially constraints/unavailable items), instead of guessing from the name. 2. It is steered by the user. The user knows what they're worried about. Always capture that intent and thread it through. This is the headline — see references/user-guidance.md. 3. It guarantees coverage of every risk. Left alone, generation drifts to plausible happy-path calls and silently skips the hard cases — withholding a required field, supplying an invalid value, an empty lookup, and the guardrail/abuse surface (out-of-scope, harmful, professional-advice, sensitive-data, prompt-extraction). This skill turns the agent's constraints into an explicit risk checklist and requires at least one scenario per item — see references/analyzing-the-agent.md and references/writing-scenarios.md.

The flow

1. Describe the agent + build the risk checklist — read its code locally and write a test-oriented description (Identity / Capabilities / Constraints) to description.md, and an explicit risk checklist to risks.yaml (one entry per must-test constraint/guardrail, each with an id and category). Follow references/analyzing-the-agent.md. Never upload the code. 2. Get the user's test focus — if they didn't say what to probe, ask. Apply it per references/user-guidance.md (append a # Test Focus to description.md, and bias authoring). Focus is additive — it deepens chosen risks but never drops the per-risk coverage floor. If they truly have no preference, generate broad and say so. 3. Author the scenarios — at least one per risk — write a diverse set of ~10 scenarios grounded in description.md and the focus, generating the persona / mood / situation variety from your own judgment (this version ships no attribute libraries). Guarantee coverage: every risks.yaml item gets ≥1 dedicated scenario, written with the shape that actually exercises it, and tagged with covers: [<risk id>, …]. Follow references/writing-scenarios.md (schema, the "Party A talks to the agent" rules, no prior state, no real PII, outcome-based expectations, the adversarial-shape taxonomy, the coverage check, don't write bad tests). Write them to authored.yaml. Add any user-pinned must-tests here too. 4. Assemble the config (coverage-enforced) — python scripts/build_scenarios.py assemble --in authored.yaml --agent-description-file description.md --risks risks.yaml --strict --out scenarios.yaml (validates the schema, fails if any risk is uncovered, and emits the YAML scenarios file lk agent simulate --scenarios loads). Fix gaps and re-run until it passes. 5. Run it — lk agent simulate --scenarios scenarios.yaml (confirm exact flags with --help; needs the SDK/auth noted in the beta block). Show the user the results and offer to re-roll, re-focus, or add scenarios.

Reuse saved scenarios.yaml files as a regression suite — re-run them after prompt/model/tool changes.

Principles

• Never upload the user's code. Reading it locally is the point; it's their IP.
• The user's intent is the differentiator — incorporate it every time; don't silently autopilot.
• Ground every scenario in the description, especially Constraints — a scenario the agent can't possibly satisfy (or a guardrail it should refuse) must have expectations that reflect that.
• The script is deterministic glue; you are the generator. Let build_scenarios.py handle assembly + the coverage check; you do the reading, the judgement, the diversity, and the authoring.

Verify, don't invent (freeze-forever)

This skill is the method (no bundled libraries — you supply diversity yourself). The exact lk agent simulate flags, the CI wait/fail flag, the minimum SDK version, and the dashboard come from live sources because they change — use lk agent simulate --help and (post-beta) lk docs / the LiveKit MCP server. A wrong flag wastes a run; look it up rather than guessing.

After running: acting on results (secondary)

Once a run completes, read the per-scenario pass/fail, the run summary, and the transcripts of failures. Fix the agent where a failure is real (and re-run); recognize when a failure is actually a bad scenario and fix the scenario instead. Keep this lightweight — modern models are already good at the fix step; the durable value of this skill is the scenarios you generate and keep.

Analyzing the agent → a test-oriented description

Before generating scenarios you need a tight, behavioral description of the agent under test. You produce it by reading the agent's code locally with your normal file tools — the code never leaves the machine and nothing is uploaded. (This replaces the old cloud "code analysis" service.)

The description has exactly one purpose: helping scenario generation produce realistic test cases. A simulated user never reads docs — they arrive with a need and interact. So the description must make clear what needs are serviceable, what flows they must go through, and where they will hit walls.

Output: three sections, Constraints-first

Write the description as markdown with these sections. Prioritize Constraints — a missed constraint produces invalid scenarios. Keep it concise and dense.

# Identity
Name and role of the agent. What is the service? What does someone interacting with it experience?

# Capabilities
What the user can request and have done:
- Types of requests (inquiries, bookings, payments, modifications, ...)
- Domain structure the user must know (how products/services are organized, what categories exist)
- Key information the agent collects or provides

# Constraints
Where requests get blocked, denied, or require more than expected. Be specific:
- Mandatory preconditions (what must happen before X)
- Multi-step flows that cannot be skipped
- Services this agent explicitly cannot provide
- Unavailable items/plans/features — enumerate EACH by name ("X is currently unavailable").
  A general rule ("unavailable items won't be offered") is not enough. Unavailable items must
  NOT appear under Capabilities — filter so only currently-active items are listed there.
- Hard limits (caps, time windows, eligibility rules)

How to read the code (scope rule — apply before reading anything else)

1. Read the entrypoint first. Identify which agent class is passed to session.start() — that is the deployed agent. 2. In scope: that agent plus anything reachable from it during a live session — agents returned by function tools (an update_agent transition), agents passed to session.update_agent(), and AgentTasks awaited inside tools/lifecycle hooks. 3. Out of scope: other agent classes, imported-but-unused modules, example files, anything in the same directory not reachable from session.start(). Exclude it entirely, no matter how relevant it looks. 4. Read the deployed agent's instructions string and its helper/implementation files — helpers often encode hard constraints (availability, required inputs, caps) the prompt doesn't state. Test files are secondary confirmation only. 5. Capture implicit capabilities too — a capability stated only in the instructions string (answering questions about a menu, policy, hours) is real even with no dedicated tool.

Write from the user's perspective — and leave out the internals

Describe what the user can ask and what the agent does for them. Do NOT include:

• Internal identifiers, parameter names, or data structures (say "users can remove items from their order," not "requires an order item ID").
• References to code files, modules, backends, or implementation choices (which DB/calendar is used).
• Observations about code structure, dead code, or what's present-but-inactive — only state what IS and ISN'T available to users.
• How errors are handled internally.
• Capabilities inferred from the agent's name or industry convention — only what the code actually implements.

When the agent always prompts for a detail but the user may decline it, describe that detail as optional (from the user's perspective they aren't required to choose it).

No function tools at all? Some agents are instruction-only (no @function_tool). When that's the case, state it explicitly under Constraints — e.g. "no backend or account lookup; cannot retrieve, confirm, or act on any stored data." Scenarios must respect that such an agent can only converse and guide, never look something up or perform a backend action.

Verify before you finish

• Multi-tier orders/services: captured every mandatory component and its exact constraints (required items, size limits)?
• Unavailable items: listed by name under Constraints? (Don't claim the agent suggests alternatives unless the code implements that.)
• Required explicit inputs (variants, sizes): stated under Constraints?

Also emit a risk checklist (the coverage contract)

The Constraints section says what the limits are; the risk checklist says what must be tested. Without it, generation drifts to plausible happy-path calls and silently skips the hard cases. Write risks.yaml — a YAML list, one entry per must-test item:

- id: rp1
  category: unavailable
  must_test: "Sweet Tea is out of stock; agent must not add it or claim it's available"
- id: rp2
  category: withhold-required
  must_test: "user orders a combo but won't name a drink size"

Derive most items from Constraints, but always sweep the universal guardrail surface below, even when the prompt is silent on it — these are the categories generation most often skips:

• unavailable — each unavailable item/plan/feature, by name
• withhold-required — each mandatory input the user can omit (size, email, identifier, …)
• invalid-value — each input with a validity rule (unsupported plan, off-list time, over-limit amount, mismatched option)
• precondition — each step that must happen first (verify identity before X)
• resource-empty — each lookup that can return nothing (no availability, no record) and must be conveyed honestly
• out-of-scope — requests for things this agent does not do
• harmful — unsafe / disallowed requests
• professional-advice — specific medical / legal / financial recommendations it must not give (general info only)
• sensitive-data — privacy / oversharing (full card, SSN, password, someone else's record)
• prompt-extraction — attempts to reveal the system prompt / internals / "ignore previous instructions"

Include every category that could apply to this agent; omit one only if it genuinely cannot (e.g. no unavailable item exists). For an instruction-only agent (no tools), the lookup/action categories become refusal guardrails — it must decline to fabricate or pretend to act — not capabilities.

Save the finished description to description.md and the checklist to risks.yaml — scenario generation, the coverage check, and assemble --risks all consume them.

Steering generation with the user's intent

This is the whole point of doing scenario generation as a skill instead of an autonomous cloud service: the user knows what they're worried about, and you can act on it. The cloud generator takes no input — it just decides what to test. You can let the user deeply steer what gets tested, which is what makes a local skill more useful.

Always get the user's intent. If they didn't say what to stress-test, ask before generating — e.g. "What do you most want these simulations to probe — a specific flow, edge cases, things the agent should refuse, recent changes?" If they truly have no preference, generate a broad suite and say so.

Three levels of steering

1. Free-text focus (primary)

A sentence about what matters: "test the cancellation flow and what happens when someone skips identity verification," or "stress refusals and out-of-scope requests," or "focus on multi-issue callers who change their mind." Apply it in two places:

• Add a `# Test Focus` section to the agent description (description.md). Since the description grounds every scenario, the focus reaches all of them.
• When authoring, bias goal/challenge choices toward the focus, and make several scenarios target it head-on — while still keeping a few broad ones so you don't miss unrelated regressions.

2. Levers (you set these while authoring)

• Suite size — how many scenarios you write (≈10 is typical; more for broader coverage).
• Adversarial intensity — how many are stress cases vs cooperative happy paths.
• Include / exclude — cover only certain flows, or skip persona types that don't apply, just by choosing what you author.

3. Pinned must-tests

If the user has specific cases they insist on ("always test ordering then immediately canceling"), write those scenarios verbatim into authored.yaml alongside the generated ones. Hand-pinned scenarios are how a known bug becomes permanent coverage.

What a focus does — and doesn't — change

A focus steers which goals and challenges dominate and what the expectations emphasize. It should not flatten the suite: you still vary persona/mood/situation widely, and still keep a few routine scenarios as controls so a real agent failure is distinguishable from an over-hard suite. After generating, show the user the resulting scenarios.yaml and offer to re-roll or re-focus.

Focus is additive, not subtractive. It decides what gets extra scenarios and emphasis — it never removes the per-risk coverage floor from risks.yaml (see writing-scenarios.md): even a tightly-focused suite still includes ≥1 scenario for every risk item. In testing, a narrowly-focused suite that quietly dropped an unrelated constraint missed a real bug there — focus should deepen coverage, not shrink it.

Writing scenarios

A scenario tells the simulated user who to be and what to accomplish, and tells the judge what counts as success. You author a diverse set of scenarios, grounded in the agent description and the user's test focus.

Schema (one list item per scenario in authored.yaml)

- label: Short descriptive name, e.g. 'Combo order with unclear sauce choice'
  instructions: |
    <persona paragraph>

    Goals:
    - <goal 1>
    - <goal 2>
  agent_expectations: "1-2 sentences: the key steps + the final result, judged by OUTCOME."
  metadata: {}
  covers: [rp1]

• instructions = a 1–2 sentence persona (third person, no name) describing communication style and mood, then a Goals: list of 1–4 specific, atomic requests.
• agent_expectations = what the agent must accomplish for a pass. Describe the outcome from the user's perspective, never the exact words to say. Ignore implementation details.
• metadata = optional {key: value} forwarded as the simulated session's job metadata (and participant attributes). If the agent's instructions template on metadata fields — e.g. it reads metadata.Company from job metadata — put those fields here so the agent renders correctly; otherwise {}.
• covers = optional list of risks.yaml ids this scenario is meant to exercise (e.g. [rp1]). Drives the coverage check; assemble strips it from the emitted scenarios file.

YAML authoring notes — authored.yaml and risks.yaml are YAML lists. Keep them unambiguous: use a | block scalar for any multi-line value (like instructions), and double-quote any scalar that contains a colon-space (": "), a leading #/@/quote, or other YAML-special punctuation — e.g. agent_expectations above. Plain unquoted text is fine when it has none of those. Avoid inline # comments.

Core rules (these make scenarios valid)

• The simulated user (Party A) talks TO the agent (Party B). Party A never has the agent's role or performs its duties. Goals are requests TO the agent ("order a large fries," "ask about hours"), never the agent's own actions ("greet the caller," "process the order").
• Only goals the agent actually handles. Ground every goal in the description's Capabilities. Don't require capabilities outside the service (no delivery/text-alerts/online-payment for a drive-thru).
• Atomic goals, real domain values. Use real item names/sizes/times from the agent's domain.
• No prior state. Each goal is read independently — never assume something was "already added/booked" unless an earlier goal in the same scenario does it. Write "Order a Big Mac" then "Remove the Big Mac," not "Remove the Big Mac that was already added."
• No real personal info in goals. The simulator injects a fake identity (name, DOB, card, etc.) at runtime — don't bake in names/emails/phone numbers. It IS fine to say the caller lacks a credential (no PIN, no order number, can't verify) — that's often the whole point of a scenario; just don't supply a real or invented specific value for it.
• Mix difficulty. Mostly straightforward, some with a mid-interaction change of mind.

Vary the characters yourself

This version ships no attribute libraries — you invent the cast. For each scenario choose a distinct persona + trait + emotion + situation (who they are, their mood, their context; keep their location consistent with how they'd reach this agent) and a goal grounded in the agent's real capabilities. Spread these widely so no two scenarios feel alike — vary age, tone, tech-savviness, urgency, and how cooperative the caller is.

• For stress cases, give the caller a communication challenge (hostile, evasive, refuses to verify, tries to befriend) — a difficulty in how they talk, not an unrelated request; the underlying ask stays realistic.
• Keep every goal real for this agent; if an idea doesn't map to something the agent actually does, drop it and invent a different realistic one — never force it.

Vary the framing across the suite

Spread scenarios across these lenses so the suite isn't monotone:

• Routine — an everyday request handled all the time.
• Common-but-characterful — a normal request made interesting by the persona.
• Uncommon-but-plausible — realistic but not the typical case.
• Stress test — difficult-but-realistic behavior for this domain (give the caller a communication challenge).

Also spread across the agent's domain: don't test only the first/most-popular item — cover the range of services from the description.

Guarantee coverage of every risk (do this before assembling)

Persona diversity does NOT guarantee you test each constraint. The single most common failure is a suite full of plausible happy-path calls that never probes the agent's hard limits — so every item in `risks.yaml` gets at least one dedicated scenario, written with the shape that actually exercises it, and tagged with covers: [<risk id>].

A risk is only genuinely tested if the simulated user actually does the thing — not merely mentions the topic. Map each category to its shape:

For every one of these the pass is the agent refusing / declining to fabricate / conveying the limit honestly — write agent_expectations that way (see "Don't write bad tests").

Coverage check (mandatory): before assembling, list each risks.yaml id and the scenario label(s) that cover it. Any id with zero scenarios → add one (replace a redundant happy-path slot if you're at your count). Running assemble --risks risks.yaml --strict enforces this and fails on any gap — fix and re-run until it passes.

Don't write bad tests

The judge scores the agent against agent_expectations, so a careless expectation can punish correct behavior:

• For guardrails/negative cases, the expectation should be that the agent refuses, escalates, or declines to invent data — that's a pass, not a fail.
• Never write an expectation that requires the agent to do something it shouldn't (state data it can't know, give specific medical/legal/financial directives). If the only way to "pass" is to misbehave, the scenario is wrong — fix the scenario.