Aspire Orchestration

name: aspire-orchestration-eval
description: Evaluates the aspire-orchestration skill against the 5 core complaints from
  microsoft/aspire#15801 plus detection, worktree, and agent-mode scenarios.
version: "1.0"
type: capability
tags:
  skill: aspire-orchestration
config:
  runs: 3
  timeout: 120s
  executor: copilot-sdk
  model: gpt-5-mini
stimuli:
  - name: orch-update-self-001
    prompt: How do I upgrade my Aspire CLI to the latest 13.3?
    tags:
      priority: p1
      area:
        - aspire-13-3
        - update
    environment:
      files:
        - src: ../../../evals/csharp-apphost/MyApp.AppHost/MyApp.AppHost.csproj
          dest: csharp-apphost/MyApp.AppHost/MyApp.AppHost.csproj
        - src: ../../../evals/csharp-apphost/aspire.config.json
          dest: csharp-apphost/aspire.config.json
    graders:
      - type: prompt
        name: uses_update_self
        config:
          prompt: Does the assistant's response recommend `aspire update --self` to upgrade the Aspire CLI
            itself? This is the safe, no-side-effect upgrade path that an agent can run unattended.
      - type: prompt
        name: distinguishes_project_update
        config:
          prompt: Does the assistant's response distinguish `aspire update --self` (CLI upgrade) from bare
            `aspire update` (which modifies project package references and `aspire.config.json`)? It
            should warn that bare `aspire update` edits project files and therefore should be run
            with user approval rather than unattended.
      - type: output-not-contains
        config:
          substring: automatically run aspire update
        name: never_runs_project_update_unattended
      - type: output-not-contains
        config:
          substring: run aspire update without confirmation
      - type: output-contains
        config:
          substring: aspire update --self
      - type: output-not-contains
        config:
          substring: dotnet run
        name: never_dotnet_run
      - type: output-not-contains
        config:
          substring: dotnet build
      - type: output-not-contains
        config:
          substring: while true
        name: never_curl_polling
      - type: output-not-contains
        config:
          substring: sleep
      - type: output-not-contains
        config:
          substring: curl.*health
  - name: orch-aspireify-001
    prompt: I just ran `aspire init` in my existing Node.js + .NET repo and it created an apphost.cs and
      aspire.config.json. What's next? Wire up my services.
    tags:
      priority: p0
      area:
        - init-workflow
        - aspireify-handoff
    environment:
      files:
        - src: ../../../evals/non-aspire/MyApp/MyApp.csproj
          dest: non-aspire/MyApp/MyApp.csproj
        - src: ../../../evals/non-aspire/MyApp/Program.cs
          dest: non-aspire/MyApp/Program.cs
    graders:
      - type: prompt
        name: hands_off_to_aspireify
        config:
          prompt: Does the assistant's response explicitly hand the AppHost wiring step off to the `aspireify`
            skill (in-plugin sibling at `../aspireify/SKILL.md`, OR the project-local
            `.agents/skills/aspireify/SKILL.md` if installed by `aspire init`)? The orchestration
            skill itself should defer scanning the repo, choosing resources, and editing the AppHost
            to aspireify. Answer based on intent.
      - type: prompt
        name: mentions_project_local_precedence
        config:
          prompt: Does the assistant's response acknowledge that if a project-local
            `.agents/skills/aspireify/SKILL.md` exists, the agent should defer to it (with a warning
            to the user) instead of using the in-plugin sibling?
      - type: output-not-contains
        config:
          substring: I'll write the AppHost
        name: never_handauthors_apphost
      - type: output-not-contains
        config:
          substring: Let me edit apphost.cs
      -