Eval Creator Ci

Eval Creator CI

Install

gh skill install pskoett/pskoett-skills eval-creator-ci

For interactive sessions, use:

gh skill install pskoett/pskoett-skills eval-creator

Fallback using the Agent Skills CLI:

npx skills add pskoett/pskoett-skills/skills/eval-creator-ci
npx skills add pskoett/pskoett-skills/skills/eval-creator

Purpose

Runs the outer loop's regress-test step in CI. Executes all eval cases in .evals/, reports pass/fail results, and optionally blocks merges on regressions. Can also create new eval cases from promotion candidates flagged by learning-aggregator-ci.

The interactive eval-creator skill is designed for in-session use where the user creates evals and runs them with immediate feedback. This CI variant runs on schedule or per-PR and posts results as check annotations.

Context Limitation (Important)

CI agents do not have implementation context. They execute mechanical verification methods (grep checks, command checks, file checks, rule checks) defined in eval case files. They do not interpret results beyond pass/fail — nuanced judgment is left to human review of the posted report.

Prerequisites

• GitHub Actions enabled on the repository
• gh CLI authenticated with repo access
• gh-aw extension installed (gh extension install github/gh-aw, v0.40.1+)
• .evals/ directory with eval cases (created by eval-creator or eval-creator-ci)
• .evals/EVAL_INDEX.md with eval case index

CI Contract

Hard rules for headless execution:

1. Eval execution is read-only for code — eval cases read files and run check commands but do not modify source code 2. Eval case creation writes to `.evals/` only — when creating new evals from promotion candidates 3. Headless — no interactive prompts, no approval gates 4. Structured output — emit results as YAML under eval_creator_ci key 5. Gate policy — can fail the check run on eval regressions (configurable) 6. Single comment — post one consolidated results comment per run

Authoring Workflow (gh-aw)

1. Copy references/workflow-example.md into .github/workflows/eval-creator-ci.md 2. Customize trigger and gate policy 3. Validate: gh aw compile (add --actionlint --zizmor for security scan) 4. Push to enable

Persistence and Chaining

• `cache-memory:` stores eval run history (last-run dates, result trends) across runs. Avoids re-running evals that haven't changed.
• `workflow_call:` in create mode, triggered by learning-aggregator-ci via call-workflow. Receives promotion candidates as workflow inputs.
• `upload-artifact:` persists eval results YAML for downstream consumption.

Workflow Rules

The CI agent follows these rules in order:

Mode: Run Evals (default)

1. Read .evals/EVAL_INDEX.md to get the list of all eval cases 2. For each eval case file in .evals/cases/: a. Read the eval case metadata and verification method b. Check preconditions — if not met, mark as skip c. Execute the verification method:

• grep-check: Search target files for pattern, compare to expected (found/not_found)
• command-check: Run the command, check exit code and/or output
• file-check: Verify file or section exists
• rule-check: Read target file, search for expected content

d. Compare result to expected outcome e. Record pass/fail/skip 3. Update .evals/EVAL_INDEX.md with last-run date and last-result for each case 4. Emit structured YAML under key eval_creator_ci 5. Post results summary as a PR comment or check annotation 6. If gate policy is enabled and any eval fails: fail the check run

Mode: Create Evals (from promotion candidates)

1. Read the learning_aggregator_ci artifact or gap report from the most recent learning-aggregator-ci run 2. For each promotion-ready pattern with eval_candidate: true: a. Determine the appropriate verification method based on the pattern type b. Create the eval case file in .evals/cases/ with proper frontmatter c. Add the entry to .evals/EVAL_INDEX.md 3. Commit the new eval cases (if running with write permissions) 4. Report created evals in the output

Output Schema

eval_creator_ci:
  version: "0.1.0"
  source:
    run_id: "<workflow run ID>"
    trigger: "pull_request | schedule | workflow_dispatch"
    run_date: "YYYY-MM-DD"
  mode: "run | create | both"
  run_results:
    total: 12
    passed: 10
    failed: 1
    skipped: 1
    failures:
      - id: "eval-20260301-001"
        pattern_key: "harden.input_validation"
        rule_summary: "Always validate external inputs"
        expected: "not_found"
        actual: "found"
        target: "src/api/handler.ts"
        recovery_action: "Add input validation to new handler endpoint"
    skips:
      - id: "eval-20260315-003"
        reason: "Precondition not met: project does not use TypeScript"
  create_results:
    created: 2
    cases:
      - id: "eval-20260411-001"
        pattern_key: "simplify.dead_code"
        verification_method: "grep-check"
        source_learning: "LRN-20260301-001"
      - id: "eval-20260411-002"
        pattern_key: "harden.authorization"
        verification_method: "rule-check"
        source_learning: "LRN-20260315-003"
  summary:
    regressions: 1
    new_evals_created: 2
    gate_result: "fail"
    followup_required: true

Recommended Outputs

Gate Policy

Configure blocking behavior:

Default: advisory (report but don't block). Teams should escalate to strict once eval coverage stabilizes.

Trigger Configuration

Recommended: per-PR + weekly schedule

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
  schedule:
    - cron: '0 10 * * 1'  # Monday 10am UTC (after learning-aggregator-ci)
  workflow_dispatch:

Per-PR runs catch regressions before merge. Weekly runs catch drift in the harness itself. Schedule after learning-aggregator-ci so new evals from promotions are available.

Integration with Other Skills

Upstream (feeds from)

• eval-creator (interactive) — creates eval cases manually
• learning-aggregator-ci — produces promotion candidates with eval_candidate: true
• harness-updater (interactive) — flags eval candidates after promoting patterns

Downstream (feeds into)

• self-improvement / self-improvement-ci — regression failures become new error entries in .learnings/
• Human review — failure report posted for team triage
• PR merge gate — can block merge on regressions (configurable)

Data Flow

learning-aggregator-ci → promotion candidates (eval_candidate: true)
                              ↓
                    eval-creator-ci (create mode)
                              ↓
                         .evals/cases/
                              ↓
                    eval-creator-ci (run mode, per-PR)
                              ↓
                    pass/fail report → PR comment + check annotation
                              ↓
                    regression failures → self-improvement-ci → .learnings/

Differences from Interactive Version

Workflow Example (Non-Active)

This is an example template only. Keep it outside .github/workflows so nothing runs automatically.

When you are ready to enable CI automation: 1. Copy this template into .github/workflows/eval-creator-ci.md 2. Customize trigger, gate policy, and schedule 3. Validate with gh aw compile (add --actionlint --zizmor for security scan)

Run Mode (per-PR regression check)

---
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
  workflow_dispatch:

permissions:
  contents: read
  actions: read
  pull-requests: read

tools:
  github:
    toolsets: [pull_requests, actions]
  cache-memory: true
  # Optional: durable git-branch persistence for eval run history and the
  # created-pattern ledger. Survives beyond the 7-day cache-memory window.
  repo-memory:
    branch-name: learnings/eval-state
    max-file-size: 51200

safe-outputs:
  add-comment:
    max: 1
    hide-older-comments: true
  upload-artifact:
    max-uploads: 1

tracker-id: eval-creator

concurrency:
  group: eval-creator-run
  cancel-in-progress: false

strict: true
---

1. Read `.evals/EVAL_INDEX.md` to get the list of all eval cases. If the file does not exist or `.evals/` is empty, report zero evals and exit cleanly.

2. Check cache-memory at `/tmp/gh-aw/cache-memory/eval-run-history.json` for previous run results. Skip evals where the target files have not changed since last successful run (use git diff against cached commit SHA).

3. For each eval case file listed in the index, read its frontmatter and verification method.

4. Check the precondition for each eval. If the precondition is not met (e.g., required file does not exist, project does not use the relevant framework), mark the eval as `skip` and move to the next one.

5. Execute the verification method:
   - `grep-check`: Use grep/ripgrep to search target files for the pattern. Compare result to `expect` (found or not_found).
   - `command-check`: Run the specified command. Check exit code against `expect_exit`. Optionally check output content.
   - `file-check`: Verify the target file exists and optionally that the specified section exists within it.
   - `rule-check`: Read the target file and search for the expected content string. Compare to `expect` (found or not_found).

6. Record the result (pass, fail, skip) for each eval case.

7. Write updated run history to cache-memory at `/tmp/gh-aw/cache-memory/eval-run-history.json` with current commit SHA and results.

8. Emit the full results as structured YAML under key `eval_creator_ci` following the output schema in the skill definition.

9. Upload the results YAML as a workflow artifact named `eval-results`.

10. Post a human-readable summary as a PR comment. Format: failures first (with eval ID, pattern-key, expected vs actual, recovery action), then passes, then skips.

11. If gate policy is `strict`: fail the check run if any eval fails. If `critical-only`: fail only if a failed eval's source pattern has severity critical or high. If `advisory`: report only, do not fail.

12. Do not modify source code files. Only `.evals/EVAL_INDEX.md` metadata (last-run, last-result) may be updated.

Create Mode (triggered by learning-aggregator-ci)

---
on:
  workflow_call:
    inputs:
      gap_report_artifact:
        description: "Artifact name containing the gap report YAML"
        required: false
        type: string
        default: "gap-report"
  schedule:
    - cron: '0 10 * * 1'  # Monday 10am UTC (after learning-aggregator-ci)
  workflow_dispatch:

permissions:
  contents: read
  actions: read
  issues: read
  pull-requests: read

tools:
  github:
    toolsets: [pull_requests, actions, issues]
  cache-memory: true
  # Optional: durable git-branch persistence for the created-pattern ledger
  # so newly-promoted patterns aren't re-created on every scheduled run.
  repo-memory:
    branch-name: learnings/eval-state
    max-file-size: 51200

safe-outputs:
  add-comment:
    max: 1
  upload-artifact:
    max-uploads: 1
  create-pull-request:
    max: 1
    draft: true

tracker-id: eval-creator

concurrency:
  group: eval-creator-create
  cancel-in-progress: false

strict: true
---

1. If triggered via workflow_call: download the gap report artifact. Otherwise, read the most recent `learning_aggregator_ci` workflow artifact or gap report issue comment.

2. Extract promotion-ready patterns where `eval_candidate` is true.

3. Check cache-memory at `/tmp/gh-aw/cache-memory/eval-created-patterns.json` for patterns that already have eval cases. Skip duplicates.

4. For each new candidate, determine the appropriate verification method:
   - Knowledge gaps about conventions → `rule-check` (verify the rule exists in project instruction files)
   - Input validation patterns → `grep-check` (search for unvalidated input patterns)
   - Tool/dependency patterns → `command-check` (run the relevant tool command)
   - File structure patterns → `file-check` (verify expected files/sections exist)

5. Create an eval case file in `.evals/cases/` with proper frontmatter: id, pattern-key, source learning IDs, promoted-rule text, verification method, expected result, and recovery action.

6. Create the `.evals/` and `.evals/cases/` directories if they do not exist.

7. Add each new eval case to `.evals/EVAL_INDEX.md`. Create the index file if it does not exist.

8. Update cache-memory with the newly created pattern keys to prevent re-creation on next run.

9. Emit the creation results as structured YAML under key `eval_creator_ci` with `mode: create`.

10. Post a summary comment listing the new eval cases created.

11. Emit the new files via the `create-pull-request` safe-output (title: "chore: add eval cases from learning-aggregator-ci"). gh-aw opens the PR from a separate job that holds the write scope — the agentic job stays `contents: read` and must not `git commit` directly.

12. Do not modify source code files. Only write to `.evals/` directory.