
Eval Harness Updater
- 44 installs
- 36 repo stars
- Updated July 14, 2026
- oimiragieo/agent-studio
Helps with ai & agent building tasks.
About
eval-harness-updater is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.
- eval-harness-updater
- AI & Agent Building
- AI-coding skill
Eval Harness Updater by the numbers
- 44 all-time installs (skills.sh)
- Ranked #7,851 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 4, 2026 (Skillselion catalog sync)
npx skills add https://github.com/oimiragieo/agent-studio --skill eval-harness-updaterAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 44 |
|---|---|
| repo stars | ★ 36 |
| Last updated | July 14, 2026 |
| Repository | oimiragieo/agent-studio ↗ |
What it does
Helps with ai & agent building tasks.
Files
Eval Harness Updater
Refresh eval harnesses to keep live + fallback modes actionable under unstable environments.
Focus Areas
- Prompt and parser drift
- Timeout/partial-stream handling
- SLO and regression gates
- Dual-run fallback consistency
Workflow
1. Resolve harness path. 2. Research test/eval best practices (Exa + arXiv — see Research Gate below). 3. Add RED regressions for parsing and timeout edge cases. 4. Patch minimal harness logic. 5. Validate eval outputs and CI gates. 6. Resolve companion artifact gaps (see Cross-Reference table below).
Research Gate (Exa + arXiv — BOTH MANDATORY)
Before proposing harness changes, gather current best practices:
1. Use Exa for implementation and ecosystem patterns:
mcp__Exa__web_search_exa({ query: 'LLM eval harness 2025 best practices' })mcp__Exa__get_code_context_exa({ query: 'eval harness parser reliability timeout handling' })
2. Search arXiv for academic research on evaluation methodology (mandatory):
- Via Exa:
mcp__Exa__web_search_exa({ query: 'site:arxiv.org LLM evaluation harness 2024 2025' }) - Direct API:
WebFetch({ url: 'https://arxiv.org/search/?query=LLM+evaluation+harness&searchtype=all&start=0' })
3. Record decisions, constraints, and non-goals in memory learnings.
arXiv is mandatory (not fallback) when topic involves: LLM evaluation, agent evaluation, SLO gates, regression testing methodology, or parser reliability.
Cross-Reference: Creator Ecosystem
This skill is part of the Creator Ecosystem. When research uncovers gaps, trigger the appropriate companion creator:
| Gap Discovered | Required Artifact | Creator to Invoke | When |
|---|---|---|---|
| Domain knowledge needs a reusable skill | skill | Skill({ skill: 'skill-creator' }) | Gap is a full skill domain |
| Existing skill has incomplete coverage | skill update | Skill({ skill: 'skill-updater' }) | Close skill exists but incomplete |
| Capability needs a dedicated agent | agent | Skill({ skill: 'agent-creator' }) | Agent to own the capability |
| Existing agent needs capability update | agent update | Skill({ skill: 'agent-updater' }) | Close agent exists but incomplete |
| Domain needs code/project scaffolding | template | Skill({ skill: 'template-creator' }) | Reusable code patterns needed |
| Behavior needs pre/post execution guards | hook | Skill({ skill: 'hook-creator' }) | Enforcement behavior required |
| Process needs multi-phase orchestration | workflow | Skill({ skill: 'workflow-creator' }) | Multi-step coordination needed |
| Artifact needs structured I/O validation | schema | Skill({ skill: 'schema-creator' }) | JSON schema for artifact I/O |
| User interaction needs a slash command | command | Skill({ skill: 'command-creator' }) | User-facing shortcut needed |
| Repeated logic needs a reusable CLI tool | tool | Skill({ skill: 'tool-creator' }) | CLI utility needed |
| Narrow/single-artifact capability only | inline | Document within this artifact only | Too specific to generalize |
Iron Laws
1. ALWAYS run the Exa + arXiv research gate before updating any eval harness — updating without current external knowledge produces stale evaluation criteria. 2. NEVER remove existing evaluation criteria without replacing them with equivalent or better ones — reducing test coverage in an eval harness is a regression. 3. ALWAYS cross-reference the creator ecosystem for gaps before declaring the harness complete — missing companion artifacts (skills, agents, schemas) leave the harness unable to test new capabilities. 4. NEVER update eval harness in isolation from the skill/agent it evaluates — harness and artifact must stay synchronized or the harness tests the wrong behavior. 5. ALWAYS preserve backward compatibility in eval scoring — changing scoring semantics without migrating historical baselines makes trend analysis impossible.
Anti-Patterns
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Updating eval harness without research gate | Criteria based on outdated knowledge; misses recent evaluation methodology advances | Always run Exa + arXiv research before updating any eval criteria |
| Removing test cases to simplify the harness | Silently reduces coverage; regressions pass undetected | Only remove test cases when the behavior they tested has been deliberately removed |
| Harness and artifact in separate PRs | Harness tests wrong behavior the moment artifact changes; immediate test drift | Always update harness and artifact in the same commit |
| Changing scoring scale mid-project | Historical baselines become incomparable; trend analysis breaks | Define scoring scale once; create a migration if it must change |
| Declaring harness complete without companion check | Missing skills or schemas leave evaluation gaps | Always run companion artifact check before marking harness update complete |
Memory Protocol (MANDATORY)
Before starting: Read .claude/context/memory/learnings.md
After completing:
- New evaluation pattern →
.claude/context/memory/learnings.md - Evaluation gap found →
.claude/context/memory/issues.md - Scoring decision made →
.claude/context/memory/decisions.md
ASSUME INTERRUPTION: If it's not in memory, it didn't happen.
Invoke the eval-harness-updater skill and follow it exactly as presented to you
'use strict';
/**
* Post-execute hook for eval-harness-updater
* Auto-generated by enterprise-bundle-scaffolder
*
* Records metrics after skill execution.
*/
function postExecute(_context) {
// Record execution metrics
return { ok: true, skill: 'eval-harness-updater' };
}
module.exports = { postExecute };
'use strict';
/**
* Pre-execute hook for eval-harness-updater
* Auto-generated by enterprise-bundle-scaffolder
*
* Validates inputs before skill execution.
*/
function preExecute(context) {
// Validate skill invocation context
if (!context || typeof context !== 'object') {
return { allow: true, message: 'eval-harness-updater: no context to validate' };
}
return { allow: true };
}
module.exports = { preExecute };
eval-harness-updater Research Requirements
Generated: 2026-02-28
Skill Description
Refresh evaluation harnesses with live/fallback parser reliability, SLO gates, and regression checks.
Research Areas
- Current best practices for eval-harness-updater
- Industry standards and tooling
- Integration patterns
Source References
- To be populated by skill-updater research phase
eval-harness-updater Rules
Purpose
Refresh evaluation harnesses with live/fallback parser reliability, SLO gates, and regression checks.
Best Practices
- Follow established patterns
- Validate inputs at boundaries
Integration Points
See SKILL.md for complete documentation.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "eval-harness-updaterInput",
"description": "Input schema for Refresh evaluation harnesses with live/fallback parser reliability, SLO gates, and regression checks.",
"type": "object",
"additionalProperties": true,
"properties": {
"target": {
"type": "string",
"description": "Target file or path for the skill to operate on"
},
"options": {
"type": "object",
"description": "Additional options for skill execution",
"additionalProperties": true
}
}
}
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "eval-harness-updaterOutput",
"type": "object",
"additionalProperties": true,
"properties": {
"ok": {
"type": "boolean"
},
"summary": {
"type": "string"
}
}
}
#!/usr/bin/env node
'use strict';
const fs = require('node:fs');
const path = require('node:path');
function findProjectRoot() {
let dir = __dirname;
while (dir !== path.parse(dir).root) {
if (fs.existsSync(path.join(dir, '.claude'))) return dir;
if (path.basename(dir) === '.claude') return path.dirname(dir);
dir = path.dirname(dir);
}
return process.cwd();
}
const PROJECT_ROOT = findProjectRoot();
function parseArgs(argv) {
const options = {};
for (let i = 0; i < argv.length; i++) {
const arg = argv[i];
if (!arg.startsWith('--')) continue;
const key = arg.slice(2);
const next = argv[i + 1];
const hasValue = next && !next.startsWith('--');
options[key] = hasValue ? argv[++i] : true;
}
return options;
}
function resolveHarness(raw) {
const input = String(raw || '').trim();
if (!input) return { harness: '', path: '', exists: false };
if (input.includes('/') || input.endsWith('.cjs') || input.endsWith('.mjs')) {
const normalized = input.replace(/\\/g, '/');
return {
harness: path.basename(normalized),
path: normalized,
exists: fs.existsSync(path.join(PROJECT_ROOT, normalized)),
};
}
const candidate = `.claude/context/runtime/evals/${input}`;
return {
harness: input,
path: candidate,
exists: fs.existsSync(path.join(PROJECT_ROOT, candidate)),
};
}
function main(input = null) {
const options = input || parseArgs(process.argv.slice(2));
if (options.help) {
return {
ok: true,
usage:
'node .claude/skills/eval-harness-updater/scripts/main.cjs --harness <path-or-name> [--trigger reflection|evolve|manual]',
};
}
const target = resolveHarness(options.harness || options.name);
const trigger = ['reflection', 'evolve', 'manual'].includes(options.trigger)
? options.trigger
: 'manual';
if (!target.harness) return { ok: false, stage: 'input', error: 'Missing --harness' };
return {
ok: true,
trigger,
target,
checks: [
'parser scans full stream + partial timeout output',
'live mode emits actionable diagnostics',
'fallback mode emits deterministic metrics',
'SLO gates use explicit thresholds',
],
tddBacklog: [
{ phase: 'RED', items: ['add failing parser/timeout/fallback tests'] },
{ phase: 'GREEN', items: ['patch harness parser/prompts minimally'] },
{ phase: 'REFACTOR', items: ['simplify report mode-selection logic'] },
{ phase: 'VERIFY', items: ['run eval tests + validate metrics outputs'] },
],
};
}
if (require.main === module) {
const result = main();
if (result.usage) {
console.log(result.usage);
process.exit(0);
}
console.log(JSON.stringify(result, null, 2));
process.exit(result.ok ? 0 : 1);
}
module.exports = { parseArgs, resolveHarness, main };
eval-harness-updater Implementation Template
Goal
- Define target outcome and acceptance criteria.
TDD
1. Red 2. Green 3. Refactor
Verification
- lint
- format
- targeted tests