Autoany

Name: Autoany
Author: broomva

broomva/autoany

4 installs
2 repo stars
Updated April 25, 2026
broomva/autoany

autoany is a skill implementing the Evaluator-Governed Recursive Improvement (EGRI) framework to turn ambiguous goals into safe, measurable, rollback-capable improvement loops.

About

A framework skill for turning vague optimization goals into safe, measurable, rollback-capable recursive-improvement systems (Evaluator-Governed Recursive Improvement). A developer uses it to formalize a goal into a problem-spec, build an evaluator and harness, define a mutation surface, and run a bounded improvement loop. It defines autonomy modes and safety rules and scaffolds a project via an init script.

EGRI framework turning ambiguous goals into safe recursive-improvement loops
Evaluator-first design with mutable-artifact + immutable-evaluator architecture
Four autonomy modes (suggestion, sandbox, auto-promote, portfolio) with safety rules

Autoany by the numbers

4 all-time installs (skills.sh)
Ranked #13,372 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
Data as of Jul 30, 2026 (Skillselion catalog sync)

At a glance

autoany capabilities & compatibility

Capabilities: orchestration
Use cases: orchestration · research

From the docs

What autoany says it does

Turn ambiguous user goals into safe, measurable, rollback-capable recursive improvement systems.

SKILL.md

The evaluator must exist and produce a baseline score before any mutation begins.

SKILL.md

Default to **sandbox**. Escalate only with explicit user approval.

SKILL.md

npx skills add https://github.com/broomva/autoany --skill autoany

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/broomva/autoany.svg)](https://skillselion.com/skills/broomva/autoany)

Installs	4
repo stars	★ 2
Last updated	April 25, 2026
Repository	broomva/autoany ↗

What it does

Turn an ambiguous optimization goal into a safe, evaluator-governed recursive-improvement loop for any domain.

Who is it for?

Formalizing a vague optimization goal into a bounded, evaluator-governed improvement loop

When should I use this skill?

You want to build a self-improving system or turn 'make X better' into a safe optimization process

What you get

A problem-spec plus evaluator, harness, and promotion policy for a bounded improvement loop

problem-spec.yaml, evaluator, harness, ledger, and a bounded improvement loop

By the numbers

6 operating-procedure phases
4 autonomy modes
6 safety rules

Files

SKILL.mdMarkdownGitHub ↗

Autoany — EGRI Skill

Turn ambiguous user goals into safe, measurable, rollback-capable recursive improvement systems.

Core Principle

Do not grant an agent more mutation freedom than your evaluator can reliably judge.

Operating Procedure

Phase 1: Problem Compilation

Extract from the user's goal:

1. Objective — metric(s) to optimize (scalar or vector) 2. Hard constraints — what must never be violated (memory, latency, cost, compliance) 3. Mutable artifacts — what the loop may change (the train.py equivalent) 4. Immutable artifacts — what stays fixed (the prepare.py equivalent) 5. Evaluator — how to score candidates reliably enough to compare them 6. Execution backend — where candidates run (local, container, simulator, API) 7. Budget — time, tokens, money, or trial count per candidate 8. Promotion policy — keep-if-improves, Pareto, threshold, human-gate 9. Autonomy mode — suggestion, sandbox, auto-promote, or portfolio

Produce a problem-spec.yaml. See assets/problem-spec.template.yaml for the schema and references/PROBLEM-SPEC.md for field-by-field semantics.

Phase 2: Evaluator-First Design

Before touching the mutable artifact:

1. Define the evaluator — what it measures, how it scores, what thresholds matter 2. Build or identify the benchmark / replay set / test suite 3. Establish baseline score by running the current artifact through the evaluator 4. Confirm the evaluator is trusted — if not, fix it before proceeding

Law: The evaluator must exist and produce a baseline score before any mutation begins.

Phase 3: Harness Construction

Build the immutable execution shell:

1. Execution script — runs the candidate artifact deterministically 2. Scoring script — invokes the evaluator, outputs structured results 3. Constraint checker — rejects candidates violating hard constraints 4. Rollback mechanism — restores previous state on failure or rejection 5. Telemetry — logs trial metadata (duration, resource use, errors) 6. Ledger — append-only record of all trials (see assets/ledger.schema.json)

Phase 4: Mutation Surface Definition

1. Identify artifact type (code, config, prompt, graph, parameters) 2. Define mutation operators (edit, replace, compose, parameterize, restructure) 3. Start with the smallest viable mutation surface — expand only after baseline is stable 4. Mark everything else as immutable

Phase 5: Loop Execution

x_t = current best artifact state
while budget remains:
    m = propose_mutation(x_t, ledger, strategy)
    x' = apply(m, x_t)
    result = execute(x', harness)
    score = evaluate(result)
    if violates_constraints(result): discard(x'), log("rejected")
    elif promotion_policy(score, x_t_score): promote(x'), x_t = x'
    else: discard(x'), log("no improvement")
    record(ledger, trial_metadata)

Phase 6: Ledger Review and Strategy Distillation

After each batch of trials:

1. Review ledger for patterns (what helped, what failed, what is exhausted) 2. Induce reusable abstractions ("depth increases hurt under this budget") 3. Update search strategy based on accumulated evidence 4. Decide: continue, branch, simplify, or escalate to human

Autonomy Modes

Mode	Mutate	Execute	Promote	When to use
Suggestion	Propose only	No	No	Evaluator untrusted or high-risk domain
Sandbox	Yes	Yes	No	Evaluator exists but promotion needs human review
Auto-promote	Yes	Yes	Yes	Strong evaluator, bounded damage, clear constraints
Portfolio	Yes	Yes	Yes	Multiple loops, budget allocation across subproblems

Default to sandbox. Escalate only with explicit user approval.

Safety Rules

1. Never mutate evaluator and artifact in the same trial 2. Never promote without constraint checks passing 3. Never exceed budget — fail closed, not open 4. Always maintain rollback capability to last promoted state 5. Log every trial, including failures and rejections 6. If evaluator is suspected gamed, halt and escalate

Domain Adaptation

Read references/DOMAIN-MAPPINGS.md for concrete artifact/harness/evaluator choices per domain.

Formal Model

Read references/REFERENCE.md for full EGRI formal model: Π = (X, M, H, E, J, C, B, P, L).

Nested Loops and Meta-Optimization

Read references/META-LOOP.md for Level 1-3 loops (policy, portfolio, org).

Scaffold Initialization

python3 scripts/autoany_init.py <project-name> --domain <code|rag|workflow|etl|ui|generic> --path <output-dir>

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Autoany Trial Ledger Entry",
  "description": "Schema for a single trial record in an EGRI ledger.",
  "type": "object",
  "required": [
    "trial_id",
    "timestamp",
    "parent_state",
    "mutation",
    "outcome",
    "decision"
  ],
  "properties": {
    "trial_id": {
      "type": "string",
      "description": "Unique identifier for this trial (e.g., UUID or sequential)"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp of trial start"
    },
    "parent_state": {
      "type": "string",
      "description": "Identifier of the artifact state this trial mutated from"
    },
    "mutation": {
      "type": "object",
      "required": ["operator", "description"],
      "properties": {
        "operator": {
          "type": "string",
          "description": "Name of the mutation operator applied"
        },
        "description": {
          "type": "string",
          "description": "Human-readable description of what was changed"
        },
        "diff": {
          "type": "string",
          "description": "Patch or diff of the change (optional but recommended)"
        },
        "hypothesis": {
          "type": "string",
          "description": "Why this mutation was proposed — the reasoning"
        }
      }
    },
    "execution": {
      "type": "object",
      "properties": {
        "duration_s": {
          "type": "number",
          "description": "Wall-clock seconds for execution"
        },
        "resource_usage": {
          "type": "object",
          "description": "Memory, CPU, GPU, tokens, cost consumed",
          "additionalProperties": true
        },
        "exit_code": {
          "type": "integer"
        },
        "error": {
          "type": ["string", "null"],
          "description": "Error message if execution failed"
        }
      }
    },
    "outcome": {
      "type": "object",
      "required": ["score", "constraints_passed"],
      "properties": {
        "score": {
          "description": "Evaluator output — scalar number or object for vector metrics",
          "oneOf": [
            { "type": "number" },
            { "type": "object", "additionalProperties": { "type": "number" } }
          ]
        },
        "constraints_passed": {
          "type": "boolean",
          "description": "Whether all hard constraints were satisfied"
        },
        "constraint_violations": {
          "type": "array",
          "items": { "type": "string" },
          "description": "List of violated constraints (empty if all passed)"
        },
        "evaluator_metadata": {
          "type": "object",
          "description": "Additional evaluator output (confusion matrix, per-sample scores, etc.)",
          "additionalProperties": true
        }
      }
    },
    "decision": {
      "type": "object",
      "required": ["action"],
      "properties": {
        "action": {
          "type": "string",
          "enum": ["promoted", "discarded", "branched", "escalated"],
          "description": "What happened to this candidate"
        },
        "reason": {
          "type": "string",
          "description": "Why this decision was made"
        },
        "new_state_id": {
          "type": ["string", "null"],
          "description": "Identifier of the new promoted state (null if discarded)"
        }
      }
    },
    "strategy_notes": {
      "type": ["string", "null"],
      "description": "Post-trial observations for strategy distillation"
    }
  }
}

# Autoany Problem Spec — EGRI Instance Definition
# Fill this template to define a recursive improvement loop.
# See references/PROBLEM-SPEC.md for field-by-field semantics.

name: ""  # Human-readable problem name

objective:
  metric: ""           # e.g., "minimize validation_loss", "maximize throughput"
  type: scalar         # scalar | vector
  direction: minimize  # minimize | maximize
  baseline: null       # Filled after Phase 2 (evaluator-first)

constraints:
  - ""  # e.g., "memory_mb <= 48000"
  - ""  # e.g., "runtime_s <= 300"

artifacts:
  mutable:
    - path: ""
      type: code       # code | config | prompt | graph | parameters
      description: ""
  immutable:
    - path: ""
      reason: ""       # e.g., "evaluator — must not change during trials"

evaluator:
  script: ""           # Path to evaluation script or command
  inputs: []           # What the evaluator reads
  outputs: {}          # Structured score format
  trusted: false       # Set true only after validating against known outcomes
  baseline_score: null # Filled after running evaluator on initial artifact

execution:
  backend: local       # local | container | simulator | api | lab
  command: ""          # Execution command template
  timeout_s: 300       # Max seconds per trial
  sandbox: true        # Whether execution is isolated

budget:
  max_trials: 50
  time_per_trial_s: 300
  total_time_s: null   # Overall time cap (null = no cap beyond per-trial)
  token_budget: null   # LLM token cap for the loop
  cost_budget: null    # Monetary cap

promotion:
  policy: keep_if_improves  # keep_if_improves | pareto | threshold | human_gate
  threshold: null            # Minimum improvement to promote (null = any improvement)
  require_constraint_check: true  # Always true, cannot be overridden

autonomy:
  mode: sandbox        # suggestion | sandbox | auto-promote | portfolio
  escalation_triggers:
    - "evaluator_score_degrades_3_consecutive_trials"
    - "constraint_violation_detected"
    - "budget_75_percent_exhausted_without_improvement"

# Optional fields

search:
  proposer: llm        # llm | random | bayesian | evolutionary | hybrid
  strategy_notes: ""   # Free-text guidance for the mutation proposer

ledger:
  format: jsonl        # jsonl | sqlite | tsv
  path: "./ledger.jsonl"
  schema: "./ledger.schema.json"

domain:
  preset: generic      # code | rag | workflow | etl | ui | generic
  notes: ""

meta:
  created_by: ""
  created_at: ""       # ISO timestamp
  version: "0.1.0"
  parent_spec: null    # Path to parent spec if this is a refinement

Domain Mappings

Concrete EGRI instantiations for common domains.

Code Optimization (ML Training)

The original autoresearch pattern.

Component	Concrete form
Mutable artifact	`train.py` — model architecture, optimizer, hyperparameters, training loop
Immutable harness	`prepare.py` — data prep, tokenizer, dataset splits
Evaluator	`val_bpb` (validation bits-per-byte)
Constraints	VRAM <= budget, runtime <= 5 min, no external deps
Budget	Fixed time per trial, fixed trial count
Promotion	Keep if val_bpb improves without constraint violation
Ledger	`results.tsv` + git commit history
Execution	Local GPU

Code Optimization (General)

Component	Concrete form
Mutable artifact	Source files, compiler flags, build configs
Immutable harness	Test suite, benchmark suite, CI pipeline
Evaluator	Test pass rate, benchmark throughput, binary size, compile time
Constraints	All tests pass, no regressions on key benchmarks
Budget	N trials, M minutes per trial
Promotion	Keep if primary metric improves and all tests pass
Ledger	JSONL with trial ID, diff hash, scores, duration
Execution	Local or container

RAG Pipeline

Component	Concrete form
Mutable artifact	Retrieval config, chunking strategy, prompts, reranker settings, embedding model choice
Immutable harness	Document corpus, golden eval set, judge config
Evaluator	Answer accuracy (judge-scored), retrieval recall@k, latency, cost per query
Constraints	Latency < threshold, cost < budget, no hallucinated sources
Budget	N eval runs, token budget
Promotion	Keep if accuracy improves without latency/cost regression
Ledger	JSONL with query ID, retrieved docs, answer, judge score
Execution	API calls or local inference

Workflow / Operations

Component	Concrete form
Mutable artifact	Decision graph, routing policy, prompts, retry logic, escalation thresholds
Immutable harness	Replay log corpus, simulation environment, sandbox
Evaluator	Completion rate, error rate, latency, compliance score, cost
Constraints	No compliance violations, no data leakage, rollback on failure
Budget	N replayed cases, wall-clock time
Promotion	Keep if completion rate improves without compliance regression
Ledger	JSONL with case ID, decision path, outcome, scores
Execution	Replay executor or sandboxed live execution

ETL Pipeline

Component	Concrete form
Mutable artifact	Transform logic, schema mappings, dedup rules, validation rules
Immutable harness	Source data snapshot, expected output snapshot, data quality checks
Evaluator	Row accuracy, schema conformance, throughput, error rate
Constraints	Zero data loss, schema must match target, idempotent
Budget	N pipeline runs on test data
Promotion	Keep if accuracy improves and zero data loss maintained
Ledger	JSONL with run ID, row counts, error counts, duration
Execution	Local or container with test data

UI / Product Optimization

Component	Concrete form
Mutable artifact	Copy, layout, flow structure, component config, recommendation logic
Immutable harness	A/B testing platform, user simulator, screenshot comparison
Evaluator	Conversion rate, task completion time, error rate, accessibility score
Constraints	WCAG compliance, no broken flows, performance budget
Budget	N simulated sessions or A/B test duration
Promotion	Keep if primary metric improves; human gate recommended
Ledger	JSONL with variant ID, session data, scores
Execution	Browser automation or user simulator

Note: UI optimization is noisier than other domains because humans are in the loop and rewards are delayed. Default to suggestion or sandbox autonomy mode.

Compiler Optimization

Component	Concrete form
Mutable artifact	Pass ordering, inlining heuristics, scheduling, codegen flags
Immutable harness	Compiler + benchmark suite (SPEC, Embench, custom)
Evaluator	Runtime, code size, compile time, energy consumption
Constraints	All benchmarks must compile and pass correctness checks
Budget	N compilation + benchmark cycles
Promotion	Pareto over runtime and code size
Ledger	JSONL with pass config hash, benchmark scores, compile time
Execution	Local or CI

Prompt Engineering

Component	Concrete form
Mutable artifact	System prompt, few-shot examples, output format instructions
Immutable harness	Eval dataset, judge prompt/model, scoring rubric
Evaluator	Judge accuracy, format compliance, latency, token cost
Constraints	Token budget per call, no prohibited content patterns
Budget	N eval runs, token/cost budget
Promotion	Keep if judge score improves without cost regression
Ledger	JSONL with prompt version, eval scores, token counts
Execution	API calls

Nested Loops and Meta-Optimization

EGRI supports recursive application at multiple levels.

Loop Levels

Level 0: Artifact Loop

Optimize the artifact itself. This is the base autoresearch behavior.

mutate artifact → execute → evaluate → promote/discard → record → repeat

LLM role: Hypothesis generation, diagnosis, tradeoff judgment.

Level 1: Policy Loop

Optimize how mutations are proposed. The mutation strategy itself becomes the mutable artifact.

mutate search_policy → run N artifact trials → evaluate policy effectiveness → promote/discard policy → record → repeat

Mutable: Search heuristics, decomposition policies, branching strategy, stopping criteria. Evaluator: Rate of improvement per trial, cost per improvement, diversity of solutions found. LLM role: Theory formation from search history, identifying exhausted branches.

Example abstractions to induce:

"Depth increases hurt under this time budget"
"Attention pattern changes are high-risk/high-reward"
"Optimizer changes only help when batch regime changes too"
"This branch of search is exhausted"

Level 2: Portfolio Loop

Allocate budget across multiple Level 0/1 loops running in parallel.

observe all active loops → reallocate budget → spawn/prune loops → record → repeat

Mutable: Budget allocation, loop priorities, spawn/prune decisions. Evaluator: Portfolio-level progress rate, resource efficiency, coverage. LLM role: Strategic resource allocation, identifying complementary vs redundant loops.

Level 3: Org Loop

Optimize the organization code: who explores what, when to branch, when to simplify, when to exploit vs explore.

observe portfolio performance → modify coordination policy → evaluate org effectiveness → record → repeat

Mutable: Coordination rules, escalation thresholds, team composition, communication protocols. Evaluator: Overall research velocity, discovery rate, resource utilization. LLM role: Meta-optimization — improving the rules that govern the improvement process.

This is what Karpathy means by iterating on program.md and building an "autonomous research org."

When to Use Each Level

Level	Prerequisite	Trigger
0	Evaluator exists, mutable artifact defined	Default starting point
1	Level 0 has run enough trials to show patterns	Improvement rate plateaus
2	Multiple valid subproblems or approaches exist	Single loop is insufficient
3	Portfolio is running but coordination is suboptimal	Budget is wasted on redundant work

Strategy Distillation

After sufficient Level 0 trials, distill the ledger into explicit learned strategy:

1. Cluster trials by mutation type and outcome 2. Identify winning patterns — what kinds of mutations reliably help? 3. Identify dead ends — what kinds of mutations consistently fail? 4. Form hypotheses — why do the patterns hold? 5. Update search policy — bias future proposals toward winning patterns 6. Record distilled strategy in the ledger as a special entry

This is not just memory. It is theory formation from search history.

The Three-Layer Architecture

For production systems, separate concerns:

autoany-skill     → compiler: interprets user intent, produces problem-spec
autoany-core      → microkernel: loop orchestration, ledger, executor abstraction
problem-instance  → generated: actual evaluator, harness, artifact space, operators

Skill = compiler. Decides what kind of system to build. Core = microkernel. Provides the reusable loop substrate. Instance = generated runtime. Contains the domain-specific implementation.

Where LLM Reasoning Is Most Valuable

High-entropy decisions (use LLM):

Choosing representations
Identifying causal hypotheses
Designing evaluators
Translating vague goals into formal specs
Clustering failures into categories
Deciding when to branch vs exploit
Discovering reusable modules across domains

Low-entropy decisions (use scripts):

Brute-force parameter sweeps
Rerunning the same command
Parsing fixed-format metrics
Maintaining append-only logs
Simple keep/discard comparisons once evaluator is trusted

Problem Spec Field Semantics

Each EGRI problem instance is defined in a problem-spec.yaml file.

Required Fields

`name`

Human-readable identifier for this problem instance.

`objective`

metric: String — what to optimize (e.g., "minimize validation_loss", "maximize throughput")
type: scalar | vector — single metric or multi-objective
direction: minimize | maximize — optimization direction per metric
baseline: Number or null — current score before any mutations (filled after Phase 2)

`constraints`

List of hard predicates that must hold for every candidate:

constraints:
  - "memory_mb <= 48000"
  - "runtime_s <= 300"
  - "no_external_network_calls"
  - "output_format == 'json'"

Violations cause immediate rejection. Not tradeoffs — hard boundaries.

`artifacts`

`mutable`

List of files, configs, prompts, or artifact identifiers the loop may modify. Start with the smallest viable set. Each entry should include:

path: File path or identifier
type: code | config | prompt | graph | parameters
description: What this artifact does and why it is mutable

`immutable`

List of artifacts that must NOT be modified during the loop:

path: File path or identifier
reason: Why this must stay fixed (evaluator, data prep, benchmark, etc.)

`evaluator`

script: Path to the evaluation script or command
inputs: What the evaluator reads (output files, metrics, logs)
outputs: Structured score format (JSON with metric fields)
trusted: true | false — has the evaluator been validated against known outcomes?
baseline_score: Filled after running evaluator on the initial artifact

`execution`

backend: local | container | simulator | api | lab
command: The execution command template
timeout_s: Maximum seconds per trial
sandbox: true | false — whether execution is isolated

`budget`

max_trials: Integer — maximum number of mutation attempts
time_per_trial_s: Integer — max seconds per trial execution
total_time_s: Integer or null — overall time cap
token_budget: Integer or null — LLM token cap for the loop
cost_budget: Float or null — monetary cap

`promotion`

policy: keep_if_improves | pareto | threshold | human_gate
threshold: Number or null — minimum improvement to promote
require_constraint_check: true (always true, cannot be overridden)

`autonomy`

mode: suggestion | sandbox | auto-promote | portfolio
escalation_triggers: List of conditions that force human review

Optional Fields

`search`

proposer: llm | random | bayesian | evolutionary | hybrid
strategy_notes: Free-text guidance for the mutation proposer

`ledger`

format: jsonl | sqlite | tsv
path: Where to store the ledger
schema: Path to ledger schema (default: ledger.schema.json)

`domain`

preset: code | rag | workflow | etl | ui | generic
notes: Domain-specific context for the mutation proposer

`meta`

created_by: Who/what created this spec
created_at: ISO timestamp
version: Spec version (current: 0.1.0)
parent_spec: Path to parent spec if this is a refinement

EGRI Formal Model

Problem Instance

A problem instance is a tuple:

Π = (X, M, H, E, J, C, B, P, L)

Symbol	Name	Definition
X	Artifact state space	Set of valid artifact states
M	Mutation operators	Proposal functions over X
H	Immutable harness	Fixed execution shell specification
E	Execution backend	Where candidates run (local, container, simulator, lab)
J	Evaluator	Returns scalar or vector score
C	Hard constraints	Safety predicates that must hold
B	Budget policy	Time, money, tokens, or trial count
P	Promotion policy	Decision rule: keep, discard, branch, escalate
L	Ledger	Append-only record of trajectories, scores, lineage, failures

Canonical Loop

Given current state x_t:

1. Propose candidate set Q_t = {m_i(x_t)} for m_i ∈ M 2. Execute each q ∈ Q_t inside (H, E, B) 3. Observe outcomes o(q) 4. Compute score s(q) = J(o(q)) 5. Reject any q that violates C 6. Choose next state x_{t+1} = P(x_t, Q_t, s, L) 7. Append all outcomes to L

This is abstract enough to cover: hill climbing, Bayesian optimization, beam search, evolutionary search, bandits, PBT, planner-executor loops, and multi-agent portfolio search.

Core Laws

Law 1: Evaluator Supremacy

An optimization loop is only as safe and useful as the evaluator that governs it.

Law 2: Mutation-Evaluation Proportionality

Do not grant an agent more mutation freedom than your evaluator can reliably judge.

Law 3: Immutability of the Evaluator

The evaluator and the mutable artifact must never be changed in the same trial. If both need to change, that is a new problem instance.

Law 4: Budget Closure

The loop must fail closed when budget is exhausted. No "one more try" exceptions.

Law 5: Rollback Guarantee

Every promoted state must be recoverable. The system must be able to return to the last known-good state at any point.

Minimal Formal Conditions

EGRI works best when four conditions hold:

1. Mutable artifact exists — something concrete can be changed 2. Executable harness exists — candidates can be run repeatably 3. Trusted evaluator exists — outcomes can be scored reliably enough to compare 4. Bounded damage — bad candidates can be rejected, rolled back, sandboxed

Failure Modes

Failure	Symptom	Remedy
Evaluator too noisy	Promoted states oscillate	Increase eval samples, use paired comparisons
Evaluator gameable	Score improves but real quality degrades	Add holdout set, adversarial checks
Mutation surface too large	Search is diffuse, no signal	Shrink surface, decompose into sub-problems
Budget too tight	Loop halts before finding signal	Reduce mutation cost or expand budget
No rollback	Failed promotion corrupts state	Add versioning before any mutation
Reward hacking	Agent optimizes proxy, not intent	Add constraint predicates, human review gates

Abstraction Levels

The primitive supports three progressively stronger versions:

Version A — Optimize existing artifact: Tune a training loop, retrieval config, compiler pass
Version B — Generate then optimize: Synthesize baseline + iterate
Version C — Synthesize artifact class: Invent new topology, replace rules engine with learned controller

The hard part is not C. The hard part is making the evaluator strong enough that C does not devolve into garbage.

#!/usr/bin/env python3
"""
Autoany scaffold initializer.

Bootstraps a minimal EGRI project directory for a new problem instance.

Usage:
    python3 autoany_init.py <name> --domain <preset> --path <dir>
    python3 autoany_init.py --help
"""

import argparse
import json
import os
import sys
from datetime import datetime, timezone

DOMAIN_PRESETS = {
    "code": {
        "objective": {
            "metric": "minimize test_failure_rate",
            "type": "scalar",
            "direction": "minimize",
        },
        "mutable_artifact": {
            "path": "src/",
            "type": "code",
            "description": "Source code under optimization",
        },
        "immutable_artifact": {
            "path": "tests/",
            "reason": "Test suite — evaluator must not change during trials",
        },
        "evaluator_script": "eval/run_eval.sh",
        "execution_backend": "local",
        "timeout_s": 300,
        "proposer": "llm",
    },
    "rag": {
        "objective": {
            "metric": "maximize answer_accuracy",
            "type": "scalar",
            "direction": "maximize",
        },
        "mutable_artifact": {
            "path": "config/retrieval.yaml",
            "type": "config",
            "description": "Retrieval config, chunking, prompts, reranker",
        },
        "immutable_artifact": {
            "path": "eval/golden_set.jsonl",
            "reason": "Golden eval set — ground truth for scoring",
        },
        "evaluator_script": "eval/judge.py",
        "execution_backend": "api",
        "timeout_s": 120,
        "proposer": "llm",
    },
    "workflow": {
        "objective": {
            "metric": "maximize completion_rate",
            "type": "scalar",
            "direction": "maximize",
        },
        "mutable_artifact": {
            "path": "workflow/policy.yaml",
            "type": "graph",
            "description": "Decision graph, routing, escalation thresholds",
        },
        "immutable_artifact": {
            "path": "eval/replay_logs/",
            "reason": "Replay corpus — fixed test cases",
        },
        "evaluator_script": "eval/replay_eval.py",
        "execution_backend": "local",
        "timeout_s": 60,
        "proposer": "llm",
    },
    "etl": {
        "objective": {
            "metric": "maximize row_accuracy",
            "type": "scalar",
            "direction": "maximize",
        },
        "mutable_artifact": {
            "path": "transforms/",
            "type": "code",
            "description": "Transform logic, schema mappings, dedup rules",
        },
        "immutable_artifact": {
            "path": "fixtures/",
            "reason": "Source and expected output snapshots",
        },
        "evaluator_script": "eval/data_quality.py",
        "execution_backend": "container",
        "timeout_s": 180,
        "proposer": "llm",
    },
    "ui": {
        "objective": {
            "metric": "maximize task_completion_rate",
            "type": "scalar",
            "direction": "maximize",
        },
        "mutable_artifact": {
            "path": "src/components/",
            "type": "code",
            "description": "UI components, copy, layout, flow",
        },
        "immutable_artifact": {
            "path": "eval/scenarios/",
            "reason": "User simulation scenarios",
        },
        "evaluator_script": "eval/ui_eval.py",
        "execution_backend": "local",
        "timeout_s": 60,
        "proposer": "llm",
    },
    "generic": {
        "objective": {"metric": "", "type": "scalar", "direction": "minimize"},
        "mutable_artifact": {"path": "", "type": "code", "description": ""},
        "immutable_artifact": {"path": "", "reason": ""},
        "evaluator_script": "eval/run_eval.sh",
        "execution_backend": "local",
        "timeout_s": 300,
        "proposer": "llm",
    },
}


def generate_problem_spec(name: str, domain: str) -> str:
    preset = DOMAIN_PRESETS[domain]
    obj = preset["objective"]
    mut = preset["mutable_artifact"]
    imm = preset["immutable_artifact"]
    now = datetime.now(timezone.utc).isoformat()

    return f"""# Autoany Problem Spec — {name}
# Domain preset: {domain}
# Generated: {now}

name: "{name}"

objective:
  metric: "{obj["metric"]}"
  type: {obj["type"]}
  direction: {obj["direction"]}
  baseline: null

constraints:
  - "runtime_s <= {preset["timeout_s"]}"

artifacts:
  mutable:
    - path: "{mut["path"]}"
      type: {mut["type"]}
      description: "{mut["description"]}"
  immutable:
    - path: "{imm["path"]}"
      reason: "{imm["reason"]}"

evaluator:
  script: "{preset["evaluator_script"]}"
  inputs: []
  outputs: {{}}
  trusted: false
  baseline_score: null

execution:
  backend: {preset["execution_backend"]}
  command: ""
  timeout_s: {preset["timeout_s"]}
  sandbox: true

budget:
  max_trials: 50
  time_per_trial_s: {preset["timeout_s"]}
  total_time_s: null
  token_budget: null
  cost_budget: null

promotion:
  policy: keep_if_improves
  threshold: null
  require_constraint_check: true

autonomy:
  mode: sandbox
  escalation_triggers:
    - "evaluator_score_degrades_3_consecutive_trials"
    - "constraint_violation_detected"
    - "budget_75_percent_exhausted_without_improvement"

search:
  proposer: {preset["proposer"]}
  strategy_notes: ""

ledger:
  format: jsonl
  path: "./ledger.jsonl"

domain:
  preset: {domain}
  notes: ""

meta:
  created_by: "autoany_init"
  created_at: "{now}"
  version: "0.1.0"
  parent_spec: null
"""


def generate_eval_script() -> str:
    return """#!/usr/bin/env bash
# Autoany evaluator stub
# Replace this with your actual evaluation logic.
# Must output JSON to stdout with at least a "score" field.
set -euo pipefail

echo '{"score": 0.0, "constraints_passed": true, "constraint_violations": []}'
"""


def generate_readme(name: str) -> str:
    return f"""# {name}

An Autoany (EGRI) project — evaluator-governed recursive improvement.

## Structure

```
{name}/
├── problem-spec.yaml    # Problem definition
├── eval/                # Evaluator (immutable during trials)
│   └── run_eval.sh      # Evaluation script
├── artifacts/           # Mutable artifacts
├── ledger.jsonl         # Trial ledger (append-only)
└── harness/             # Execution harness
```

## Quick Start

1. Edit `problem-spec.yaml` to define your problem
2. Implement the evaluator in `eval/`
3. Place your baseline artifact in `artifacts/`
4. Run the evaluator on the baseline to get `baseline_score`
5. Begin the EGRI loop

## Core Law

> Do not grant an agent more mutation freedom than your evaluator can reliably judge.
"""


def main():
    parser = argparse.ArgumentParser(
        description="Bootstrap an Autoany (EGRI) project scaffold."
    )
    parser.add_argument("name", help="Project name")
    parser.add_argument(
        "--domain",
        choices=list(DOMAIN_PRESETS.keys()),
        default="generic",
        help="Domain preset (default: generic)",
    )
    parser.add_argument(
        "--path",
        default=".",
        help="Output directory (default: current directory)",
    )
    args = parser.parse_args()

    project_dir = os.path.join(args.path, args.name)

    if os.path.exists(project_dir):
        print(f"Error: {project_dir} already exists.", file=sys.stderr)
        sys.exit(1)

    # Create directories
    dirs = [
        project_dir,
        os.path.join(project_dir, "eval"),
        os.path.join(project_dir, "artifacts"),
        os.path.join(project_dir, "harness"),
    ]
    for d in dirs:
        os.makedirs(d, exist_ok=True)

    # Write problem-spec.yaml
    spec_path = os.path.join(project_dir, "problem-spec.yaml")
    with open(spec_path, "w") as f:
        f.write(generate_problem_spec(args.name, args.domain))

    # Write evaluator stub
    eval_path = os.path.join(project_dir, "eval", "run_eval.sh")
    with open(eval_path, "w") as f:
        f.write(generate_eval_script())
    os.chmod(eval_path, 0o755)

    # Write ledger schema
    schema_src = os.path.join(
        os.path.dirname(os.path.dirname(__file__)), "assets", "ledger.schema.json"
    )
    schema_dst = os.path.join(project_dir, "ledger.schema.json")
    if os.path.exists(schema_src):
        import shutil

        shutil.copy2(schema_src, schema_dst)
    else:
        # Inline minimal schema
        with open(schema_dst, "w") as f:
            json.dump(
                {
                    "$schema": "https://json-schema.org/draft/2020-12/schema",
                    "title": "Autoany Trial Ledger Entry",
                    "type": "object",
                },
                f,
                indent=2,
            )

    # Write README
    readme_path = os.path.join(project_dir, "README.md")
    with open(readme_path, "w") as f:
        f.write(generate_readme(args.name))

    # Write empty ledger
    ledger_path = os.path.join(project_dir, "ledger.jsonl")
    with open(ledger_path, "w") as f:
        pass  # Empty file

    # Output structured result
    result = {
        "status": "success",
        "project": args.name,
        "domain": args.domain,
        "path": os.path.abspath(project_dir),
        "files_created": [
            spec_path,
            eval_path,
            schema_dst,
            readme_path,
            ledger_path,
        ],
        "next_steps": [
            "Edit problem-spec.yaml to define your problem",
            "Implement the evaluator in eval/",
            "Place baseline artifact in artifacts/",
            "Run evaluator on baseline to get baseline_score",
            "Begin the EGRI loop",
        ],
    }
    print(json.dumps(result, indent=2))


if __name__ == "__main__":
    main()

Related skills

Setup Matt Pocock SkillsScaffold the per-repo configuration that Matt Pocock’s engineering agent skills rely on so they understand the issue tracker, triage labels, and domain documentation la495k196k

Lark AppsConnect Claude, Cursor or custom agents directly to Lark (Feishu) for messaging, document automation, approval workflows and enterprise data access.402k

CavemanSlash token usage by roughly 75% while keeping every technical detail intact when working with Claude Code, Cursor or similar agents.391k94.7k

Lark Skill MakerQuickly turn any Lark/Feishu OpenAPI call or multi-step workflow into a reusable agent skill with its own SKILL.md.386k16k

Running Claude Code Via Litellm CopilotRun Claude Code at a fraction of the cost by routing requests through LiteLLM to the GitHub Copilot Chat API.270k72

Codex PetGenerate a complete Codex Pet spritesheet and metadata from one reference image without needing an OpenAI key or Codex Pro.246k10

FAQ

What is the default autonomy mode?

Sandbox: mutate and execute candidates but require human review before promotion; escalate to auto-promote only with explicit approval.

What must exist before any mutation?

The evaluator must exist and produce a baseline score before any mutation begins.

AI & Agent Buildingagentsautomationresearch

About

Autoany by the numbers

autoany capabilities & compatibility

What autoany says it does

Add your badge

What it does

Who is it for?

When should I use this skill?

What you get

By the numbers

Files

Autoany — EGRI Skill

Core Principle

Operating Procedure

Phase 1: Problem Compilation

Phase 2: Evaluator-First Design

Phase 3: Harness Construction

Phase 4: Mutation Surface Definition

Phase 5: Loop Execution

Phase 6: Ledger Review and Strategy Distillation

Autonomy Modes

Safety Rules

Domain Adaptation

Formal Model

Nested Loops and Meta-Optimization

Scaffold Initialization

Domain Mappings

Code Optimization (ML Training)

Code Optimization (General)

RAG Pipeline

Workflow / Operations

ETL Pipeline

UI / Product Optimization

Compiler Optimization

Prompt Engineering

Nested Loops and Meta-Optimization

Loop Levels

Level 0: Artifact Loop

Level 1: Policy Loop

Level 2: Portfolio Loop

Level 3: Org Loop

When to Use Each Level

Strategy Distillation

The Three-Layer Architecture

Where LLM Reasoning Is Most Valuable

Problem Spec Field Semantics

Required Fields

name

objective

constraints

artifacts

mutable

immutable

evaluator

execution

budget

promotion

autonomy

Optional Fields

search

ledger

domain

meta

EGRI Formal Model

Problem Instance

Canonical Loop

Core Laws

Law 1: Evaluator Supremacy

Law 2: Mutation-Evaluation Proportionality

Law 3: Immutability of the Evaluator

Law 4: Budget Closure

Law 5: Rollback Guarantee

Minimal Formal Conditions

Failure Modes

Abstraction Levels

Related skills

FAQ

What is the default autonomy mode?

What must exist before any mutation?

This week in AI coding

`name`

`objective`

`constraints`

`artifacts`

`mutable`

`immutable`

`evaluator`

`execution`

`budget`

`promotion`

`autonomy`

`search`

`ledger`

`domain`

`meta`