
Reins Quest
- 3 installs
- 1 repo stars
- Updated July 7, 2026
- park-jun-woo/reins
Builds a Go quest CLI on the reins framework where a deterministic machine gate, not the LLM, decides when generation is done in a generate-verify-retry loop.
About
Guides building a quest CLI in Go with the reins framework, moving the authority to declare done from the AI to a deterministic machine gate via gate.Definition rules. A developer uses it when wiring an unattended generate-gate-retry loop whose output is judged by a machine.
- Plug in one gate.Definition; reins supplies ratchet, skeleton, aggregation and export
- Deterministic gate is the sole PASS authority; LLM never judges its own completion
Reins Quest by the numbers
- 3 all-time installs (skills.sh)
- Ranked #13,677 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 4, 2026 (Skillselion catalog sync)
npx skills add https://github.com/park-jun-woo/reins --skill reins-questAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 3 |
|---|---|
| repo stars | ★ 1 |
| Last updated | July 7, 2026 |
| Repository | park-jun-woo/reins ↗ |
What it does
Builds a Go quest CLI on the reins framework where a deterministic machine gate, not the LLM, decides when generation is done in a generate-verify-retry loop.
Files
reins-quest — Build a Quest CLI Whose "Done" Is Judged by a Machine
reins is a quest-CLI framework (Go). It moves the authority to declare "done" from the AI to a deterministic machine gate. "Generation is probabilistic; verification is deterministic." A consumer plugs in only the domain logic (gate.Definition); reins supplies the ratchet, the command skeleton, aggregation, feedback, and export. Agents are disposable; progress is cumulative and irreversible.
When to Use This Skill
- Building a new quest CLI / reins consumer in Go
- Designing a deterministic completion gate (a set of violation-detecting rules)
- Wiring an unattended generate→gate→retry loop (LLM generates, the gate judges)
- Adding rules, a defeat-graph backend, or network-ground verification to an existing reins quest
Do NOT Use This Skill When
- The "done" check cannot be made machine-deterministic and genuinely needs an agent's tool-using exploration (open-ended coding, "fix the repo"). reins generators are pure L0 — no tools, no Act→Observe. Use a normal agentic loop instead.
- You want the LLM to judge its own completion. reins forbids this by design (only the machine locks PASS). If you need LLM-as-judge, reins is the wrong tool.
Install
reins is a library you import from your own Go module:
go get github.com/park-jun-woo/reins@latestPrerequisites: Go 1.25+. reins pulls github.com/park-jun-woo/toulmin (only when you use the graph backend).
The Mental Model (read first)
- Ratchet — a one-way state machine. Once PASS, it is irreversible; remaining work only decreases.
- Gate — a set of violation-detecting rules. A rule fires (true) when it finds a problem and carries a
Fact. Severity is a level (Fail/Review), never a weight. One decisive Fail ⇒ FAIL. - Authority asymmetry — only the machine locks PASS. L1 machine (deterministic, sole PASS authority) / L2 AI (skeptic, REVIEW only) / L3 human (the remainder).
- Fact feedback — a FAIL is not an opinion but a located, quantified value (
Fact{Where,Expected,Actual}). It turns a sycophantic model toward convergence.
Quickstart: the simplest quest (4 methods + one line of main)
Implement the four gate.Definition methods; reins supplies the rest.
// definition.go
type myDef struct{}
func (myDef) Seed(args []string) ([]*quest.Item, error) {
// input → initial TODO items (one per line/file/record)
items := make([]*quest.Item, len(args))
for i, a := range args {
it := &quest.Item{Key: fmt.Sprintf("item-%d", i), State: quest.TODO}
it.SetPayload(map[string]string{"source": a}) // never write Payload directly
items[i] = it
}
return items, nil
}
func (myDef) Render(s *quest.Session, it *quest.Item) (string, error) {
var p map[string]string
it.DecodePayload(&p)
// the authoring prompt `next` shows; READ-ONLY on s.Meta (next never Saves)
return "From this source, output ONLY a JSON {\"summary\": \"...\"}:\n" + p["source"], nil
}
func (myDef) Prepare(s *quest.Session, it *quest.Item, raw []byte) (gate.Context, *quest.Verdict, error) {
var sub struct{ Summary string `json:"summary"` }
if err := json.Unmarshal(raw, &sub); err != nil {
return gate.Context{}, nil, err // decode failure → caller sees the error
}
var p map[string]string
it.DecodePayload(&p)
// short verdict (3rd return) != nil short-circuits the gate (e.g. OutSkip for untrusted input)
return gate.Context{Item: it, Submission: &sub, Source: p["source"]}, nil, nil
}
func (myDef) Rules() []gate.Rule { return []gate.Rule{summaryPresent, summaryGrounded} }
// main.go — one line wires the whole CLI
func main() {
cli.NewQuestCmd("myquest", myDef{}, cli.Options{Version: "0.1"}).Execute()
}One rule = one violation detector
A rule fires (true, Fact{...}) when it finds a problem; otherwise (false, _).
var summaryGrounded = gate.Rule{
Meta: gate.RuleMeta{ID: "summary-grounded", Level: gate.LevelFail, Desc: "summary tokens exist in source (no hallucination)"},
Check: func(ctx gate.Context) (bool, quest.Fact) {
sub := ctx.Submission.(*struct{ Summary string `json:"summary"` })
if miss := textmatch.MissingTokens(ctx.Source, strings.Fields(sub.Summary)); len(miss) > 0 {
return true, quest.Fact{Where: "summary", Expected: "source substring", Actual: miss[0]}
}
return false, quest.Fact{}
},
}gate.Evaluate(rules, ctx) aggregates fired rules by level: any Fail→FAIL, else any Review→REVIEW, else PASS. Deterministic: same (rules, ctx) → same Verdict.
Commands (auto-generated by NewQuestCmd)
| Command | Purpose |
|---|---|
scan <input> | Seed N quests from input |
next | Show one TODO + its authoring prompt / verification context |
| `submit --key <k> --in <file>\ | -` |
status | Progress tally (PASS/REVIEW/DONE/TODO/SKIPPED/BLOCKED) |
export | Emit terminal results as JSONL (originals preserved, emit-once) |
rules | Print the gate's rule catalog (the auto rulebook — audit what it blocks) |
loop | Opt-in unattended drive (LLM generates → gate judges → retry). Attached only if Options.Loop != nil |
Every submit auto-emits terminal items to --out (default <name>-results.jsonl). Tune with Options{Out, Version, ExtraCommands, Loop}.
Core types
| Type | Shape |
|---|---|
quest.Item | { Key; State; Tries; Payload json.RawMessage; Log; Emitted } — use it.SetPayload(v)/it.DecodePayload(&v), never the field |
quest.State | TODO PASS REVIEW DONE SKIPPED BLOCKED (terminal = all but TODO) |
quest.Verdict | { Outcome; Facts []Fact; Feedback; RootCause } — RootCause names the rule that caused FAIL/REVIEW (agent coaching) |
quest.Fact | { Rule, Where, Expected, Actual string } |
gate.Context | { Item; Submission any; Source string; Grounds map[string]string } |
gate.Level | `LevelFail \ |
| const | quest.MaxTries = 3 — FAIL accrued to MaxTries ⇒ lock to DONE (monotone convergence) |
quest.Apply(it, v, now) applies a verdict to the ratchet (PASS/REVIEW/SKIPPED/BLOCKED lock, FAIL is Tries++).
Verification primitives
textmatch.Normalize(s) // NFC + whitespace fold + Trim (no case-fold — ToLower first if needed)
textmatch.Contains(source, token) // substring after normalization
textmatch.MissingTokens(source, toks) // tokens absent from source — the hallucination block
temporal.Resolve(spec, ref) // structured Spec + ref time.Time → Gregorian ISO (undetermined ⇒ Determined=false)
temporal.ComponentsInAnchor(...) // extract time components from an anchor stringUnattended drive: the loop command (opt-in)
Closes the next→submit cycle in-process: an LLM generates each TODO's payload, the gate judges, FAIL feedback is fed back until PASS or MaxTries. The LLM is only the generator (L0); only the gate locks PASS — there is no API to grant the LLM PASS authority.
cli.NewQuestCmd("myquest", myDef{}, cli.Options{
Version: "0.1",
Loop: &cli.LoopOptions{
DefaultModel: "ollama:gemma4:e4b", // "" ⇒ this default
System: "You are a strict generator.", // global generation system prompt
RuleSystem: map[string]string{ // per-rule coaching keyed by Verdict.RootCause
"summary-grounded": "Use ONLY words present in the source.",
},
// LLM: injected llm.Backend (tests); when set, --model is ignored
},
}).Execute()Run: myquest loop [--model backend:model] [--max-items N]. On the MaxTries-th FAIL the item locks DONE → NextTODO drops it → the loop terminates (monotone convergence).
LLM backends (pkg/llm)
| Token | Transport | Auth |
|---|---|---|
ollama:<model> | HTTP (local, num_ctx auto-sized) | none |
xai:<model> / gemini:<model> | HTTP (OpenAI-compat / Gemini) | env-only API key |
claude:<model> | subprocess claude -p (--max-turns 1 --tools "") | CLI login — no API key read |
grok:<model> | subprocess grok -p (single-turn) | CLI login — no API key read |
codex:<model> | subprocess codex exec (-s read-only) | CLI login — no API key read |
geminicli:<model> | subprocess gemini -p (--approval-mode plan) | Google login — no API key read (separate token; gemini: is the HTTP backend) |
- HTTP options (struct fields, zero ⇒ prior default = backward-compatible): all three HTTP backends take
MaxOutputTokens int(0 ⇒ 2048) andTemperature *float64(nil ⇒ 0); ollama also takesThink *bool. Or pass them in the--modelquery:ollama:qwen3:8b?max_output_tokens=8192&think=false. Raisemax_output_tokensso reasoning models aren't truncated (ollama growsnum_ctxto match). An unknown key for a backend is a loud error (allowed: ollamamax_output_tokens/num_ctx/temperature/think, xai & geminimax_output_tokens/temperature, subprocess none). - Subprocess backends: token
:<model>or:default(CLI's configured model).REINS_<NAME>_BINoverrides the binary. - Session is fully stateless by default (matches HTTP backends + reins' deterministic FAIL-feedback convergence). Opt into carrying the CLI's own conversation with
REINS_<NAME>_SESSION=continue(stateless recommended — session mode double-exposes the prior attempt). - Inject
llm.CallFunc(HTTP) or theexec<Name>package-var seam (subprocess) for network-free tests.
Advanced: defeat-graph backend (pkg/graph)
Use only when rules are not independent (one violation makes another moot) and you need inter-rule precedence or root-cause feedback. If level aggregation suffices, do not use the graph (it is overkill). Implement gate.Evaluator (Evaluate(ctx) quest.Verdict) and reins takes that path.
g := graph.NewGraph("myquest")
// tautology PASS warrant — supply your own always-true fn (graph has no exported helper)
pass := g.Warrant(func(toulmin.Context, toulmin.Specs) (bool, any) { return true, nil })
fmtR := g.Counter(ruleEmailFormat, gate.LevelFail).Attacks(pass)
holder := g.Counter(ruleSourceLacksEmail, gate.LevelFail).Attacks(pass).Needs("source-body")
fmtR.Supersedes(holder) // deterministic precedence (replaces hand-rolled guards)
v := g.EvaluateStaged(ctx, snap, provider) // tier-0 (no ground) first; resolve ground only if clean- `.Attacks(target)` = toulmin graph edge (verdict/contest). `.Supersedes(...)` = reins-side deterministic precedence (excludes a downstream counter from the tally).
- Side effects through ground, rules stay pure: declare
.Needs("name"), map it in a provider viaground.Snapshot(HTTPBody/MXResolves), andEvaluateStagedresolves it once (cached) intoctx.Grounds["name"]. Inject theground.Resolverinterface for network-free tests.
Conventions (the philosophy — follow these)
1. State the deterministic gate — judge from input alone; only the machine locks PASS. 2. Rules are violation detectors; severity is a level — never fake a hard check with a weight (continuous weighting is for the graph's genuine contest only). 3. Cheese defense first — for every answer to "how would I fool this gate?", add one rule (audited by rules). 4. Side effects through ground, rules stay pure — the network is owned by reins ground primitives, isolated by staged eval. 5. No N=1 abstraction — freeze a new abstraction only after a second consumer validates it. 6. Follow filefunc conventions if the project uses them — 1 file = 1 func/type (tests included), //ff:func///ff:type + //ff:what annotations at the top of each file.
Quick decision guide
| Situation | Use |
|---|---|
| Independent rules, simple gate | gate.Rule + Rules() (level aggregation) |
| Inter-rule precedence / root-cause feedback | pkg/graph + Evaluator + Supersedes |
| Side-effect verification (HTTP/DNS) | pkg/ground + .Needs() + EvaluateStaged |
| Block body hallucination | textmatch.MissingTokens |
| Date/time normalization | pkg/temporal |
| Short-circuit an untrusted submission | Prepare's short verdict |
| Unattended drive (LLM generates, gate judges) | Options{Loop} + pkg/llm |
Common Errors and Fixes
| Symptom | Cause | Fix |
|---|---|---|
loop subcommand missing | Options.Loop == nil | Set Options{Loop: &cli.LoopOptions{...}} |
invalid --model ...: model name is empty | empty model token | Use backend:model or the :default sentinel (e.g. claude:default) |
Gate never reaches PASS in loop | rule fires forever; sycophantic generator | Add RuleSystem[ruleID] coaching keyed on Verdict.RootCause; verify the rule is satisfiable |
| Item locks DONE without PASS | hit MaxTries (3) FAILs | Expected (monotone convergence); inspect Facts — the gate or prompt is mis-specified |
| Payload reads wrong/empty | wrote it.Payload directly | Use it.SetPayload(v) / it.DecodePayload(&v) only |
Render mutated state but it vanished | next does not Save; Render is read-only | Mutate s.Meta in Prepare (submit Saves after Prepare), not Render |
| Subprocess backend "command not found" | CLI not on PATH | Install the CLI (claude/grok/codex) and log in, or set REINS_<NAME>_BIN |
| Linking toulmin you don't use | imported pkg/graph | pkg/gate+pkg/cli don't import toulmin — skip the graph if level aggregation suffices |
Full Documentation
- `MANUAL.md` (repo root) — the complete manual for AI agents: package map, defeat-graph topology, staged-eval ground, walkthrough feedback, every backend.
- Reference consumer —
comail/main.goshows the one-linecli.NewQuestCmdwiring. - Quest philosophy — https://www.parkjunwoo.com/tech/how-make-quest.md ("generation is probabilistic, verification is deterministic").
# 별도 Go 모듈(자체 go.mod) — reins filefunc 검증 대상 아님
ccnews/
comail/
# 별도 repo로 분리된 하위 프로젝트 (루트 앵커 — plans/ccnews 등 하위 동명 디렉터리는 무시 안 함)
/comail/
/ccnews/
# 로컬 전용
CLAUDE.md
plans/
.clari/
.claude/
files/
bugs/
# go workspace (로컬 전용 — ../toulmin·./comail 미발행 모듈을 잇는 개발 편의. go.mod replace가 SSOT)
go.work
go.work.sum
# tsma 런타임 세션 상태
.tsma/
# whyso 로컬 맵 캐시
.whyso/
# 별도 Go 모듈(자체 go.mod)이라 reins 모듈에서 테스트 불가 — tsma 인덱싱 제외
ccnews/
comail/
required:
feature:
textmatch: "deterministic substring verifier — does a token literally appear in the source (NFC + whitespace-collapse, no fuzzy/synonyms); the anti-hallucination primitive gate rules call for cheese defense"
quest: "irreversible-progress core — one-way ratchet state machine, session persistence, progress tally, verdict application, terminal export (pure: no Cobra/toulmin/domain)"
gate: "deterministic verifier framework — a quest gate is a catalog of violation-detecting rules; each rule fires + emits a Fact, severity is a Level, Evaluate aggregates by level"
graph: "toulmin defeat-graph gate backend (isolated) — tautology PASS warrant + violation Counters that Attacks it; reins Level meta per node (no toulmin Strength); reins-side Supersedes precedence; trace Activated × Level − Supersedes → Verdict; edge-zero graph ≡ gate.Evaluate. Counter.Needs declares ground deps → staged EvaluateStaged: tier-0 (no-ground) first, residual Fail short-circuits (zero network), else lazy-resolve grounds → tier-1 (G5). pkg/gate does NOT import this (one-way, keeps comail toulmin-free)"
ground: "network ground primitives (reins asset like textmatch/temporal) — HTTPBody(url)→body, MXResolves(domain)→deliverable; per-evaluation Snapshot resolves once on first read + caches (determinism, open decision #8); injectable Resolver (default real net stack, tests inject fake → network-free); resolve error surfaced for the caller to reduce to a FAIL Fact"
temporal: "deterministic time-spec normalizer — converts an AI-identified structured time Spec (calendar/components/offset) to canonical Gregorian ISO (single/interval); undecidable → Determined=false"
cli: "Cobra command scaffold — the how-make-quest canonical skeleton (scan/next/submit/status/export) plus the rules rulebook; does IO/parsing/output only, never judges state"
llm: "LLM 호출 어댑터 — ollama/xai/gemini chat completion + claude(`claude -p`)·grok(`grok -p`)·codex(`codex exec`) 서브프로세스; generate 단계(L0)만 담당, 판정/래칫과 무관(권위 비대칭); HTTP는 net/http+encoding/json·API 키 env 전용(yaml 폴백 없음), 서브프로세스는 runSubprocess seam 공유·인증은 CLI 자체 로그인 위임(키 없음)"
type:
command: "cobra command entrypoint"
model: "data type definition"
helper: "internal utility function"
adapter: "external service client — wraps one provider endpoint behind the Backend contract (http+json, env key)"
loader: "environment/config resolver — reads a value from env only, no file fallback"
optional:
level:
error: "error-path handling"
module github.com/park-jun-woo/reins
go 1.22
require (
github.com/park-jun-woo/toulmin v0.1.0
github.com/spf13/cobra v1.10.2
golang.org/x/text v0.16.0
)
require (
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/spf13/pflag v1.0.9 // indirect
)
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/park-jun-woo/toulmin v0.1.0 h1:h94uQS+3lczUx+ajMjSwLhRLihHk9lrNXN6rFFqxslc=
github.com/park-jun-woo/toulmin v0.1.0/go.mod h1:WL7uvrIvlD+pZkynLdqyMbQUjG39AVg9GPB6KQinXys=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
MIT License
Copyright (c) 2026 Park Junwoo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
reins — Manual for AI Agents
A quest-CLI development framework (Go). It moves the authority to declare "done" from the AI to a deterministic machine gate. "Generation is probabilistic; verification is deterministic."
A consumer plugs in only the domain logic (gate.Definition); reins supplies the ratchet, the command skeleton, aggregation, feedback, and export. Agents are disposable; progress is cumulative and irreversible.
---
Core model
- Ratchet — a one-way state machine. Once PASS, it is irreversible; the remaining work
monotonically decreases.
- Gate — a set of violation-detecting rules. Each rule fires (true) when it finds a problem and
carries a Fact. Severity is a level (Fail/Review): a single decisive violation is FAIL.
- Authority asymmetry — only the machine locks PASS. L1 machine (deterministic, sole PASS
authority) / L2 AI (skeptic, REVIEW only) / L3 human (the remainder).
- Fact feedback — a FAIL is not an opinion but a located, quantified value (
Fact). It turns a
sycophantic model toward convergence.
Package map
| Package | Role | Deps |
|---|---|---|
pkg/quest | Ratchet core — Item·State·Verdict/Fact·Session·Apply·Export | (pure) |
pkg/gate | Gate contract — Definition·Rule·Level·Context·Evaluate(level aggregation)·Evaluator | quest |
pkg/graph | Defeat-graph backend — toulmin h-Categoriser. Graph·Counter·Supersedes·staged eval | gate, quest, toulmin |
pkg/ground | Network ground primitives — HTTPBody·MXResolves (injectable, snapshot) | (pure net) |
pkg/textmatch | Body-containment verification — Normalize(NFC)·Contains·MissingTokens. Hallucination block | x/text |
pkg/temporal | Time normalization — structured Spec → Gregorian ISO | (pure) |
pkg/llm | LLM call adapters — Backend(ollama/xai/gemini HTTP + claude/grok/codex/geminicli subprocess)·CallFunc·FromFlag·shared runSubprocess/no-tools preamble·auto num_ctx. Generation (L0) only; never judges/locks | net/http, os/exec |
pkg/cli | Cobra scaffold — NewQuestCmd → scan/next/submit/status/export/rules (+ opt-in loop) | cobra, llm |
toulmin isolation: onlypkg/graph/pkg/groundare heavy.pkg/gate·pkg/clido not import
toulmin, so a consumer that doesn't use the graph never links toulmin.
---
The simplest quest (4 methods + one line of main)
Implement the four gate.Definition methods and reins supplies the rest:
type Definition interface {
Seed(args []string) ([]*quest.Item, error) // input → initial TODOs
Render(s *quest.Session, it *quest.Item) (string, error) // the authoring prompt `next` shows (read-only s.Meta)
Prepare(s *quest.Session, it *quest.Item, raw []byte) (gate.Context, *quest.Verdict, error) // decode a submission (short-circuit if non-nil; may update s.Meta)
Rules() []gate.Rule // the gate's violation-rule catalog
}
func main() { cli.NewQuestCmd("myquest", myDef{}, cli.Options{Version: "0.1"}).Execute() }One rule = one violation detector. On fire it returns (true, Fact{Where,Expected,Actual}):
var whoAnchorPresent = gate.Rule{
Meta: gate.RuleMeta{ID: "who-anchor-present", Level: gate.LevelFail, Desc: "required who anchor is real"},
Check: func(ctx gate.Context) (bool, quest.Fact) {
sub := ctx.Submission.(*Event6)
if miss := textmatch.MissingTokens(ctx.Source, sub.Who.Anchors); len(miss) > 0 {
return true, quest.Fact{Where: "who.anchors", Expected: "source substring", Actual: miss[0]}
}
return false, quest.Fact{}
},
}gate.Evaluate(rules, ctx) aggregates fired rules by level: any Fail→FAIL, else any Review→REVIEW, else PASS. Deterministic (same (rules, ctx) → same Verdict).
Core types
// quest
type Item struct { Key string; State State; Tries int; Payload json.RawMessage; Log []Attempt; Emitted bool; … }
// Payload is raw JSON — write/read it via it.SetPayload(v) / it.DecodePayload(&v), never the field directly.
type State string // TODO PASS REVIEW DONE SKIPPED BLOCKED (terminal = PASS/REVIEW/DONE/SKIPPED/BLOCKED)
type Verdict struct { Outcome Outcome; Facts []Fact; Feedback string; RootCause string } // Outcome: PASS REVIEW FAIL SKIPPED BLOCKED; RootCause = the rule that caused FAIL/REVIEW (agent coaching)
type Fact struct { Rule, Where, Expected, Actual string }
const MaxTries = 3 // FAIL accrued to MaxTries → lock to DONE
// gate
type Context struct { Item *quest.Item; Submission any; Source string; Grounds map[string]string }
type Level int // LevelFail | LevelReviewquest.Apply(it, v, now)— applies a verdict to the ratchet (PASS/REVIEW/SKIPPED/BLOCKED lock, FAIL
is Tries++).
quest.Export(s, sink)— emits terminal, not-yet-emitted Items to the sink once (the export ratchet).- If
Prepare'sshort *quest.Verdict != nil, the gate is skipped and that verdict short-circuits
(e.g. an untrusted submission → OutSkip).
Command skeleton (NewQuestCmd)
scan input → seed N quests (for a streaming source, the consumer adds a run variant)
next one TODO + the authoring prompt / verification context
submit submit → gate eval → verdict → lock PASS / on FAIL emit Fact feedback
status progress tally (PASS/REVIEW/DONE/TODO/SKIPPED/BLOCKED)
export emit terminal results as JSONL (originals preserved, emit-once)
rules the gate's rule catalog (auto rulebook — audit the cheese it blocks)submit takes --key <k> + --in <file>|- (raw bytes → Prepare decodes). Every submit auto-emits terminal items to --out (default <name>-results.jsonl). Tune via Options{Out, Version}.
---
Unattended drive: the loop command (opt-in)
The same next→submit an external agent runs by hand, closed in-process as a generate→gate→retry loop: an LLM generates each TODO's payload, the gate judges, FAIL feedback is fed back until PASS or MaxTries. Opt in with Options{Loop: &LoopOptions{…}} (nil ⇒ the command is not attached, fully backward-compatible).
type LoopOptions struct {
DefaultModel string // "" ⇒ "ollama:gemma4:e4b"
System string // global generation system prompt
RuleSystem map[string]string // toulmin rule ID → extra system coaching when that rule was the FAIL root cause
LLM llm.Backend // injected backend (tests); when set, --model is ignored
}The loop (loop [--model backend:model] [--max-items N]):
for it := s.NextTODO(); it != nil; it = s.NextTODO() {
system := opts.System + RuleSystem[verdict.RootCause] // global + last-FAIL rule coaching
raw := backend.Complete(system, def.Render(s,it)+feedback) // LLM generates (L0)
verdict := evaluateAndApply(...) // SAME path as submit: gate→Apply→export
if verdict.Outcome != quest.OutFail { break } // PASS/REVIEW/SKIP/BLOCK → lock, next item
feedback = renderVerdictText(...) // identical to what submit prints
}- Authority asymmetry holds — the LLM is only the generator (L0). Only the gate locks PASS. The
loop calls quest.Apply; on the MaxTries-th FAIL it locks DONE → NextTODO drops it → the loop terminates (monotone convergence). The framework exposes no API to grant the LLM PASS authority.
- `Verdict.RootCause` (additive, backward-compatible field) names the rule that caused FAIL/REVIEW —
set deterministically on both paths: flat gate.Evaluate (first fired Fail rule's ID) and the graph backend (selectRootCause's top counter). RuleSystem[RootCause] turns it into rule-specific coaching on retry.
- Feedback parity — the FAIL text fed to the model is the very string
submitprints
(renderVerdict/renderVerdictText shared), so human-visible and model-visible feedback never drift.
- Backends (
pkg/llm): HTTP —ollama:<model>(local, no key,num_ctxauto-sized from prompt
length), xai:<model>/gemini:<model> (OpenAI-compat / Gemini, env-only API keys); all three are net/http. HTTP options (struct fields; zero ⇒ prior default, fully backward-compatible): all three take MaxOutputTokens int (0 ⇒ 2048 — raise it so reasoning models like qwen3/gpt-oss aren't truncated; for ollama it also grows the auto-sized num_ctx so the window holds the larger output) and Temperature *float64 (nil ⇒ 0); ollama additionally takes Think *bool (false ⇒ disable reasoning to save the output budget). Set them on the injected backend, or via a `--model` query: ollama:qwen3:8b?max_output_tokens=8192&think=false / xai:grok-4?max_output_tokens=4096&temperature=0.7. FromFlag parses ?k=v&… after the model and rejects any key a backend doesn't accept (allowed: ollama max_output_tokens/num_ctx/temperature/think, xai & gemini max_output_tokens/temperature, subprocess none) — no silent caps. Subprocess — claude:<model>/grok:<model>/codex:<model>/geminicli:<model> shell out to a CLI via os/exec; auth is delegated entirely to that CLI's own login (subscription/OAuth/ keychain/env key) so reins reads no API key for them. All subprocess backends share runSubprocess and the fixed no-tools preamble (withNoToolsPreamble); each exposes a var exec<Name> package seam for subprocess-free tests, a pointer-receiver adapter that carries an opt-in session id, and a :<model> or :default token (default ⇒ the CLI's configured model, since FromFlag rejects an empty model). Inject llm.CallFunc for network-free HTTP tests.
- `claude:<model>` — Claude CLI (
claude -p, headless print):--max-turns 1pins a single-shot L0
generator (no agentic tool loop), --tools "" + no-tools preamble block tool narration, --permission-mode dontAsk keeps it from blocking. Token claude:opus/claude:sonnet/claude:default; REINS_CLAUDE_BIN overrides the binary; envelope --output-format json → result/session_id/is_error.
- `grok:<model>` — Grok CLI (
grok -p, single-turn): the claude twin (--max-turns 1+
--tools ""/--disable-web-search/--no-subagents/--no-memory + preamble + --permission-mode dontAsk), reaching the same xAI models as the xai: HTTP backend but over the CLI login (no API key). The user prompt travels via --prompt-file (grok's -p takes the prompt as a value; stdin is not accepted). Envelope --output-format json → text/sessionId/stopReason (success = EndTurn). REINS_GROK_BIN overrides the binary.
- `codex:<model>` — Codex CLI (
codex exec, headless agent): has no `--max-turns`, so the single-shot
L0 guarantee leans on -s read-only (block side effects) + the no-tools preamble + --ignore-user-config/ --ignore-rules (keep CODEX_HOME auth, drop repo AGENTS.md/rules). Output is a --json JSONL event stream: the last item.completed with agent_message is the result text; thread.started.thread_id is the session id. The system prompt is prepended into the stdin prompt (codex has no system channel). Token codex:gpt-5/codex:o3/codex:default; REINS_CODEX_BIN overrides the binary.
- `geminicli:<model>` — Gemini CLI (
gemini -p, headless agent): same model family as thegemini:HTTP
backend but over the Google-account login (no `GEMINI_API_KEY`) — hence a separate token, since gemini: is taken by the HTTP backend. Like codex it has no `--max-turns`, so the single-shot L0 guarantee leans on --approval-mode plan (read-only) + the no-tools preamble. The prompt rides on stdin (-p "" triggers headless; stdin carries the body). Output is --output-format json → .response (result) / .error / .stats.tools.calls (a calls == 0 smoke confirms no tool was used). Token geminicli:gemini-2.5-pro/geminicli:gemini-2.5-flash/geminicli:default; REINS_GEMINI_BIN overrides the binary.
- Session (subprocess backends, default fully stateless): each
Completeis independent (claude
--no-session-persistence / codex --ephemeral / grok no-resume), matching the HTTP backends and reins' own deterministic FAIL-feedback convergence. Opt into carrying the CLI's own conversation across a run with REINS_<NAME>_SESSION=continue — the first reply's session id is carried into later calls as --resume (claude/grok), the exec resume <id> subcommand (codex; the read-only flag stays at exec level, before the subcommand), or a reins-issued UUID via --session-id then --resume latest (geminicli). Any other value falls back to stateless (no forged session). Note: session mode double-exposes the prior attempt (model history and re-fed FAIL text), so stateless is recommended.
---
Advanced: the defeat-graph backend (pkg/graph)
Use it when rules are not independent (one violation makes another moot). If a Definition implements gate.Evaluator (Evaluate(ctx) quest.Verdict), reins takes that path instead of Rules() (Rules() is kept for the rules catalog). If level aggregation suffices, do not use the graph (it is overkill).
Topology: one tautology PASS warrant + every violation = a Counter that attacks the warrant.
g := graph.NewGraph("myquest")
pass := g.Warrant(alwaysTrue) // always-active PASS warrant
fmtR := g.Counter(ruleEmailFormat, gate.LevelFail).Attacks(pass)
holder := g.Counter(ruleSourceLacksEmail, gate.LevelFail).Attacks(pass)
free := g.Counter(ruleFreemail, gate.LevelReview).Attacks(pass)
fmtR.Supersedes(holder) // bad format → drop the source check from the tally (precedence)
free.Supersedes(holder) // freemail → absorb the source check → preserve REVIEW- `.Supersedes(...)` = reins-side deterministic precedence (an active upstream counter excludes a
downstream one from the tally). It replaces hand-rolled guards. (toulmin's Attacks defeat only lowers the verdict float and cannot clear Activated, so crisp precedence goes through Supersedes.)
- `.Attacks(target)` = a toulmin graph edge (for the verdict/contest). Violation→warrant is Attacks.
- Decision:
g.Evaluate(ctx)takes active counters − superseded = remaining and aggregates them by
Level → Verdict (+ the walkthrough Feedback). With zero edges it equals gate.Evaluate (graph.FromRules(rules)).
Side effects / network: ground provider + staged eval (G5)
Do not put side effects (HTTP/DNS) inside rules. reins provides ground primitives:
snap := ground.NewSnapshot(nil) // nil = real net; tests inject a fake Resolver
// a counter declares its ground dependency → automatic tier classification
holder := g.Counter(ruleSourceLacksEmail, gate.LevelFail).Attacks(pass).Needs("source-body")
mx := g.Counter(ruleMxMissing, gate.LevelFail).Attacks(pass).Needs("mx")
// the consumer maps a ground name → the actual resolve
provider := func(name string, ctx gate.Context, snap *ground.Snapshot) (string, error) {
c := ctx.Submission.(*Candidate)
switch name {
case "source-body": return snap.HTTPBody(c.Source)
case "mx": b, e := snap.MXResolves(domain(c.Email)); return fmt.Sprint(b), e
}
return "", fmt.Errorf("unknown ground %q", name)
}
v := g.EvaluateStaged(ctx, snap, provider)- Staged: the no-ground tier 0 is evaluated first → **if a FAIL remains, stop immediately (no
ground is resolved = zero network)**. If clean, each ground is snapshot-resolved once → injected into ctx.Grounds → tier 1 is evaluated.
- A ground is snapshotted/cached once per request (re-reading the same URL is still one call).
Rules read ctx.Grounds["source-body"] and stay pure. A resolve failure is a deterministic FAIL Fact.
- Inject the
ground.Resolverinterface (Fetch/LookupMX) so tests are deterministic with no
network.
Walkthrough feedback (Verdict.Feedback)
On FAIL/REVIEW, graph evaluation fills Verdict.Feedback with an agent-facing walkthrough — not a flat Fact list but "why you lost + what to change to win":
FAIL. root cause = email-format (remaining active FAIL, upstream).
Fact: where=email expected="valid email format" actual="not-an-email"
source-lacks-email: superseded by email-format → side-branch.
→ to flip the verdict, clear email-format.The cli submit prints Feedback when present, otherwise the Fact list (backward-compatible for level-aggregation consumers).
---
Verification primitives
textmatch.Normalize(s) // NFC + whitespace fold + Trim (no case-fold — ToLower first if needed)
textmatch.Contains(source, token) // substring after normalization
textmatch.MissingTokens(source, toks) // tokens absent from source (hallucination block)
temporal.Resolve(spec, ref) // structured Spec (calendar/components/offset) + ref time.Time (injected now) → Gregorian ISO (undetermined ⇒ Determined=false)
temporal.ComponentsInAnchor(...) // extract time components from an anchor stringMatch source-language anchors with Normalize (combining marks, NFC); the recommended pattern is to unify the output value in English.
---
Conventions (philosophy)
- State the deterministic gate — judge from input alone; only the machine locks PASS.
- Rules are violation detectors; severity is a level — never fake a hard check with a weight
(continuous weighting is for the graph's genuine contest only).
- Cheese defense first — for every answer to "how would I fool this gate?", add one rule (audited
by the auto catalog).
- Side effects through ground, rules stay pure — the network is owned by reins ground primitives,
isolated by staged eval.
- No N=1 abstraction — freeze a new abstraction only after a second consumer validates it.
Quick decision guide
| Situation | Use |
|---|---|
| Independent rules, simple gate | gate.Rule + Rules() (level aggregation) |
| Inter-rule precedence / root-cause feedback | pkg/graph + Evaluator + Supersedes |
| Side-effect verification (HTTP/DNS) | pkg/ground + .Needs() + EvaluateStaged |
| Block body hallucination | textmatch.MissingTokens |
| Date/time normalization | pkg/temporal |
| Short-circuit an untrusted submission | Prepare's short verdict (OutSkip/OutBlock) |
| A streaming source instead of a one-shot seed | the consumer adds a run command (not yet shipped by reins) |
| Unattended drive (LLM generates, gate judges) | Options{Loop} + pkg/llm — the opt-in loop command; rule-specific coaching via RuleSystem/Verdict.RootCause |
reins
Copyright (c) 2026 Park Junwoo
This product is licensed under the MIT License (see the LICENSE file).
================================================================================
Third-party software
================================================================================
This product depends on the following third-party modules. Their license texts
are available in the respective upstream repositories and in the Go module
cache of a source checkout.
--------------------------------------------------------------------------------
github.com/spf13/cobra
Licensed under the Apache License, Version 2.0.
Copyright 2013-2023 The Cobra Authors.
--------------------------------------------------------------------------------
github.com/spf13/pflag
Licensed under the BSD 3-Clause License.
Copyright (c) 2012 Alex Ogier. All rights reserved.
Copyright (c) 2012 The Go Authors. All rights reserved.
--------------------------------------------------------------------------------
github.com/inconshreveable/mousetrap
(included in Windows builds via cobra)
Licensed under the Apache License, Version 2.0.
Copyright 2022 Alan Shreve (@inconshreveable)
--------------------------------------------------------------------------------
golang.org/x/text
Licensed under the BSD 3-Clause License, with an additional IP rights grant
(see the upstream PATENTS file).
Copyright (c) 2009 The Go Authors. All rights reserved.
--------------------------------------------------------------------------------
github.com/park-jun-woo/toulmin
Copyright (c) 2026 Park Junwoo (same author as this product).
================================================================================
No third-party source code is copied or vendored into this repository; the
modules above are consumed as ordinary Go module dependencies.
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestApplyVerdictExportError — sink 생성은 성공하되 export Emit이 실패하는 outPath(디렉터리 경로)에서 applyVerdict가 exportAndSave 에러를 전파하는지 단언. PASS로 아이템을 terminal화해 Export가 실제 Emit을 시도하게 한 뒤 그 실패 분기를 고정한다.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestApplyVerdictExportError: the sink is created fine, but the export Emit fails
// because outPath is a directory (OpenFile on a dir errors). A PASS verdict makes the
// item terminal so Export actually tries to emit it, exercising the exportAndSave
// error branch — distinct from the Save and sink-open branches.
func TestApplyVerdictExportError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "outdir")
if err := os.Mkdir(out, 0o755); err != nil {
t.Fatal(err)
}
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v := quest.Verdict{Outcome: quest.OutPass}
if err := applyVerdict(s, it, v, out, session); err == nil {
t.Fatal("err = nil, want exportAndSave (Emit) error to propagate")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestApplyVerdictFailRatchets — FAIL verdict 적용 시 it.Tries가 1 증가하고(잠금 없음, State는 TODO 유지) 세션이 디스크에 Save되어 재로드 가능한지 단언. verdict가 래칫을 바꾸는 단일 지점의 FAIL 분기를 고정한다.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestApplyVerdictFailRatchets: applying a FAIL verdict bumps it.Tries by one (no
// lock — the item stays TODO) and persists the session so it reloads from disk.
func TestApplyVerdictFailRatchets(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v := quest.Verdict{Outcome: quest.OutFail, RootCause: "r1"}
if err := applyVerdict(s, it, v, out, session); err != nil {
t.Fatalf("applyVerdict error: %v", err)
}
if it.Tries != 1 {
t.Fatalf("tries = %d, want 1", it.Tries)
}
if it.State != quest.TODO {
t.Fatalf("state = %v, want TODO (FAIL must not lock before MaxTries)", it.State)
}
reloaded, err := quest.Load(session)
if err != nil {
t.Fatalf("session not reloadable: %v", err)
}
if len(reloaded.Items) != 1 || reloaded.Items[0].Tries != 1 {
t.Fatalf("reloaded tries = %v, want 1", reloaded.Items)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestApplyVerdictPassLocks — PASS verdict 적용 시 아이템이 PASS로 잠기고 세션·export(JSONL)가 영속되는지 단언. applyVerdict는 게이트가 건넨 PASS를 그대로 적용·emit하는 단일 지점임을 고정한다.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestApplyVerdictPassLocks: applying a PASS verdict locks the item PASS and persists
// both the session and the export (JSONL) file.
func TestApplyVerdictPassLocks(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v := quest.Verdict{Outcome: quest.OutPass}
if err := applyVerdict(s, it, v, out, session); err != nil {
t.Fatalf("applyVerdict error: %v", err)
}
if it.State != quest.PASS {
t.Fatalf("state = %v, want PASS", it.State)
}
if _, err := os.Stat(session); err != nil {
t.Fatalf("session not saved: %v", err)
}
data, err := os.ReadFile(out)
if err != nil || len(data) == 0 {
t.Fatalf("export not written: %v / %d bytes", err, len(data))
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestApplyVerdictSaveError — 쓰기 불가 sessionPath(정규 파일 하위 경로)에서 applyVerdict가 Save 실패를 그대로 전파하는지 단언. 정합성 위협(영속화 실패)은 침묵 금지.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestApplyVerdictSaveError: when the session path is unwritable (nested under a
// regular file), applyVerdict propagates the Save error rather than swallowing it —
// a persistence failure must never be silent.
func TestApplyVerdictSaveError(t *testing.T) {
dir := t.TempDir()
out := filepath.Join(dir, "out.jsonl")
badParent := filepath.Join(dir, "afile")
if err := os.WriteFile(badParent, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
session := filepath.Join(badParent, "nested", "session.json")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v := quest.Verdict{Outcome: quest.OutFail}
if err := applyVerdict(s, it, v, out, session); err == nil {
t.Fatal("err = nil, want Save error to propagate")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestApplyVerdictSinkError — 세션 Save는 성공하되 export sink 생성이 실패하는 outPath(정규 파일 하위 경로)에서 applyVerdict가 newJSONLSink 에러를 전파하는지 단언. Save와 sink-open 두 실패 분기를 분리해 고정한다.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestApplyVerdictSinkError: the session saves fine, but the export sink cannot be
// created (outPath nested under a regular file), so applyVerdict propagates the
// newJSONLSink error — distinct from the Save-failure branch.
func TestApplyVerdictSinkError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
badParent := filepath.Join(dir, "afile")
if err := os.WriteFile(badParent, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
out := filepath.Join(badParent, "nested", "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v := quest.Verdict{Outcome: quest.OutFail}
if err := applyVerdict(s, it, v, out, session); err == nil {
t.Fatal("err = nil, want newJSONLSink error to propagate")
}
if _, statErr := os.Stat(session); statErr != nil {
t.Fatalf("session should have been saved before the sink error: %v", statErr)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what applyVerdict — verdict를 아이템 래칫에 적용하고 영속화하는 공용 꼬리. quest.Apply(UTC RFC3339)→s.Save→newJSONLSink→exportAndSave(Export 실패여도 Save로 Emitted 래칫 보존). verdict가 래칫을 바꾸는 단일 지점으로, 게이트 경로(evaluateAndApply)와 백엔드-에러 경로(runLoopItem)가 공유한다. PASS 잠금 권한은 여전히 게이트뿐 — 이 헬퍼는 주어진 verdict를 적용할 뿐이다.
package cli
import (
"time"
"github.com/park-jun-woo/reins/pkg/quest"
)
// applyVerdict ratchets a verdict onto an item and persists it: quest.Apply(UTC) →
// Save → exportAndSave (the Emitted ratchet survives an Export failure). It is the
// single place a verdict mutates the ratchet, shared by the gate path
// (evaluateAndApply) and the backend-error path (runLoopItem). It does not lock PASS
// on its own — the caller supplies the verdict, and PASS still originates only at the
// gate.
func applyVerdict(s *quest.Session, it *quest.Item, v quest.Verdict, outPath, sessionPath string) error {
now := time.Now().UTC().Format(time.RFC3339)
quest.Apply(it, v, now)
if err := s.Save(sessionPath); err != nil {
return err
}
sink, err := newJSONLSink(outPath)
if err != nil {
return err
}
if _, err := exportAndSave(s, sink, sessionPath); err != nil {
return err
}
return nil
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what backendErr.Error — 래핑한 문자열을 error 메시지로 반환한다(error 인터페이스 만족).
package cli
// Error satisfies the error interface for backendErr.
func (e backendErr) Error() string { return string(e) }
//ff:type feature=cli type=model
//ff:what 테스트용 에러 타입. backendErr는 문자열을 error로 감싸 스텁 Backend가 반환할 에러(errBackend)를 만든다.
package cli
// backendErr is a string-typed error for stub backends to return.
type backendErr string
// errBackend is the canonical stub backend error.
var errBackend = backendErr("boom")
//ff:func feature=cli type=helper control=sequence
//ff:what TestBackendErrorVerdict — L0 생성 오류를 감싼 verdict가 재시도 가능한 FAIL(RootCause=backend-error)이고, 원문 에러를 backend.Complete 위치의 단일 Fact로 싣는지 단언. 게이트 실패와 생성 실패의 분류 경계를 고정한다.
package cli
import (
"errors"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestBackendErrorVerdict: the verdict wrapping an L0 generation error is a retryable
// FAIL whose RootCause is the reserved "backend-error" rule, carrying the original
// error text in a single Fact located at backend.Complete.
func TestBackendErrorVerdict(t *testing.T) {
err := errors.New("boom")
v := backendErrorVerdict(err)
if v.Outcome != quest.OutFail {
t.Fatalf("outcome = %v, want %v", v.Outcome, quest.OutFail)
}
if v.RootCause != "backend-error" {
t.Fatalf("root cause = %q, want %q", v.RootCause, "backend-error")
}
if len(v.Facts) != 1 {
t.Fatalf("facts len = %d, want 1", len(v.Facts))
}
f := v.Facts[0]
if f.Rule != "backend-error" {
t.Fatalf("fact rule = %q, want %q", f.Rule, "backend-error")
}
if f.Where != "backend.Complete" {
t.Fatalf("fact where = %q, want %q", f.Where, "backend.Complete")
}
if f.Actual != err.Error() {
t.Fatalf("fact actual = %q, want %q", f.Actual, err.Error())
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what backendErrorVerdict — L0 생성 오류(backend.Complete 실패)를 재시도 가능한 FAIL로 감싼다. 예약 규칙 "backend-error" 아래 Fact 한 건에 원문 에러 텍스트를 실어, 내용 비평을 지어내지 않고도 run 로그가 '왜' 시도가 실패했는지 기록한다. RootCause=backend-error로 게이트 실패와 생성 실패를 구분 가능.
package cli
import "github.com/park-jun-woo/reins/pkg/quest"
// backendErrorRule is the reserved synthetic rule ID marking a generation-stage
// failure (an L0 backend.Complete error), distinct from any consumer rule a gate
// might fire. RootCause==backendErrorRule on a Verdict means "the generator failed",
// not "the gate rejected the content".
const backendErrorRule = "backend-error"
// backendErrorVerdict wraps an L0 generation error as a retryable FAIL: a single
// Fact under the reserved rule "backend-error" carrying the error text, so the run
// log records why the attempt failed without inventing a content critique.
func backendErrorVerdict(err error) quest.Verdict {
return quest.Verdict{
Outcome: quest.OutFail,
RootCause: backendErrorRule,
Facts: []quest.Fact{{
Rule: backendErrorRule,
Where: "backend.Complete",
Actual: err.Error(),
}},
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what TestComposeSystemFallbackUsed — 빈 전역·빈 코칭일 때 fallback이 reins 기본 가이던스("deterministic gate")를 담는지 검증.
package cli
import (
"strings"
"testing"
)
// TestComposeSystemFallbackUsed: the fallback contains the canonical reins guidance.
func TestComposeSystemFallbackUsed(t *testing.T) {
got := composeSystem("", "")
if !strings.Contains(got, "deterministic gate") {
t.Fatalf("fallback = %q, want generic reins prompt", got)
}
}
//ff:func feature=cli type=helper control=iteration dimension=1
//ff:what TestComposeSystem — 세 분기를 테이블로 검증: 빈 전역→fallback, 빈 코칭→전역만, 둘 다 있으면 줄바꿈 결합.
package cli
import (
"testing"
)
// TestComposeSystem covers the three branches: empty global falls back to the
// generic prompt; empty coaching returns the global unchanged; both present are
// joined with a newline.
func TestComposeSystem(t *testing.T) {
cases := []struct {
name string
global string
coach string
want string
}{
{"both-empty", "", "", fallbackSystem},
{"global-only", "G", "", "G"},
{"fallback-with-coach", "", "C", fallbackSystem + "\nC"},
{"both", "G", "C", "G\nC"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := composeSystem(c.global, c.coach)
if got != c.want {
t.Fatalf("composeSystem(%q,%q) = %q, want %q", c.global, c.coach, got, c.want)
}
})
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what composeSystem — 전역 system 프롬프트에 규칙별 코칭을 결합한다. 빈 전역은 reins 기본 프롬프트(fallbackSystem)로, 빈 코칭은 결합 없이 전역만 반환한다.
package cli
// fallbackSystem is the generic system prompt used when LoopOptions.System is empty.
const fallbackSystem = "You produce a submission a deterministic gate will judge. " +
"Output only the payload in the exact format the prompt specifies; no prose."
// composeSystem combines the global system prompt with optional rule-specific
// coaching. An empty global falls back to the generic reins prompt; empty coaching
// adds nothing.
func composeSystem(global, ruleCoach string) string {
if global == "" {
global = fallbackSystem
}
if ruleCoach == "" {
return global
}
return global + "\n" + ruleCoach
}
// Package cli is reins' Cobra command scaffold (reins Phase004). It implements the
// how-make-quest canonical command skeleton — scan / next / submit / status / export
// — plus reins' own `rules` (the auto rulebook). A new quest gets the whole standard
// CLI by plugging in a gate.Definition (Seed/Render/Prepare/Rules); everything else
// (the ratchet, level aggregation, export) is driven by reins.
//
// cli does IO, parsing, and output formatting only — it never judges state. The
// verdict comes from gate (deterministic rule aggregation) and the transition from
// quest.Apply (pure). Commands use RunE and cmd.OutOrStdout/InOrStdin for testability.
package cli
//ff:func feature=cli type=helper control=sequence level=error
//ff:what errDef.Prepare. prepareErr면 에러를, 아니면 raw를 Submission으로 담은 Context를 돌려준다(에러 분기 테스트 더블).
package cli
import (
"errors"
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/quest"
)
func (d errDef) Prepare(_ *quest.Session, it *quest.Item, raw []byte) (gate.Context, *quest.Verdict, error) {
if d.prepareErr {
return gate.Context{}, nil, errors.New("prepare boom")
}
return gate.Context{Item: it, Submission: string(raw)}, nil, nil
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what errDef.Render. renderErr면 에러를, 아니면 "render:<key>"를 돌려준다(에러 분기 테스트 더블).
package cli
import (
"errors"
"github.com/park-jun-woo/reins/pkg/quest"
)
func (d errDef) Render(_ *quest.Session, it *quest.Item) (string, error) {
if d.renderErr {
return "", errors.New("render boom")
}
return "render:" + it.Key, nil
}
//ff:func feature=cli type=helper control=sequence
//ff:what errDef.Rules. 규칙이 없는 게이트를 흉내내 nil을 돌려준다(테스트 더블).
package cli
import "github.com/park-jun-woo/reins/pkg/gate"
func (d errDef) Rules() []gate.Rule { return nil }
//ff:func feature=cli type=helper control=iteration dimension=1 level=error
//ff:what errDef.Seed. seedErr면 에러를, 아니면 args 하나당 TODO 아이템을 시드한다(에러 분기 테스트 더블).
package cli
import (
"errors"
"github.com/park-jun-woo/reins/pkg/quest"
)
func (d errDef) Seed(args []string) ([]*quest.Item, error) {
if d.seedErr {
return nil, errors.New("seed boom")
}
items := make([]*quest.Item, len(args))
for i, a := range args {
items[i] = &quest.Item{Key: a, State: quest.TODO}
}
return items, nil
}
//ff:type feature=cli type=model
//ff:what 테스트용 gate.Definition. seed/render/prepare 플래그가 켜지면 해당 단계가 실패해 명령의 에러 분기(level=error)를 자극한다. 플래그가 꺼지면 stubDef처럼 동작해 happy path도 배선된다.
package cli
// errDef is a Definition whose Seed/Render/Prepare each fail when the corresponding
// flag is set, so the command error branches (level=error) can be exercised. When a
// flag is unset it behaves like stubDef so the happy path still wires up.
type errDef struct {
seedErr bool
renderErr bool
prepareErr bool
}
//ff:func feature=cli type=helper control=iteration dimension=1
//ff:what escalateRootCauses — EscalateOn 슬라이스를 O(1) 조회용 set으로 변환. nil·빈 슬라이스는 빈 set을 내므로 에스컬레이션이 절대 발화하지 않는다.
package cli
// escalateRootCauses builds a set from the EscalateOn slice for O(1) lookup. A nil
// or empty slice yields an empty set, so escalation never fires.
func escalateRootCauses(ids []string) map[string]bool {
m := make(map[string]bool, len(ids))
for _, id := range ids {
m[id] = true
}
return m
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplyExportEmitError — newJSONLSink는 성공(부모 디렉터리 존재)하지만 out 경로가 디렉터리라 Emit이 실패하는 Export 에러 분기를 자극한다(sink 생성 분기와 구분).
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplyExportEmitError: newJSONLSink succeeds (parent dir exists) but
// Emit fails because the out path is itself a directory — exercising the Export
// error branch (distinct from the sink-construction branch).
func TestEvaluateAndApplyExportEmitError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
// out is an existing directory: MkdirAll(parent) succeeds, OpenFile(out) fails.
out := filepath.Join(dir, "outdir")
if err := os.MkdirAll(out, 0o755); err != nil {
t.Fatal(err)
}
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
if _, err := evaluateAndApply(stubDef{}, s, it, []byte("good"), out, session); err == nil {
t.Fatal("evaluateAndApply = nil error, want Export emit error")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplyGraphFailRootCause — Evaluator(graph) 분기가 defeat 그래프에서 verdict를 채우고 FAIL이 RootCause를 담으며 아이템을 잠그지 않는지 검증.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplyGraphFailRootCause: the Evaluator (graph) branch fills the
// verdict from the defeat graph, and a FAIL carries a RootCause.
func TestEvaluateAndApplyGraphFailRootCause(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v, err := evaluateAndApply(graphDef{}, s, it, []byte("bad"), out, session)
if err != nil {
t.Fatalf("evaluateAndApply error: %v", err)
}
if v.Outcome != quest.OutFail {
t.Fatalf("outcome = %v, want FAIL", v.Outcome)
}
if v.RootCause == "" {
t.Fatalf("graph FAIL verdict missing RootCause: %+v", v)
}
if it.State != quest.TODO {
t.Fatalf("state = %v, want TODO (FAIL does not lock)", it.State)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplyPassLocks — flat Rules() 경로에서 통과 제출이 아이템을 PASS로 잠그고 세션·export 파일을 영속하는지 검증.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplyPassLocks: a passing submission on the flat Rules() path locks
// the item PASS and persists the session and the export file.
func TestEvaluateAndApplyPassLocks(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v, err := evaluateAndApply(stubDef{}, s, it, []byte("good"), out, session)
if err != nil {
t.Fatalf("evaluateAndApply error: %v", err)
}
if v.Outcome != quest.OutPass {
t.Fatalf("outcome = %v, want PASS", v.Outcome)
}
if it.State != quest.PASS {
t.Fatalf("state = %v, want PASS", it.State)
}
if _, err := os.Stat(session); err != nil {
t.Fatalf("session not saved: %v", err)
}
data, err := os.ReadFile(out)
if err != nil || len(data) == 0 {
t.Fatalf("export not written: %v / %d bytes", err, len(data))
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplyPrepareError — Prepare 에러가 그대로 반환되고(apply 없음) 아이템이 TODO로 남는지 검증.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplyPrepareError: a Prepare error is returned directly (no apply).
func TestEvaluateAndApplyPrepareError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
if _, err := evaluateAndApply(errDef{prepareErr: true}, s, it, []byte("x"), out, session); err == nil {
t.Fatal("evaluateAndApply = nil error, want Prepare error")
}
if it.State != quest.TODO {
t.Fatalf("state = %v, want TODO (no apply on Prepare error)", it.State)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplySessionSaveError — 쓰기 불가 세션 경로(부모가 파일)가 verdict 계산·적용 후 Save 에러를 표면화하는지 검증.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplySessionSaveError: an unwritable session path surfaces the
// save error after the verdict is computed and applied.
func TestEvaluateAndApplySessionSaveError(t *testing.T) {
dir := t.TempDir()
out := filepath.Join(dir, "out.jsonl")
// A session path under a file (not a dir) makes Save fail.
badParent := filepath.Join(dir, "afile")
if err := os.WriteFile(badParent, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
session := filepath.Join(badParent, "nested", "session.json")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
if _, err := evaluateAndApply(stubDef{}, s, it, []byte("good"), out, session); err == nil {
t.Fatal("evaluateAndApply = nil error, want session save error")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplyShortVerdict — Prepare 단락 verdict(SKIP)가 규칙 카탈로그를 거치지 않고 그대로 쓰이는지 검증.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplyShortVerdict: a Prepare short-circuit verdict (SKIP) is used
// verbatim without invoking the rule catalog.
func TestEvaluateAndApplyShortVerdict(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
v, err := evaluateAndApply(stubDef{}, s, it, []byte("skip"), out, session)
if err != nil {
t.Fatalf("evaluateAndApply error: %v", err)
}
if v.Outcome != quest.OutSkip {
t.Fatalf("outcome = %v, want SKIP", v.Outcome)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestEvaluateAndApplySinkError — 쓰기 불가 export 경로(부모가 파일)가 verdict 계산 후 sink 생성 에러를 표면화하는지 검증(OS별 메시지 차이에 관대).
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEvaluateAndApplySinkError: an unwritable export path surfaces an error after
// the verdict is computed.
func TestEvaluateAndApplySinkError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
// A path whose parent is a file (not a dir) makes the sink mkdir/open fail.
badParent := filepath.Join(dir, "afile")
if err := os.WriteFile(badParent, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
out := filepath.Join(badParent, "nested", "out.jsonl")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
if _, err := evaluateAndApply(stubDef{}, s, it, []byte("good"), out, session); err == nil {
t.Fatal("evaluateAndApply = nil error, want sink error")
} else if !strings.Contains(err.Error(), "afile") && !strings.Contains(strings.ToLower(err.Error()), "not a directory") {
// Tolerant: just ensure an error happened; the message varies by OS.
t.Logf("sink error (ok): %v", err)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what evaluateAndApply — submit·loop 공용 헬퍼. def.Prepare(s,it,raw)→(short verdict, def가 gate.Evaluator면 ev.Evaluate(그래프), 아니면 gate.Evaluate(Rules))→quest.Apply(UTC RFC3339)→Save→exportAndSave(Export 실패여도 Save로 Emitted 래칫 보존) 후 verdict 반환. 게이트가 PASS를 잠그는 단일 지점. submit과 loop가 같은 판정·래칫·export 경로를 공유한다(DRY).
package cli
import (
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/quest"
)
// evaluateAndApply runs the gate over one submission, applies the ratchet
// transition, persists the session, and exports terminal items. It is the single
// place that locks PASS, shared by submit and the loop command.
func evaluateAndApply(def gate.Definition, s *quest.Session, it *quest.Item, raw []byte, outPath, sessionPath string) (quest.Verdict, error) {
ctx, short, err := def.Prepare(s, it, raw)
if err != nil {
return quest.Verdict{}, err
}
var verdict quest.Verdict
if short != nil {
verdict = *short
} else if ev, ok := def.(gate.Evaluator); ok {
verdict = ev.Evaluate(ctx)
} else {
verdict = gate.Evaluate(def.Rules(), ctx)
}
if err := applyVerdict(s, it, verdict, outPath, sessionPath); err != nil {
return verdict, err
}
return verdict, nil
}
//ff:type feature=cli type=model
//ff:func feature=cli type=helper control=sequence level=error
//ff:what 테스트용 fake sink(failAtSink). n번째 Emit(1-based)에서 에러를 주입해 exportAndSave가 부분 실패 시에도 세션을 저장하고 Emitted 래칫을 디스크에 보존하는지, 재export 시 1번째가 재방출되지 않는지(emit-once 보존 게이트)를 검증한다.
package cli
import (
"errors"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// failAtSink fails on the nth Emit (1-based) and records emitted keys otherwise.
type failAtSink struct {
calls int
failAt int
keys []string
}
func (f *failAtSink) Emit(it *quest.Item) error {
f.calls++
if f.calls == f.failAt {
return errors.New("emit boom")
}
f.keys = append(f.keys, it.Key)
return nil
}
// TestExportAndSavePartialFailure: with two terminal items, the first Emit succeeds
// and the second fails — exportAndSave must still save the session so the first
// item's Emitted ratchet persists, and a re-export after reload emits only the
// second item (the first is never re-emitted: emit-once preserved).
func TestExportAndSavePartialFailure(t *testing.T) {
session := filepath.Join(t.TempDir(), "session.json")
s := &quest.Session{Version: 1, Items: []*quest.Item{
{Key: "a", State: quest.PASS},
{Key: "b", State: quest.PASS},
}}
n, err := exportAndSave(s, &failAtSink{failAt: 2}, session)
if err == nil {
t.Fatal("exportAndSave = nil error, want emit error")
}
if n != 1 {
t.Fatalf("n = %d, want 1 (first emitted before failure)", n)
}
// The partial ratchet must be on disk despite the export error.
reloaded, err := quest.Load(session)
if err != nil {
t.Fatalf("reload session: %v", err)
}
a, err := reloaded.Find("a")
if err != nil {
t.Fatal(err)
}
if !a.Emitted {
t.Fatal("item a Emitted ratchet lost: session was not saved after the partial export")
}
// Re-export must emit only "b" — "a" is never re-emitted.
sink := &failAtSink{failAt: 0}
n2, err := exportAndSave(reloaded, sink, session)
if err != nil {
t.Fatalf("re-export: %v", err)
}
if n2 != 1 || len(sink.keys) != 1 || sink.keys[0] != "b" {
t.Fatalf("re-export n=%d keys=%v, want exactly [b] (no duplicate emission of a)", n2, sink.keys)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what exportAndSave — quest.Export 후 결과와 무관하게 세션을 먼저 Save해 Emit 성공분의 Emitted 래칫을 영속화한다(중간 Emit 실패 시 Save 없이 리턴하면 기방출 아이템이 다음 실행에서 중복 방출됨 — emit-once 보존). Export 에러가 Save 에러보다 우선 전파. evaluateAndApply와 export 명령이 공유.
package cli
import "github.com/park-jun-woo/reins/pkg/quest"
// exportAndSave runs quest.Export and then always saves the session, so the
// Emitted ratchet of any successfully emitted item is persisted even when a later
// Emit fails (otherwise the next run would re-emit it, breaking emit-once).
// Partial progress is a ratchet; preserving it is correct. The export error takes
// precedence over a save error. Shared by evaluateAndApply and the export command.
func exportAndSave(s *quest.Session, sink quest.Sink, sessionPath string) (int, error) {
n, exportErr := quest.Export(s, sink)
saveErr := s.Save(sessionPath)
if exportErr != nil {
return n, exportErr
}
return n, saveErr
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what export가 sink의 Emit 실패 시 quest.Export 에러를 표면화하는지 검증한다(out 경로가 디렉터리라 OpenFile 실패).
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestExportEmitError: export surfaces a quest.Export error when the sink's Emit
// fails. A terminal, not-yet-emitted item is seeded directly into the session, and
// the out path is an existing directory so newJSONLSink succeeds but Emit's OpenFile
// fails.
func TestExportEmitError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
s := &quest.Session{Version: 1, Items: []*quest.Item{{Key: "a", State: quest.PASS}}}
if err := s.Save(session); err != nil {
t.Fatalf("save: %v", err)
}
outDir := filepath.Join(dir, "outdir")
if err := os.Mkdir(outDir, 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}
runCmdErr(t, stubDef{}, session, outDir, "", "export")
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what export가 깨진 세션 파일의 load 에러를 표면화하는지 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
)
// TestExportLoadError: export surfaces a load error from a corrupt session file.
func TestExportLoadError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
if err := os.WriteFile(session, []byte("{bad"), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
runCmdErr(t, stubDef{}, session, out, "", "export")
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what export가 out 경로의 부모를 만들 수 없을 때(부모 슬롯을 일반 파일이 차지) sink 생성 에러를 표면화하는지 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
)
// TestExportSinkOpenError: export surfaces a sink-construction error when the out
// path's parent cannot be created (a regular file occupies a parent slot).
func TestExportSinkOpenError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
blocker := filepath.Join(dir, "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
out := filepath.Join(blocker, "sub", "out.jsonl")
runCmdErr(t, stubDef{}, session, out, "", "export")
}
//ff:func feature=cli type=helper control=sequence
//ff:what export가 종단 아이템을 JSONL로 쓰고 멱등인지(2회차 0건) 검증한다.
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
)
// TestExportTerminal: export writes terminal items to JSONL and is idempotent — a
// second export emits 0 new records.
func TestExportTerminal(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
runCmd(t, stubDef{}, session, out, "", "scan", "a")
runCmd(t, stubDef{}, session, out, "good", "submit", "--key", "a")
// submit already swept the PASS item, so a follow-up export adds nothing new.
got := runCmd(t, stubDef{}, session, out, "", "export")
if !strings.Contains(got, "exported 0 new record(s)") {
t.Fatalf("export = %q", got)
}
b, err := os.ReadFile(out)
if err != nil {
t.Fatalf("read out: %v", err)
}
if !strings.Contains(string(b), `"key":"a"`) {
t.Fatalf("out file = %q", b)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what `export` 명령. exportAndSave로 종단 아이템을 JSONL sink에 증분 방출(원본 보존)하고 — Emit이 중간에 실패해도 세션을 Save해 기방출 Emitted 래칫을 보존 — 새로 방출한 레코드 수를 알린다.
package cli
import (
"fmt"
"github.com/spf13/cobra"
)
// newExportCmd emits terminal items to the JSONL sink and saves the export ratchet.
func newExportCmd(sessionPath, outPath *string, load sessionLoader) *cobra.Command {
return &cobra.Command{
Use: "export",
Short: "export terminal results to JSONL (originals preserved)",
RunE: func(cmd *cobra.Command, args []string) error {
s, err := load()
if err != nil {
return err
}
sink, err := newJSONLSink(*outPath)
if err != nil {
return err
}
n, err := exportAndSave(s, sink, *sessionPath)
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "exported %d new record(s) to %s\n", n, *outPath)
return nil
},
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestGeneratePayloadBackendError — generatePayload 의 backend-error 강등 경로. Complete 에러 시 backendErrorVerdict 로 합성→applyVerdict 로 래칫(Tries++)·영속화 후 handled=true(호출부가 루프 continue)·err=nil·raw="" 임을 단언. raw 가 비고 handled 가 참인 분기 시맨틱과 합성 FAIL 의 backend-error 어트리뷰션을 관측.
package cli
import (
"io"
"path/filepath"
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestGeneratePayloadBackendError: a backend.Complete error is demoted, not
// propagated. applyVerdict ratchets the synthetic backend-error verdict (Tries++),
// generatePayload returns handled=true (so the caller continues the loop) with err=nil
// and empty raw. The demoted FAIL is observable under the reserved backend-error rule
// carrying the original error text.
func TestGeneratePayloadBackendError(t *testing.T) {
dir := t.TempDir()
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
backend := llm.CallFunc(func(system, user string) (string, error) {
return "ignored-on-error", errBackend
})
raw, handled, err := generatePayload(stubDef{}, &LoopOptions{}, backend, "", "",
s, it, filepath.Join(dir, "out.jsonl"), filepath.Join(dir, "session.json"), io.Discard)
if err != nil {
t.Fatalf("err = %v, want nil (generation error is demoted, not propagated)", err)
}
if !handled {
t.Fatal("handled = false, want true (caller must continue, not gate-evaluate)")
}
if raw != "" {
t.Fatalf("raw = %q, want empty on the demotion path", raw)
}
// The demotion ratcheted one try.
if it.Tries != 1 {
t.Fatalf("Tries = %d, want 1 (applyVerdict ratchets the demoted FAIL)", it.Tries)
}
// Observability: the synthetic FAIL surfaces under "backend-error" with the
// original error text.
if len(it.Log) != 1 {
t.Fatalf("Log has %d attempts, want 1", len(it.Log))
}
last := it.Log[len(it.Log)-1]
if !strings.Contains(last.Reason, backendErrorRule) {
t.Fatalf("attempt reason = %q, want it to mention %q", last.Reason, backendErrorRule)
}
if !strings.Contains(last.Reason, errBackend.Error()) {
t.Fatalf("attempt reason = %q, want it to carry the original error %q", last.Reason, errBackend.Error())
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestGeneratePayloadRenderError — generatePayload 의 def.Render 에러 fail-fast 분기. Render 가 에러나면 backend.Complete 호출 전에 그 에러를 그대로 전파(handled=false·raw="")함을 render-에러 def 더블로 단언. 강등은 backend 인프라 에러 한정이고 프롬프트 렌더 실패는 즉시 중단임을 경계짓는다.
package cli
import (
"io"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestGeneratePayloadRenderError: a def.Render error propagates immediately —
// generatePayload returns that error (handled=false, raw="") without ever calling
// backend.Complete. Demotion is reserved for backend infra errors; a prompt-render
// failure is a hard fail-fast.
func TestGeneratePayloadRenderError(t *testing.T) {
dir := t.TempDir()
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
called := false
backend := llm.CallFunc(func(system, user string) (string, error) {
called = true
return "should-not-run", nil
})
raw, handled, err := generatePayload(errDef{renderErr: true}, &LoopOptions{}, backend, "", "",
s, it, filepath.Join(dir, "out.jsonl"), filepath.Join(dir, "session.json"), io.Discard)
if err == nil {
t.Fatal("err = nil, want def.Render error to propagate")
}
if handled {
t.Fatal("handled = true, want false (render error is propagated, not demoted)")
}
if raw != "" {
t.Fatalf("raw = %q, want empty", raw)
}
if called {
t.Fatal("backend.Complete was called, want it skipped after a Render error")
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestGeneratePayloadSaveError — generatePayload 의 강등 경로 fail-fast 경계. backend-error 강등 verdict 의 영속화(applyVerdict 의 Save) 실패는 강등에 묻히지 않고 err 로 전파됨(정합성 위협 침묵 금지)을 쓰기 불가 세션 경로로 단언. raw="" 이고 handled=false(전파는 handled 신호가 아님).
package cli
import (
"io"
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestGeneratePayloadSaveError: on the backend-error demotion path, if persisting the
// demoted verdict fails (unwritable session path), generatePayload returns that error
// rather than swallowing it — persistence failures stay fatal even when the generation
// error itself is demoted. raw is empty and handled is false (propagation is not a
// continue signal).
func TestGeneratePayloadSaveError(t *testing.T) {
dir := t.TempDir()
out := filepath.Join(dir, "out.jsonl")
// A session path nested under a regular file (not a dir) makes Save fail.
badParent := filepath.Join(dir, "afile")
if err := os.WriteFile(badParent, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
session := filepath.Join(badParent, "nested", "session.json")
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
backend := llm.CallFunc(func(system, user string) (string, error) { return "", errBackend })
raw, handled, err := generatePayload(stubDef{}, &LoopOptions{}, backend, "", "",
s, it, out, session, io.Discard)
if err == nil {
t.Fatal("err = nil, want save error to propagate on the backend-error path")
}
if handled {
t.Fatal("handled = true, want false (a propagated error is not a continue signal)")
}
if raw != "" {
t.Fatalf("raw = %q, want empty", raw)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestGeneratePayload — generatePayload 정상 생성 경로. backend.Complete 가 정상 payload 를 돌려주면 그 출력을 raw 로 그대로 반환하고 handled=false(호출부가 계속 게이트 평가하라는 신호)·err=nil 임을 단언. composeSystem(System,ruleCoach)·def.Render→Complete 가 호출돼 system/prompt 가 backend 로 전달됨도 관측.
package cli
import (
"io"
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestGeneratePayload: when backend.Complete succeeds, generatePayload returns the
// backend's output verbatim as raw, with handled=false (the caller should proceed to
// gate evaluation) and err=nil. The composed system + rendered prompt+feedback reach
// the backend.
func TestGeneratePayload(t *testing.T) {
dir := t.TempDir()
s := quest.New()
it := &quest.Item{Key: "a", State: quest.TODO}
s.Items = append(s.Items, it)
var gotSystem, gotUser string
backend := llm.CallFunc(func(system, user string) (string, error) {
gotSystem, gotUser = system, user
return "PAYLOAD-OK", nil
})
opts := &LoopOptions{System: "SYS"}
raw, handled, err := generatePayload(stubDef{}, opts, backend, "COACH", "FEEDBACK",
s, it, dir+"/out.jsonl", dir+"/session.json", io.Discard)
if err != nil {
t.Fatalf("err = %v, want nil", err)
}
if handled {
t.Fatal("handled = true, want false (success defers to caller's gate evaluation)")
}
if raw != "PAYLOAD-OK" {
t.Fatalf("raw = %q, want backend output %q", raw, "PAYLOAD-OK")
}
// The composed system carries opts.System and the rule coach.
if !strings.Contains(gotSystem, "SYS") || !strings.Contains(gotSystem, "COACH") {
t.Fatalf("system = %q, want it to compose opts.System and ruleCoach", gotSystem)
}
// The user prompt is def.Render output with feedback appended.
if !strings.Contains(gotUser, "render:a") || !strings.Contains(gotUser, "FEEDBACK") {
t.Fatalf("user = %q, want rendered prompt + feedback", gotUser)
}
// No demotion occurred: the item stays TODO with no try consumed.
if it.Tries != 0 || it.State != quest.TODO {
t.Fatalf("item mutated on success: Tries=%d State=%q, want 0/TODO", it.Tries, it.State)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what generatePayload — runLoopItem 한 시도의 L0 생성 단계. composeSystem(System,ruleCoach)으로 system을 짜고 def.Render→backend.Complete로 페이로드를 만든다. Complete 에러는 Phase012 강등: backendErrorVerdict로 합성→applyVerdict로 래칫(Tries++, MaxTries에서 DONE 잠금)→renderVerdict 출력 후 handled=true로 반환해 호출부가 루프를 continue하게 한다(루프 abort 아님). def.Render·applyVerdict(영속화) 에러만 err로 전파해 fail-fast. 인프라 에러는 내용 비평이 아니므로 피드백을 되먹이지 않는다(호출부가 피드백 비갱신).
package cli
import (
"io"
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// generatePayload runs one attempt's L0 generation: it composes the system prompt,
// renders the item, and calls backend.Complete. A def.Render error propagates (err).
// A backend.Complete error is demoted (Phase012): a synthetic backend-error verdict
// is ratcheted (Tries++, locking DONE at MaxTries) and rendered, then handled=true is
// returned so the caller continues the loop — generation failure is a retryable item
// FAIL, not a run abort. A persistence failure inside applyVerdict is still fatal
// (err). No feedback is fed back for an infra error.
func generatePayload(def gate.Definition, opts *LoopOptions, backend llm.Backend, ruleCoach, feedback string, s *quest.Session, it *quest.Item, outPath, sessionPath string, out io.Writer) (raw string, handled bool, err error) {
system := composeSystem(opts.System, ruleCoach)
prompt, err := def.Render(s, it)
if err != nil {
return "", false, err
}
raw, err = backend.Complete(system, prompt+feedback)
if err == nil {
return raw, false, nil
}
verdict := backendErrorVerdict(err)
if aerr := applyVerdict(s, it, verdict, outPath, sessionPath); aerr != nil {
return "", false, aerr
}
renderVerdict(out, it.Key, it, verdict)
return "", true, nil
}
//ff:func feature=cli type=helper control=sequence
//ff:what graphDef.badRule. 제출물 텍스트가 "bad"면 발동(FAIL)하는 공유 카운터를 돌려준다(테스트 더블).
package cli
import (
"strings"
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/quest"
)
// badRule is the shared counter: it fires (FAIL) when the submission text is "bad".
func (graphDef) badRule() gate.Rule {
return gate.Rule{
Meta: gate.RuleMeta{ID: "not-bad", Level: gate.LevelFail, Desc: "submission must not be bad"},
Check: func(ctx gate.Context) (bool, quest.Fact) {
if s, _ := ctx.Submission.(string); strings.TrimSpace(s) == "bad" {
return true, quest.Fact{Where: "body", Expected: "good", Actual: "bad"}
}
return false, quest.Fact{}
},
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what graphDef.Evaluate. defeat 그래프에서 verdict를 읽는다(Evaluator 경로). graph.FromRules가 edge-zero 등가 그래프(FAIL 카운터에 공격받는 tautology PASS warrant 하나)를 만들어 submit의 Evaluator 분기가 실제 pkg/graph로 작동함을 검증한다(테스트 더블).
package cli
import (
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/graph"
"github.com/park-jun-woo/reins/pkg/quest"
)
// Evaluate reads the verdict from a defeat graph (Evaluator path). graph.FromRules
// builds the edge-zero equivalent (one tautology PASS warrant attacked by the FAIL
// counter), so submit's Evaluator branch is exercised against the real pkg/graph.
func (d graphDef) Evaluate(ctx gate.Context) quest.Verdict {
return graph.FromRules(d.Rules()).Evaluate(ctx)
}
//ff:func feature=cli type=helper control=sequence
//ff:what graphDef.Prepare. raw 바이트를 문자열 Submission으로 담은 gate.Context를 만든다(테스트 더블).
package cli
import (
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/quest"
)
func (graphDef) Prepare(_ *quest.Session, it *quest.Item, raw []byte) (gate.Context, *quest.Verdict, error) {
return gate.Context{Item: it, Submission: string(raw)}, nil, nil
}
//ff:func feature=cli type=helper control=sequence
//ff:what graphDef.Render. 아이템 키 앞에 "render:"를 붙여 렌더 문자열을 돌려준다(테스트 더블).
package cli
import "github.com/park-jun-woo/reins/pkg/quest"
func (graphDef) Render(_ *quest.Session, it *quest.Item) (string, error) {
return "render:" + it.Key, nil
}
//ff:func feature=cli type=helper control=sequence
//ff:what graphDef.Rules. badRule 카운터 하나로 된 카탈로그를 돌려준다(rules 명령·후방호환 감사용 테스트 더블).
package cli
import "github.com/park-jun-woo/reins/pkg/gate"
func (d graphDef) Rules() []gate.Rule { return []gate.Rule{d.badRule()} }
//ff:func feature=cli type=helper control=iteration dimension=1
//ff:what graphDef.Seed. args 하나당 TODO 아이템을 시드한다(테스트 더블).
package cli
import "github.com/park-jun-woo/reins/pkg/quest"
func (d graphDef) Seed(args []string) ([]*quest.Item, error) {
items := make([]*quest.Item, len(args))
for i, a := range args {
items[i] = &quest.Item{Key: a, State: quest.TODO}
}
return items, nil
}
//ff:type feature=cli type=model
//ff:what 테스트용 그래프형 gate.Definition. gate.Evaluator를 구현(Evaluate가 pkg/graph 그래프로 판독)해 submit 배선이 Rules() 대신 Evaluator 경로를 타는지 검증한다. Rules()도 카탈로그를 제공(rules 명령·후방호환 감사)하나 submit은 Evaluate를 쓴다. 제출물 "bad"면 FAIL 카운터 발동. 테스트 바이너리가 toulmin을 링크하는 건 go.work가 해결(소스 의존 아님).
package cli
// graphDef is a graph-backed Definition: it implements gate.Evaluator so submit
// reads its Verdict from a pkg/graph defeat graph instead of the Rules() catalog.
// Rules() still returns the catalog (for the `rules` command and back-compat audit),
// but submit uses Evaluate. The single FAIL counter fires when the submission is
// "bad".
type graphDef struct{}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what Emit이 파일을 열 수 없을 때(경로가 생성 후 디렉터리로 점유됨) 에러를 내는지 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEmitOpenError: Emit returns an error when its file cannot be opened (the path
// was usurped by a directory after construction).
func TestEmitOpenError(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "occupied")
if err := os.Mkdir(path, 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}
sink := &jsonlSink{path: path}
if err := sink.Emit(&quest.Item{Key: "a", State: quest.PASS}); err == nil {
t.Fatal("Emit to a directory path: want error, got nil")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what Emit이 아이템 하나당 JSON 한 줄(개행 종단)을 append하는지 검증한다.
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestEmitAppendsJSONL: Emit appends one JSON line per item (newline-terminated).
func TestEmitAppendsJSONL(t *testing.T) {
path := filepath.Join(t.TempDir(), "out.jsonl")
sink, err := newJSONLSink(path)
if err != nil {
t.Fatalf("newJSONLSink: %v", err)
}
if err := sink.Emit(&quest.Item{Key: "a", State: quest.PASS}); err != nil {
t.Fatalf("Emit a: %v", err)
}
if err := sink.Emit(&quest.Item{Key: "b", State: quest.DONE}); err != nil {
t.Fatalf("Emit b: %v", err)
}
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read: %v", err)
}
lines := strings.Split(strings.TrimRight(string(b), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("got %d lines, want 2: %q", len(lines), b)
}
if !strings.Contains(lines[0], `"key":"a"`) || !strings.Contains(lines[1], `"key":"b"`) {
t.Fatalf("lines = %q", lines)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what jsonlSink.Emit. 아이템 하나를 JSON으로 직렬화해 개행을 붙여 sink 파일에 append한다(파일이 없으면 생성).
package cli
import (
"encoding/json"
"os"
"github.com/park-jun-woo/reins/pkg/quest"
)
// Emit appends one JSON-encoded item, newline-terminated, to the sink's file.
func (s *jsonlSink) Emit(it *quest.Item) error {
f, err := os.OpenFile(s.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
if err != nil {
return err
}
defer f.Close()
b, err := json.Marshal(it)
if err != nil {
return err
}
b = append(b, '\n')
_, err = f.Write(b)
return err
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what loadSession이 깨진(부재 아님) 세션 파일에서 load 에러를 표면화하는지(빈 세션으로 가리지 않는지) 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
)
// TestLoadSessionBadJSON: a corrupt (non-missing) session file surfaces the load
// error rather than masking it as a fresh session.
func TestLoadSessionBadJSON(t *testing.T) {
path := filepath.Join(t.TempDir(), "broken.json")
if err := os.WriteFile(path, []byte("{bad"), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := loadSession(path); err == nil {
t.Fatal("loadSession broken JSON: want error, got nil")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what TestLoadSessionClearsLoop — MetaLoop 플래그가 박힌 세션 파일(kill된 loop 프로세스의 잔류)을 loadSession이 로드 직후 자가 치유(플래그 삭제)하는지, 다른 Meta 키는 보존하는지 검증한다.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestLoadSessionClearsLoop: a session file persisted with the MetaLoop
// flag set (residue of a killed loop process) is self-healed on load — the flag is
// gone, while other Meta keys survive.
func TestLoadSessionClearsLoop(t *testing.T) {
path := filepath.Join(t.TempDir(), "session.json")
stale := quest.New()
stale.SetMeta(quest.MetaLoop, true)
stale.SetMeta("keep", "me")
if err := stale.Save(path); err != nil {
t.Fatalf("save: %v", err)
}
s, err := loadSession(path)
if err != nil {
t.Fatalf("loadSession: %v", err)
}
if _, ok := s.GetMeta(quest.MetaLoop); ok {
t.Fatal("MetaLoop survived load; want it cleared (self-heal)")
}
if v, ok := s.GetMeta("keep"); !ok || v != "me" {
t.Fatalf("other Meta key lost: keep=%v ok=%v", v, ok)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what loadSession이 기존 유효 세션 파일을 그대로 로드하는지 검증한다.
package cli
import (
"path/filepath"
"testing"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestLoadSessionExisting: an existing valid session file is loaded as-is.
func TestLoadSessionExisting(t *testing.T) {
path := filepath.Join(t.TempDir(), "session.json")
want := &quest.Session{Version: 1, Items: []*quest.Item{{Key: "a", State: quest.TODO}}}
if err := want.Save(path); err != nil {
t.Fatalf("save: %v", err)
}
s, err := loadSession(path)
if err != nil {
t.Fatalf("loadSession: %v", err)
}
if len(s.Items) != 1 || s.Items[0].Key != "a" {
t.Fatalf("loadSession = %+v, want loaded items", s)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what loadSession이 부재 세션 파일에서 에러 대신 빈 새 세션을 내는지 검증한다(첫 scan을 위해).
package cli
import (
"path/filepath"
"testing"
)
// TestLoadSessionAbsentFresh: a missing session file yields a fresh empty session
// rather than an error, so the first scan can start from nothing.
func TestLoadSessionAbsentFresh(t *testing.T) {
s, err := loadSession(filepath.Join(t.TempDir(), "absent.json"))
if err != nil {
t.Fatalf("loadSession absent: %v", err)
}
if s == nil || s.Version != 1 || len(s.Items) != 0 {
t.Fatalf("loadSession absent = %+v, want fresh quest.New()", s)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what sessionPath의 세션을 로드한다. 파일이 없으면 빈 세션(quest.New())을 새로 만들어 반환한다 — 첫 scan을 위해 부재를 에러로 보지 않는다. 로드 직후 MetaLoop를 무조건 삭제 — in-process 전용 신호라 loop 프로세스가 kill돼 플래그가 박힌 채 남아도 다음 프로세스가 자가 치유한다(Render의 실패 로그 tail 영구 억제 방지).
package cli
import (
"os"
"github.com/park-jun-woo/reins/pkg/quest"
)
// loadSession returns the session at path, creating a fresh one if absent. It
// unconditionally clears the MetaLoop flag right after loading: the flag is an
// in-process-only signal, so any residue left by a killed loop process is
// self-healed by the next process (otherwise Render would suppress its failure
// log-tail forever).
func loadSession(path string) (*quest.Session, error) {
s, err := quest.Load(path)
if os.IsNotExist(err) {
return quest.New(), nil
}
if err != nil {
return nil, err
}
delete(s.Meta, quest.MetaLoop)
return s, nil
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopBackendErrorContinues — BUG-002 정면 회귀. 1번 아이템 backend 가 항상 에러여도 그 실패가 2번 아이템을 막지 않음(1번은 MaxTries 후 DONE 잠금, 2번은 게이트 PASS)을 loop 명령 끝까지로 단언.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestLoopBackendErrorContinues: with two TODO items, item "a"'s backend always
// errors while item "b"'s succeeds. The run must not abort on "a" — it locks "a"
// DONE (residual) after MaxTries and still drives "b" through the gate to PASS. The
// backend keys off the rendered prompt (stubDef renders "render:<key>").
func TestLoopBackendErrorContinues(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
backend := llm.CallFunc(func(system, user string) (string, error) {
if strings.Contains(user, "render:a") {
return "", errBackend // item "a" generation always fails
}
return "good", nil // item "b" converges
})
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a", "b"); err != nil {
t.Fatalf("scan: %v", err)
}
stdout, err := newLoopRoot(t, opts, session, out, "loop")
if err != nil {
t.Fatalf("loop = %v, want nil (item 'a' failure must not abort the run)", err)
}
if !strings.Contains(stdout, "processed 2 item(s)") {
t.Fatalf("loop output = %q, want both items processed", stdout)
}
s, err := loadSession(session)
if err != nil {
t.Fatalf("load: %v", err)
}
a, err := s.Find("a")
if err != nil {
t.Fatalf("find a: %v", err)
}
if a.State != quest.DONE || a.Tries != quest.MaxTries {
t.Fatalf("item a = %+v, want DONE residual at MaxTries", a)
}
b, err := s.Find("b")
if err != nil {
t.Fatalf("find b: %v", err)
}
if b.State != quest.PASS {
t.Fatalf("item b = %+v, want PASS (unblocked by a's failure)", b)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopBackendError — backend 에러가 루프를 abort하지 않고 아이템 FAIL로 강등되어 런이 nil로 완주하는지 검증(BUG-002 회귀의 정면 단언).
package cli
import (
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopBackendError: a backend error no longer aborts the loop — it is demoted to
// a retryable item FAIL, so the run completes without error (BUG-002 regression).
func TestLoopBackendError(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
backend := llm.CallFunc(func(system, user string) (string, error) {
return "", errBackend
})
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := newLoopRoot(t, opts, session, out, "loop"); err != nil {
t.Fatalf("loop = %v, want nil (backend error is demoted, not propagated)", err)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopFirstGenPassLocks — 통과 페이로드를 반환하는 backend가 아이템을 한 번에 PASS로 잠그는지(backend 1회 호출) 무네트워크 검증.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopFirstGenPassLocks: a backend that returns a passing payload locks the
// item PASS in one shot.
func TestLoopFirstGenPassLocks(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
calls := 0
backend := llm.CallFunc(func(system, user string) (string, error) {
calls++
return "good", nil
})
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
got, err := newLoopRoot(t, opts, session, out, "loop")
if err != nil {
t.Fatalf("loop: %v", err)
}
if !strings.Contains(got, "a -> PASS") {
t.Fatalf("loop output = %q, want PASS", got)
}
if calls != 1 {
t.Fatalf("backend called %d times, want 1 (PASS in one shot)", calls)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopGateOnlyLocks — backend가 무엇을 뱉든 게이트가 실패시키면 PASS에 닿지 않고 MaxTries 소진 후 DONE으로 종료(잠금 권한은 게이트만)하는지 검증.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestLoopGateOnlyLocks: no matter what the backend emits, a submission the gate
// fails never reaches PASS; MaxTries is exhausted and the item is DONE (loop ends).
func TestLoopGateOnlyLocks(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
calls := 0
backend := llm.CallFunc(func(system, user string) (string, error) {
calls++
return "bad", nil // always fails the gate
})
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
got, err := newLoopRoot(t, opts, session, out, "loop")
if err != nil {
t.Fatalf("loop: %v", err)
}
if calls != quest.MaxTries {
t.Fatalf("backend called %d times, want MaxTries=%d", calls, quest.MaxTries)
}
if strings.Contains(got, "a -> PASS") {
t.Fatalf("loop must not PASS a gate-failing submission: %q", got)
}
// The item must be terminal (DONE) so NextTODO drops it (monotone convergence).
if !strings.Contains(got, "processed 1 item") {
t.Fatalf("loop output = %q, want 'processed 1 item'", got)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopMaxItems — --max-items가 처리 아이템 수를 제한하는지(3개 시드, --max-items 1 → "processed 1 item") 검증.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopMaxItems: --max-items caps the number of items processed.
func TestLoopMaxItems(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
backend := llm.CallFunc(func(system, user string) (string, error) { return "good", nil })
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a", "b", "c"); err != nil {
t.Fatalf("scan: %v", err)
}
got, err := newLoopRoot(t, opts, session, out, "loop", "--max-items", "1")
if err != nil {
t.Fatalf("loop: %v", err)
}
if !strings.Contains(got, "processed 1 item") {
t.Fatalf("loop output = %q, want 'processed 1 item' with --max-items 1", got)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopNoInjectedLLMBadModel — 주입 backend가 없으면 --model을 llm.FromFlag로 해석하고, 잘못된 model flag는 루프 전에 에러내는지 검증.
package cli
import (
"testing"
)
// TestLoopNoInjectedLLMBadModel: with no injected backend the loop resolves
// --model via llm.FromFlag; an invalid model flag errors before the loop.
func TestLoopNoInjectedLLMBadModel(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
opts := Options{Loop: &LoopOptions{}} // LLM nil ⇒ FromFlag path
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
// "nocolon" has no backend:model form ⇒ FromFlag returns an error.
if _, err := newLoopRoot(t, opts, session, out, "loop", "--model", "nocolon"); err == nil {
t.Fatal("loop = nil error, want FromFlag error for bad --model")
}
}
//ff:type feature=cli type=model
//ff:what LoopOptions — in-process generate→gate→retry 에이전트 루프 설정. LLM은 생성자(L0)만 — PASS 잠금 권한은 게이트에. DefaultModel(--model 기본)·System(전역 system 프롬프트)·RuleSystem(rule ID→FAIL 시 추가 코칭)·LLM(주입 backend, 비-nil이면 --model 무시).
package cli
import (
"github.com/park-jun-woo/reins/pkg/llm"
)
// LoopOptions configures the in-process generate→gate→retry loop. The LLM is
// the generator (L0) only — PASS lock authority stays with the gate.
type LoopOptions struct {
// DefaultModel is the --model default. Empty ⇒ "ollama:gemma4:e4b".
DefaultModel string
// System is the global system prompt.
System string
// RuleSystem maps a toulmin rule ID (verdict.RootCause) to extra system
// guidance appended when the previous attempt FAILed on that rule.
RuleSystem map[string]string
// LLM, when non-nil, is used as the backend and --model is ignored (for tests
// or a fixed backend).
LLM llm.Backend
// Escalate, when non-nil, is a stronger fallback backend. Once an item's FAIL
// carries a RootCause listed in EscalateOn (a capability-bound signal — a
// semantic mismatch, not a format slip), the item is retried with Escalate for
// its remaining tries (latched on for that item). The gate still holds sole PASS
// authority — escalation only changes which generator (L0) is asked. nil ⇒ no
// escalation (backward compatible). Cost note: a slow/paid backend here is
// invoked only on the residual the primary cannot crack.
Escalate llm.Backend
// EscalateOn is the set of FAIL RootCause IDs that promote an item to Escalate.
// Empty ⇒ never escalate (even when Escalate is set), so format/shape failures
// the consumer leaves out stay on the cheap primary.
EscalateOn []string
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopRenderError — 루프 안 def.Render 에러가 그 에러로 중단시키는지 검증(정상 def로 시드 후 render-에러 def로 loop 실행).
package cli
import (
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopRenderError: a def.Render error inside the loop aborts with that error.
func TestLoopRenderError(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
backend := llm.CallFunc(func(string, string) (string, error) { return "good", nil })
opts := Options{Loop: &LoopOptions{LLM: backend}}
// Seed with a non-erroring def first so the item exists, then run the loop with
// a render-erroring def over the same session.
if _, err := newLoopRootDef(t, stubDef{}, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := newLoopRootDef(t, errDef{renderErr: true}, opts, session, out, "loop"); err == nil {
t.Fatal("loop = nil error, want Render error")
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopRetryConverges — 첫 시도 FAIL 후 재시도 PASS로 루프가 수렴하는지(backend 2회 호출, FAIL·PASS 출력) 검증.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopRetryConverges: FAIL then PASS — the loop retries and converges.
func TestLoopRetryConverges(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
calls := 0
backend := llm.CallFunc(func(system, user string) (string, error) {
calls++
if calls == 1 {
return "bad", nil // first attempt fails the gate
}
return "good", nil // retry passes
})
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
got, err := newLoopRoot(t, opts, session, out, "loop")
if err != nil {
t.Fatalf("loop: %v", err)
}
if calls != 2 {
t.Fatalf("backend called %d times, want 2 (FAIL then PASS)", calls)
}
if !strings.Contains(got, "a -> FAIL") || !strings.Contains(got, "a -> PASS") {
t.Fatalf("loop output = %q, want a FAIL then a PASS", got)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestLoopRuleCoaching — FAIL 시 RootCause로 매핑된 RuleSystem 코칭이 다음 시도의 system 프롬프트에 합성되는지(첫 system엔 없고 재시도 system엔 있음) 검증.
package cli
import (
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
)
// TestLoopRuleCoaching: on FAIL the RootCause-mapped RuleSystem coaching is
// composed into the next attempt's system prompt.
func TestLoopRuleCoaching(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
const coach = "COACH-FOR-NOT-BAD"
var systems []string
calls := 0
backend := llm.CallFunc(func(system, user string) (string, error) {
systems = append(systems, system)
calls++
if calls == 1 {
return "bad", nil
}
return "good", nil
})
// stubDef's FAIL rule has ID "not-bad" — gate.Evaluate sets RootCause to it.
opts := Options{Loop: &LoopOptions{
LLM: backend,
RuleSystem: map[string]string{"not-bad": coach},
}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a"); err != nil {
t.Fatalf("scan: %v", err)
}
if _, err := newLoopRoot(t, opts, session, out, "loop"); err != nil {
t.Fatalf("loop: %v", err)
}
if len(systems) < 2 {
t.Fatalf("expected >=2 attempts, got %d", len(systems))
}
if strings.Contains(systems[0], coach) {
t.Fatalf("first attempt system should not yet carry coaching: %q", systems[0])
}
if !strings.Contains(systems[1], coach) {
t.Fatalf("retry system missing rule coaching: %q", systems[1])
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what TestNewLoopCmd — newLoopCmd가 loop 명령을 만들고 flag 기본값(--model: DefaultModel 미설정 ⇒ ollama:gemma4:e4b, 설정 ⇒ 그 값 / --max-items 0=전부)을 노출하며, backend 해석(주입 LLM 우선, 없으면 --model을 FromFlag로 — 잘못된 flag는 루프 전에 에러), load 실패 전파, --max-items 상한, backend 에러 강등(런 완주), defer Save 실패의 stderr 경고 표면화를 검증한다.
package cli
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
)
// TestNewLoopCmd: newLoopCmd builds the `loop` command with the documented flag
// defaults (--model falls back to ollama:gemma4:e4b unless opts.DefaultModel is
// set; --max-items defaults to 0 = all) and its RunE resolves the backend
// (injected LLM wins, otherwise --model via llm.FromFlag — a bad flag errors
// before the loop), propagates load failures, caps work at --max-items, demotes
// a backend error to an item FAIL (the run still completes), and surfaces a
// deferred Save failure as a stderr warning.
func TestNewLoopCmd(t *testing.T) {
dir := t.TempDir()
session := dir + "/session.json"
out := dir + "/out.jsonl"
// runLoop executes a standalone loop command built by newLoopCmd against sess.
runLoop := func(opts *LoopOptions, sess string, args ...string) (string, string, error) {
c := newLoopCmd(stubDef{}, opts, &sess, &out, func() (*quest.Session, error) { return loadSession(sess) })
var stdout, stderr bytes.Buffer
c.SetOut(&stdout)
c.SetErr(&stderr)
c.SetArgs(append([]string{}, args...))
err := c.Execute()
return stdout.String(), stderr.String(), err
}
// Flag defaults: --model falls back to the package default and --max-items to 0…
cmd := newLoopCmd(stubDef{}, &LoopOptions{}, &session, &out, func() (*quest.Session, error) { return loadSession(session) })
if got := cmd.Flags().Lookup("model").DefValue; got != defaultLoopModel {
t.Fatalf("model default = %q, want %q", got, defaultLoopModel)
}
if got := cmd.Flags().Lookup("max-items").DefValue; got != "0" {
t.Fatalf("max-items default = %q, want %q", got, "0")
}
// …and opts.DefaultModel overrides the --model default.
custom := newLoopCmd(stubDef{}, &LoopOptions{DefaultModel: "stub:model"}, &session, &out, func() (*quest.Session, error) { return loadSession(session) })
if got := custom.Flags().Lookup("model").DefValue; got != "stub:model" {
t.Fatalf("model default with DefaultModel = %q, want %q", got, "stub:model")
}
// With no injected backend a malformed --model errors before the loop runs.
if _, _, err := runLoop(&LoopOptions{}, session, "--model", "nocolon"); err == nil {
t.Fatal("loop = nil error, want FromFlag error for bad --model")
}
// A valid --model resolves via FromFlag, then a corrupt session fails load.
badSession := filepath.Join(dir, "bad.json")
if err := os.WriteFile(badSession, []byte("{not json"), 0o644); err != nil {
t.Fatalf("write bad session: %v", err)
}
if _, _, err := runLoop(&LoopOptions{}, badSession); err == nil {
t.Fatal("loop = nil error, want load error for corrupt session")
}
// Injected backend (--model ignored): --max-items 1 caps a 2-item session…
backend := llm.CallFunc(func(system, user string) (string, error) { return "good", nil })
opts := Options{Loop: &LoopOptions{LLM: backend}}
if _, err := newLoopRoot(t, opts, session, out, "scan", "a", "b"); err != nil {
t.Fatalf("scan: %v", err)
}
stdout, _, err := runLoop(opts.Loop, session, "--max-items", "1")
if err != nil {
t.Fatalf("loop --max-items 1: %v", err)
}
if !strings.Contains(stdout, "processed 1 item(s)") {
t.Fatalf("loop output = %q, want 'processed 1 item(s)'", stdout)
}
// …and a second uncapped run drains the remaining TODO through the gate path.
stdout, _, err = runLoop(opts.Loop, session)
if err != nil {
t.Fatalf("loop: %v", err)
}
if !strings.Contains(stdout, "processed 1 item(s)") {
t.Fatalf("second loop output = %q, want 'processed 1 item(s)'", stdout)
}
// A backend error is demoted to an item FAIL, so the loop completes without error.
failSession := filepath.Join(dir, "fail.json")
if _, err := newLoopRoot(t, opts, failSession, out, "scan", "x"); err != nil {
t.Fatalf("scan fail session: %v", err)
}
failing := &LoopOptions{LLM: llm.CallFunc(func(system, user string) (string, error) { return "", errBackend })}
if _, _, err := runLoop(failing, failSession); err != nil {
t.Fatalf("loop = %v, want nil (backend error is demoted, not propagated)", err)
}
// A deferred Save failure is surfaced as a stderr warning, not a hard error.
roDir := filepath.Join(dir, "ro")
if err := os.Mkdir(roDir, 0o555); err != nil {
t.Fatalf("mkdir ro: %v", err)
}
t.Cleanup(func() { os.Chmod(roDir, 0o755) })
_, stderr, err := runLoop(opts.Loop, filepath.Join(roDir, "session.json"))
if err != nil {
t.Fatalf("loop on read-only dir: %v", err)
}
if !strings.Contains(stderr, "warning: save session after loop") {
t.Fatalf("stderr = %q, want save warning", stderr)
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what `loop [--model ...] [--max-items N]` 명령. submit의 자동 반복 — 남은 TODO를 NextTODO로 순회하며 LLM 생성(L0)→evaluateAndApply(게이트 판정·래칫·export, submit과 동일 경로)→FAIL이면 renderVerdictText 피드백을 user에, RuleSystem[verdict.RootCause] 코칭을 system에 되먹여 재시도(it.Tries<MaxTries). PASS/REVIEW/SKIP/BLOCK은 잠금→다음 아이템. backend는 opts.LLM!=nil이면 그걸, 아니면 --model을 llm.FromFlag로 lazy 생성. 종료 보장: MaxTries 초과 시 Apply가 DONE으로 잠가 NextTODO에서 빠짐(단조 수렴). PASS 잠금 권한은 게이트에만. defer Save 실패는 stderr 경고로 표면화(에러 침묵 금지).
package cli
import (
"fmt"
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/llm"
"github.com/park-jun-woo/reins/pkg/quest"
"github.com/spf13/cobra"
)
// defaultLoopModel is the --model fallback when neither opts.DefaultModel nor the
// flag is set.
const defaultLoopModel = "ollama:gemma4:e4b"
// newLoopCmd builds the `loop` command: an automatic submit loop that lets the LLM
// generate each remaining TODO's payload, runs it through the same gate path as
// submit, and feeds FAIL feedback (plus rule-specific system coaching) back on retry.
func newLoopCmd(def gate.Definition, opts *LoopOptions, sessionPath, outPath *string, load sessionLoader) *cobra.Command {
defaultModel := opts.DefaultModel
if defaultModel == "" {
defaultModel = defaultLoopModel
}
var (
model string
maxItems int
)
cmd := &cobra.Command{
Use: "loop [--model backend:model] [--max-items N]",
Short: "auto-run the generate→gate→retry loop over remaining TODO items",
RunE: func(cmd *cobra.Command, args []string) error {
backend := opts.LLM
if backend == nil {
b, err := llm.FromFlag(model)
if err != nil {
return err
}
backend = b
}
s, err := load()
if err != nil {
return err
}
// Signal Definition.Render to suppress its own last-failure log-tail
// while the loop runs (the loop appends renderVerdict feedback itself,
// avoiding double exposure). Cleared after the loop so a later manual
// next/submit shows the tail again.
s.SetMeta(quest.MetaLoop, true)
defer func() {
delete(s.Meta, quest.MetaLoop)
if err := s.Save(*sessionPath); err != nil {
fmt.Fprintf(cmd.ErrOrStderr(), "warning: save session after loop: %v\n", err)
}
}()
out := cmd.OutOrStdout()
done := 0
for it := s.NextTODO(); it != nil; it = s.NextTODO() {
if maxItems > 0 && done >= maxItems {
break
}
if err := runLoopItem(def, opts, backend, s, it, *outPath, *sessionPath, out); err != nil {
return err
}
done++
}
fmt.Fprintf(out, "loop: processed %d item(s)\n", done)
return nil
},
}
cmd.Flags().StringVar(&model, "model", defaultModel, "LLM backend:model (ignored if a backend is injected)")
cmd.Flags().IntVar(&maxItems, "max-items", 0, "max TODO items to process (0 = all)")
return cmd
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what newJSONLSink이 단순 파일명(dir ".")엔 디렉터리 생성 없이 성공하는지 검증한다.
package cli
import "testing"
// TestNewJSONLSinkFlatPath: a bare filename (dir ".") needs no directory creation.
func TestNewJSONLSinkFlatPath(t *testing.T) {
if _, err := newJSONLSink("out.jsonl"); err != nil {
t.Fatalf("newJSONLSink flat: %v", err)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what newJSONLSink이 부모 슬롯을 일반 파일이 차지하면 MkdirAll 에러를 표면화하는지 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
)
// TestNewJSONLSinkMkdirError: newJSONLSink surfaces a MkdirAll error when a regular
// file occupies a parent slot of the requested nested path.
func TestNewJSONLSinkMkdirError(t *testing.T) {
dir := t.TempDir()
blocker := filepath.Join(dir, "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
if _, err := newJSONLSink(filepath.Join(blocker, "sub", "out.jsonl")); err == nil {
t.Fatal("newJSONLSink under a file: want error, got nil")
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what JSONL export sink을 생성한다. path의 부모 디렉터리가 있으면 미리 만들어 둔다(append 대상 파일은 첫 Emit 때 생성).
package cli
import (
"os"
"path/filepath"
)
// newJSONLSink returns a sink writing to path, creating the parent directory.
func newJSONLSink(path string) (*jsonlSink, error) {
if dir := filepath.Dir(path); dir != "" && dir != "." {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, err
}
}
return &jsonlSink{path: path}, nil
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what 테스트 헬퍼. 명시 Definition으로 loop를 옵트인한 stub 퀘스트 CLI를 만들어 session/out·args로 1회 실행하고 합쳐진 출력을 돌려준다.
package cli
import (
"bytes"
"testing"
"github.com/park-jun-woo/reins/pkg/gate"
)
// newLoopRootDef builds a quest CLI (explicit Definition) with the loop opted in
// and runs one command, returning combined output.
func newLoopRootDef(t *testing.T, def gate.Definition, opts Options, session, out string, args ...string) (string, error) {
t.Helper()
cmd := NewQuestCmd("stub", def, opts)
var buf bytes.Buffer
cmd.SetOut(&buf)
cmd.SetErr(&buf)
full := append([]string{"--session", session, "--out", out}, args...)
cmd.SetArgs(full)
err := cmd.Execute()
return buf.String(), err
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what 테스트 헬퍼. stubDef로 loop를 옵트인한 퀘스트 CLI를 session/out·args로 실행한다(newLoopRootDef 래퍼).
package cli
import (
"testing"
)
// newLoopRoot builds a quest CLI (stubDef) with the loop opted in and runs one
// command, returning combined output.
func newLoopRoot(t *testing.T, opts Options, session, out string, args ...string) (string, error) {
return newLoopRootDef(t, stubDef{}, opts, session, out, args...)
}
//ff:func feature=cli type=helper control=sequence
//ff:what NewQuestCmd가 비지 않은 Options.Out으로 기본값을 덮어쓰고 Options.Version으로 루트 버전을 설정하는지 검증한다.
package cli
import "testing"
// TestNewQuestCmdExplicitOut: a non-empty Options.Out overrides the derived default,
// and Options.Version sets the root command's version.
func TestNewQuestCmdExplicitOut(t *testing.T) {
cmd := NewQuestCmd("stub", stubDef{}, Options{Out: "custom.jsonl", Version: "9.9.9"})
if got, _ := cmd.PersistentFlags().GetString("out"); got != "custom.jsonl" {
t.Fatalf("default --out = %q, want custom.jsonl", got)
}
if cmd.Version != "9.9.9" {
t.Fatalf("Version = %q, want 9.9.9", cmd.Version)
}
}
//ff:func feature=cli type=helper control=iteration dimension=1
//ff:what NewQuestCmd가 opts.ExtraCommands의 소비자 서브명령을 루트에 부착하는지 검증한다(G1).
package cli
import (
"testing"
"github.com/spf13/cobra"
)
// TestNewQuestCmdAttachesExtraCommands: a consumer command supplied via
// Options.ExtraCommands appears on the root alongside the canonical subcommands.
func TestNewQuestCmdAttachesExtraCommands(t *testing.T) {
run := &cobra.Command{Use: "run", Short: "consumer run command"}
cmd := NewQuestCmd("stub", stubDef{}, Options{ExtraCommands: []*cobra.Command{run}})
var foundRun bool
for _, c := range cmd.Commands() {
if c.Name() == "run" {
foundRun = true
}
}
if !foundRun {
t.Fatalf("ExtraCommands 'run' not attached to root")
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what nil ExtraCommands가 표준 6개 서브명령만 남기는지 검증한다(G1 후방호환).
package cli
import "testing"
// TestNewQuestCmdNoExtraCommands: nil ExtraCommands leaves only the canonical
// subcommands (backward-compatible).
func TestNewQuestCmdNoExtraCommands(t *testing.T) {
cmd := NewQuestCmd("stub", stubDef{}, Options{})
if got := len(cmd.Commands()); got != 6 {
t.Fatalf("subcommand count = %d, want 6 canonical with no extras", got)
}
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what NewQuestCmd의 명시 out 경로가 end-to-end로 실제 사용되는지(export가 거기 쓰는지) 검증한다.
package cli
import (
"bytes"
"path/filepath"
"strings"
"testing"
)
// TestNewQuestCmdRunsWithExplicitOut: the explicit out path is actually used end to
// end (export writes to it).
func TestNewQuestCmdRunsWithExplicitOut(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "explicit.jsonl")
cmd := NewQuestCmd("stub", stubDef{}, Options{Out: out})
var buf bytes.Buffer
cmd.SetOut(&buf)
cmd.SetErr(&buf)
cmd.SetIn(strings.NewReader(""))
cmd.SetArgs([]string{"--session", session, "scan", "a"})
if err := cmd.Execute(); err != nil {
t.Fatalf("scan: %v\n%s", err, buf.String())
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what NewQuestCmd가 빈 Options.Out에서 export 경로를 "<name>-results.jsonl"로 기본 설정하는지 검증한다.
package cli
import "testing"
// TestNewQuestCmdDefaultOut: with an empty Options.Out, the export path defaults to
// "<name>-results.jsonl".
func TestNewQuestCmdDefaultOut(t *testing.T) {
cmd := NewQuestCmd("stub", stubDef{}, Options{})
if got, _ := cmd.PersistentFlags().GetString("out"); got != "stub-results.jsonl" {
t.Fatalf("default --out = %q, want stub-results.jsonl", got)
}
}
//ff:func feature=cli type=helper control=iteration dimension=1
//ff:what NewQuestCmd가 표준 서브커맨드(scan/next/submit/status/export/rules)를 모두 배선하는지 검증한다.
package cli
import "testing"
// TestNewQuestCmdWiresSubcommands: the root wires the canonical subcommands.
func TestNewQuestCmdWiresSubcommands(t *testing.T) {
cmd := NewQuestCmd("stub", stubDef{}, Options{})
want := map[string]bool{"scan": false, "next": false, "submit": false, "status": false, "export": false, "rules": false}
for _, c := range cmd.Commands() {
want[c.Name()] = true
}
for name, found := range want {
if !found {
t.Errorf("subcommand %q not wired", name)
}
}
}
//ff:func feature=cli type=command control=sequence level=error
//ff:what 한 퀘스트의 표준 reins CLI(루트 + scan/next/submit/status/export/rules)를 조립한다. persistent 플래그 --session(기본 session.json)·--out(기본 "<name>-results.jsonl")를 달고, 도메인 로직은 gate.Definition만 끼우면 된다.
package cli
import (
"github.com/park-jun-woo/reins/pkg/gate"
"github.com/park-jun-woo/reins/pkg/quest"
"github.com/spf13/cobra"
)
// NewQuestCmd builds the standard reins CLI for a quest: a root command named name
// with the canonical subcommands scan/next/submit/status/export plus rules. The
// quest's domain logic is supplied by def; reins drives the ratchet, the level
// aggregation, and the export.
//
// Persistent flags: --session (default "session.json") and --out (default
// "<name>-results.jsonl"). Subcommands load the session, mutate it via the pure
// quest core, and save.
//
// Consumer-specific subcommands supplied via opts.ExtraCommands are attached to
// the root after the canonical ones.
func NewQuestCmd(name string, def gate.Definition, opts Options) *cobra.Command {
defaultOut := opts.Out
if defaultOut == "" {
defaultOut = name + "-results.jsonl"
}
var (
sessionPath string
outPath string
)
root := &cobra.Command{
Use: name,
Short: name + " — a reins quest CLI",
Version: opts.Version,
SilenceUsage: true,
SilenceErrors: false,
}
root.PersistentFlags().StringVar(&sessionPath, "session", "session.json", "session state file")
root.PersistentFlags().StringVar(&outPath, "out", defaultOut, "export output file (JSONL)")
load := func() (*quest.Session, error) { return loadSession(sessionPath) }
root.AddCommand(
newScanCmd(def, &sessionPath, load),
newNextCmd(def, load),
newSubmitCmd(def, &sessionPath, &outPath, load),
newStatusCmd(load),
newExportCmd(&sessionPath, &outPath, load),
newRulesCmd(def),
)
// Attach the loop command only when opted in (opts.Loop != nil), before the
// consumer-specific ExtraCommands. nil ⇒ no loop command (backward-compatible).
if opts.Loop != nil {
root.AddCommand(newLoopCmd(def, opts.Loop, &sessionPath, &outPath, load))
}
// Attach consumer-specific subcommands after the canonical ones. Nil or
// empty is a no-op, keeping existing callers unaffected.
root.AddCommand(opts.ExtraCommands...)
return root
}
//ff:func feature=cli type=helper control=sequence
//ff:what 테스트 헬퍼. 주어진 stdin을 가진 빈 cobra 명령을 만든다(readSubmission의 stdin 분기 자극용).
package cli
import (
"strings"
"github.com/spf13/cobra"
)
// newReadCmd builds a bare cobra command with the given stdin, for exercising
// readSubmission's stdin branch via cmd.InOrStdin.
func newReadCmd(in string) *cobra.Command {
cmd := &cobra.Command{}
cmd.SetIn(strings.NewReader(in))
return cmd
}
//ff:func feature=cli type=helper control=sequence level=error
//ff:what next가 깨진 세션 파일의 load 에러를 표면화하는지 검증한다.
package cli
import (
"os"
"path/filepath"
"testing"
)
// TestNextLoadError: next surfaces a load error from a corrupt session file.
func TestNextLoadError(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
if err := os.WriteFile(session, []byte("{bad"), 0o644); err != nil {
t.Fatalf("write: %v", err)
}
runCmdErr(t, stubDef{}, session, out, "", "next")
}
//ff:func feature=cli type=helper control=sequence
//ff:what next가 아이템이 없을 때 남은 TODO가 없다고 알리는지 검증한다.
package cli
import (
"path/filepath"
"strings"
"testing"
)
// TestNextNoTODO: with no items, next reports nothing remaining.
func TestNextNoTODO(t *testing.T) {
dir := t.TempDir()
session := filepath.Join(dir, "session.json")
out := filepath.Join(dir, "out.jsonl")
got := runCmd(t, stubDef{}, session, out, "", "next")
if !strings.Contains(got, "no TODO items remaining") {
t.Fatalf("next = %q", got)
}
}
//ff:func feature=cli type=helper control=sequence
//ff:what printSubmit — 제출 1건의 결과를 보고한다(renderVerdict 래퍼).
package cli
import (
"io"
"github.com/park-jun-woo/reins/pkg/quest"
)
// printSubmit reports the outcome of one submission (renderVerdict wrapper).
func printSubmit(w io.Writer, key string, it *quest.Item, v quest.Verdict) {
renderVerdict(w, key, it, v)
}