Runtime Behavior Probe

Runtime Behavior Probe

Overview

Use this skill to investigate real runtime behavior, not to restate code or documentation. Start by planning the investigation, then execute a case matrix, record observed behavior, and report both the findings and the method used to obtain them.

Core Rules

• Treat this skill as manual-only. Do not rely on implicit invocation.
• A baseline success or smoke case is often the right entry point, but do not stop there when the real question involves edge cases, drift, or failure behavior.
• Plan before running anything. Write the case matrix first, then fill it in with observed results. The matrix can live in a scratch note, a temporary file, or the probe script header.
• Default to local or read-only probes. Consider a live service only when it is clearly relevant, then apply the lightweight gates below before you run it.
• Size the probe to the decision. Start with the smallest matrix that can disqualify or validate the current hypothesis, then expand only when uncertainty remains.
• Before a live probe, apply three lightweight gates:
• Destination gate. Use only a live destination that is clearly allowed for the task.
• Intent gate. Run the live probe only when the user explicitly wants runtime verification on that integration, or explicitly approves it after you propose the probe.
• Data gate. If the probe will read environment variables, mutate remote state, incur material cost, or exercise non-public or user data, name the exact variable names or data class and get explicit approval first.
• Classify each case as read-only, mutating, or costly before execution. For mutating or costly cases, or for any live case that will read environment variables, define cleanup or rollback before running the probe.
• Use temporary files or a temporary directory for one-off probe scripts.
• Keep temporary artifacts until the final response is drafted. Then delete them by default unless the user asked to keep them or they are needed for follow-up. Even when artifacts are deleted, keep a short run summary of the command shape, runtime context, and artifact status in the report.
• Before executing a live probe that will read environment variables, tell the user the exact variable names you plan to use and why, then wait for explicit approval. Examples include OPENAI_API_KEY and other expected default names for the system under test.
• Never print secrets, even when they come from standard environment variables that this skill may use.
• For OpenAI API or OpenAI platform probes in this repository, use $openai-knowledge early to confirm contract-sensitive details such as supported parameters, field names, and limits. Use runtime probing to validate or challenge the documented behavior, not to skip the documentation pass entirely. If the docs MCP is unavailable, fall back to the official OpenAI docs and say that you used the fallback in the report.
• For benchmark or comparison probes, make parity explicit before execution. Record what is held constant, what variable is under test, which response-shape constraints keep the comparison fair, and any usage or token counters that matter for interpreting latency or cost.
• For OpenAI hosted tool probes, remove setup ambiguity before attributing a negative result to runtime behavior:
• Force the tool path with the matching tool_choice when the question depends on tool invocation.
• Treat container_auto and container_reference as separate cases, not interchangeable setup details.
• Clear unsupported model or tool options first so they do not invalidate the probe.

Workflow

1. Restate the investigation target in operational terms. Name the runtime surface, the key uncertainty, and the highest-risk behaviors to test. 2. Do a short preflight. Check the relevant code or docs first, decide whether the question needs local or live validation, and note any repo, baseline, or release boundary that matters. 3. Create a validation matrix before executing probes. Cover both baseline behavior and the most relevant failure or drift cases. The matrix can live in a scratch note, a temporary file, or a structured header inside the probe script. 4. For each case, choose an execution mode up front:

• single-shot for deterministic one-run checks.
• repeat-N for cache, retry, streaming, interruption, rate-limit, concurrency, or other run-to-run-sensitive behavior.
• warm-up + repeat-N when first-run cold-start effects could distort the result.

Use these defaults unless the task clearly needs something else:

• Quick screen of a repeat-sensitive question: repeat-3.
• Decision-grade latency or release recommendation: warm-up + repeat-10.
• Costly live cases: start at repeat-3, then expand only if the answer remains unclear.

If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe. 5. When the question is benchmark-like or comparative, run in phases. Start with a high-signal pilot matrix against a control, then expand only the surviving candidates or unresolved cases. 6. If the question is about a suspected regression or behavior change, add at least one known-good control case such as origin/main, the latest release, or the same request without the suspected option. 7. For comparative probes, define parity before execution. Record prompt or input shape, tool-choice setup, model-settings parity, state reuse rules, and any response-shape constraint that keeps the comparison fair. If materially different output length could bias the result, record usage or token notes too. 8. If the question asks whether one option has the same intelligence or quality as another, decide whether the matrix supports only example-pattern parity or a broader quality claim. For broader claims, add at least one harder or more open-ended case. Otherwise say explicitly that the result is limited to the covered patterns. 9. Plan state controls before execution when hidden state could affect the result. Record whether each case uses fresh or reused state, how cache reuse or cache busting is handled, what unique IDs isolate repeated runs, and how cleanup is verified. 10. If any live case will read environment variables, list the exact variable names and purpose for each case, then ask the user for approval before execution. Keep the approval ask short and include destination, read-only versus mutating or costly risk, exact variable names, and cleanup or rollback if relevant. 11. Build task-specific probe scripts in a temporary location. Keep the script small, observable, and easy to discard. 12. In openai-agents-python, make the runtime context explicit:

• Run Python probes from the repository root with uv run python when practical.
• Record the current commit, working directory, Python executable, and Python version.
• Avoid accidental imports from a different checkout or site-packages location. If you must deviate from uv run python, say exactly why and what interpreter or environment was used instead.

13. Execute the matrix and capture evidence. Record request shape, setup, observation summary, unexpected or negative result, error details, timing, runtime context, approved environment-variable names, repeat counts, warm-up handling, variance when relevant, cleanup behavior, and for comparisons note what was held constant plus any response-shape or usage notes that affect interpretation. 14. Update the matrix with actual outcomes, not guesses. 15. Keep temporary artifacts until the final response is drafted. Then delete them unless the user asked to keep them or they are needed for follow-up. Benchmark and repeat-heavy probes often need follow-up, so keeping artifacts is normal when the result may be revisited. If deleted, retain and report a short run summary. 16. Report findings first, with unexpected or negative findings first. Then summarize how the validation was performed and which cases were covered. 17. If the probe isolates one clear defect, you may include a short implementation hypothesis or minimal repro direction. Do not expand into a larger next-step plan unless the user asked for it.

Validation Matrix

Use a matrix that makes the news easy to scan. Start from the runtime question and the observation summary, not just from expected and pass or fail.

Use a matrix with at least these columns:

• case_id
• scenario
• mode
• question
• setup
• observation_summary
• result_flag
• evidence

Add these columns when they materially improve the investigation:

• comparison_basis
• variable_under_test
• held_constant
• output_constraint
• status
• confidence
• state_setup
• repeats
• warm_up
• variance
• usage_note
• risk_profile
• env_vars
• approval
• control

Treat result_flag as a fast scan field such as unexpected, negative, expected, or blocked. Use status only when there is a credible comparison basis, baseline, or documented contract to compare against.

Always consider whether the matrix should include these categories:

• Baseline success.
• Control or baseline comparison when a regression is suspected.
• Boundary input or parameter variation.
• Invalid or unsupported input.
• Missing or incorrect configuration.
• Transient external failure such as timeout, network interruption, or rate limiting.
• Retry, idempotence, or cleanup behavior.
• Concurrency or overlapping operations when shared state or ordering may matter.
• Open-ended quality or intelligence samples when the question is broader than pattern parity.

Open validation-matrix.md when you need a stronger prioritization model or a reusable case template.

Temporary Probe Scripts

Write one-off scripts in a temporary file or temporary directory such as one created by mktemp -d or Python tempfile. Keep the script outside the repository by default, even when it imports code from the repository.

If the probe needs repository code:

• Run it with the repository as the working directory, or
• Set PYTHONPATH or the equivalent import path explicitly.
• In openai-agents-python, prefer uv run python /tmp/probe.py from the repository root.

Design the probe to maximize observability:

• Print or log the exact scenario being exercised.
• Capture runtime context such as git SHA, working directory, Python executable and version, relevant package versions, model or deployment name, endpoint or base URL alias, and any retry or tool options that materially affect behavior.
• For live probes, record only the names of environment variables that were approved for use. Never print their values.
• Capture structured outputs when possible.
• Preserve raw error type, message, and status code.
• For repeat-sensitive cases, capture the attempt index, warm-up status, and any stable identifiers that help compare runs.
• For repeated or benchmark-style probes, write both raw results and a compact summary artifact when practical.
• Keep branching minimal so each script answers a narrow question.

Before deleting the temporary script or directory, keep a short run summary of the script path, command used, runtime context, and whether the evidence was kept or deleted.

Open python_probe.py when you want a lightweight disposable Python probe scaffold.

Reporting

Report in this order:

1. Findings. Put unexpected or negative findings first. If there was no real news, say that explicitly. 2. Validation approach. Summarize the code used, the runtime surface exercised, the execution modes, and the case matrix coverage. 3. Case results. Include the matrix or a condensed version of it when the case count is large. 4. Artifact status and brief run summary. State whether temporary artifacts were deleted or kept, and provide kept paths or the retained summary. 5. Optional implementation note. Include this only when one clear defect was isolated and a short implementation direction would help.

For comparative probes, the report should also say what was held constant, what variable was under test, and whether the result supports only pattern parity or a broader quality claim.

Open reporting-format.md for the recommended response template.

Resources

• Open validation-matrix.md to design and prioritize the case matrix.
• Open error-cases.md to expand common failure scenarios.
• Open openai-runtime-patterns.md for recurring OpenAI and Responses API probe patterns.
• Open reporting-format.md for the final report structure.
• Open python_probe.py for a minimal disposable Python probe scaffold.

Common Error Cases

Use this reference to expand beyond the happy path. Favor error cases that a real user or operator is likely to hit.

Configuration Errors

Check whether the runtime behaves differently for:

• Missing required environment variables.
• Present but malformed secrets or identifiers.
• Wrong endpoint or base URL.
• Wrong model or deployment name.
• Incompatible local dependency versions.

Look for:

• Error type and status code.
• Whether the failure is immediate or delayed.
• Whether the message is actionable.
• Whether retrying without fixing configuration changes anything.

Input Errors

Probe common bad-input patterns such as:

• Missing required fields.
• Wrong data type.
• Unsupported enum or option value.
• Empty but syntactically valid input.
• Oversized input or too many items.
• Mutually incompatible options.

Prefer realistic invalid inputs over artificial nonsense. The point is to learn how the runtime fails in practice.

Transport and Availability Errors

When networked services are involved, consider:

• Connection failure.
• Read timeout.
• Server timeout or upstream gateway error.
• Rate limit response.
• Partial stream interruption.
• Reusing a connection after a failure.

Capture whether the client library retries automatically, whether it surfaces retry metadata, and whether the final exception preserves the original cause.

State and Repetition Errors

Many surprising bugs appear only when an operation is repeated or interrupted:

• Re-submit the same request.
• Repeat after a timeout.
• Retry after a partial tool call or partial stream.
• Resume after local cleanup or process restart.
• Repeat with slightly changed inputs while reusing shared state.

Observe whether the operation is idempotent, duplicated, silently ignored, or left in a partial state.

Concurrency Errors

When shared state, ordering, or isolation may matter, consider:

• Two overlapping requests with the same logical input.
• Parallel runs that reuse the same cache key, session, container, or temporary resource.
• Concurrent retries, cancellation, or cleanup racing with active work.
• Output or event streams from one run leaking into another.

Capture whether the runtime serializes, rejects, duplicates, corrupts, or cross-contaminates the work.

Investigation Heuristics

Use these heuristics to pick error cases quickly:

• Ask which failure a real engineer would debug first in production.
• Ask which failure is most expensive if it is misunderstood.
• Ask which failure would be invisible from code review alone.
• Ask which failure path is likely to differ across environments.

If the error behavior is already perfectly obvious from a local validator or type system, it is usually low priority for this skill.

OpenAI Runtime Patterns

Use this reference for recurring OpenAI investigations so you do not have to rediscover the probe strategy each time. In this repository, use $openai-knowledge up front for contract-sensitive details, then use this reference to design the runtime validation. If the docs MCP is unavailable, fall back to the official OpenAI docs and say so in the report.

General Rules

• Prefer small live probes over large harnesses.
• Keep one script focused on one uncertainty.
• For comparative or benchmark-like questions, start with a pilot and expand only when the answer is still unclear.
• Capture both the request shape and the returned item types.
• Preserve raw error payloads and status codes.
• Record whether behavior differs between the first call and a repeated call.
• When the question is about regression or contract drift, add a known-good control run before attributing the result to the change under investigation.
• Keep comparison parity explicit. Record what was held constant, what variable changed, and whether output-shape or usage differences could bias the conclusion.
• When the question depends on tool invocation, force the target path with the matching tool_choice.
• Treat container_auto and container_reference as distinct setup modes, not interchangeable details.
• Clear unsupported model or tool options before diagnosing runtime behavior.

Standard Environment Variables

Do not read these variables automatically. Before a live probe uses any of them, tell the user the exact variable names you plan to read and why each one is needed, then wait for explicit approval. Never print their values:

• OPENAI_API_KEY
• OPENAI_BASE_URL
• OPENAI_ORG_ID
• OPENAI_PROJECT_ID

If the task targets another standard integration, use that integration's expected default variable names under the same rule.

Responses API Probe Patterns

For Responses API work, start from the uncertainty instead of from the full feature surface.

Benchmark or model-switch comparisons

Use when you need to compare models, settings, transports, or providers with enough rigor to support a product or release decision.

Probe suggestions:

• Start with a pilot that includes one control and two or three highest-signal scenarios.
• Keep prompt shape, tool choice, state setup, and non-tested settings aligned across candidates.
• If the question is about speed, capture medians and, when relevant, first-token latency plus any usage note that could explain the difference.
• If the question is about "same intelligence" or "same quality," add at least one harder or more open-ended case. Otherwise report the result as pattern parity only.
• Expand to a larger matrix only when the pilot survives, the candidates are close, or a major runtime surface is still uncovered.

Plain response behavior

Use when you need to confirm:

• The shape of returned output items.
• Whether text appears in one item or multiple items.
• How metadata appears in the final object.

Probe suggestions:

• Baseline call with a minimal input.
• Same call with a slightly different instruction shape.
• Repeat the same call to check output stability where that matters.

Structured output behavior

Use when you need to observe:

• Schema rejection versus best-effort completion.
• Handling of missing required fields.
• Differences between model-compliant output and transport-level errors.

Probe suggestions:

• Valid schema and valid prompt.
• Prompt likely to produce omitted fields.
• Clearly incompatible schema or unsupported option when relevant.

Tool invocation behavior

Use when you need to learn:

• When tool calls are emitted.
• How arguments are shaped at runtime.
• What happens when the tool fails or returns malformed output.

Probe suggestions:

• Baseline tool-call success.
• Tool failure with a realistic exception.
• Tool result that is syntactically valid but semantically incomplete.

Hosted shell and code interpreter failure shields

When probing hosted tools through the Responses API, eliminate common setup ambiguity first:

• Force the tool path you want to test with the matching tool_choice. A text-only completion without forced tool choice is not a reliable negative result.
• Treat container_auto and container_reference differently. Use container_auto when the probe needs fresh container provisioning or skill attachment, and use container_reference only to reuse existing container state.
• Do not assume every environment field is accepted on every container mode. If the probe is about skills, validate that the chosen container mode actually supports skill attachment before treating an API error as a runtime defect.
• Check model-specific option support before chasing unrelated failures. Unsupported reasoning or model settings can invalidate the probe before the tool path is exercised.
• For hosted package installation, treat network-dependent setup as best-effort and separate install failures from the underlying tool behavior you are trying to observe.
• For prompt cache investigations, keep model, instructions, tool configuration, and cache key effectively identical across repeated runs before interpreting cached_tokens.

Streaming behavior

Use when the uncertainty involves:

• Event ordering.
• Partial text delivery.
• Termination after interruption.
• Tool-call events in streams.

Probe suggestions:

• Normal streamed completion.
• Early local cancellation.
• Network interruption if it can be reproduced safely.

What to Capture

For OpenAI probes, try to record:

• Request options that materially affect behavior.
• Response item types and their order.
• Whether fields are absent, null, empty, or transformed.
• Server status and error payload details for failures.
• Retry and backoff hints when present.
• Stable identifiers that help compare repeated runs, such as request IDs, response IDs, tool call IDs, or container IDs when available.
• Which environment-variable names were approved for the probe when live credentials were required.

Do not spend time rediscovering static documentation unless the runtime result seems to contradict what you expected. The value of this skill is in the observed behavior.

Reporting Format

Lead with findings, not process. The user asked for investigation results, so the answer should start with the most important observed behaviors. Put the real news first.

Recommended Order

1. Findings. 2. Validation approach. 3. Case matrix or condensed case summary. 4. Artifact status and brief run summary. 5. Optional implementation note.

Findings Section

Make each finding answer one user-relevant question. Good findings usually include:

• What was observed.
• Why it matters.
• The condition under which it happens.
• What was held constant when the finding comes from a comparison probe.
• scope: The boundary of the finding, such as commit, model, Python version, live vs local, or repeat mode.
• confidence: high, medium, or low.

Avoid burying the main result under setup details.

Put unexpected or negative findings first. If there were no unexpected or negative findings in the executed cases, say that explicitly before the rest of the findings section.

If the probe was comparative, say whether the result supports:

• Pattern parity only.
• A broader quality claim.

Do not imply a broader quality equivalence than the executed cases justify.

Validation Approach Section

Summarize:

• The runtime surface you exercised.
• The shape of the probe code, in overview only.
• Which categories of cases you covered.
• Which execution modes you used, including repeat counts or warm-up handling when relevant.
• Whether live credentials or external services were used.
• Any important state controls such as fresh state, cache reuse, cache busting, unique IDs, or cleanup verification.
• For comparison probes, what was held constant, what was varied, and whether output-shape or usage differences could still influence the conclusion.
• Whether the usual docs path or an official-docs fallback was used for contract-sensitive checks.

Keep this concise. The user needs enough detail to trust the result, not a line-by-line replay of the script.

Case Summary

Include either the full matrix or a condensed summary. At minimum, show:

• Which scenarios were executed.
• Whether the run was a quick pilot, an expanded matrix, or both.
• Which ones produced unexpected or negative results.
• Which ones passed or failed when a real comparison basis existed.
• Which cases were blocked.
• Where the supporting evidence lived, or that it was deleted.

If the matrix is large, show the highest-value cases in the main response and keep the rest as a compact appendix or note.

Artifact Status And Brief Run Summary

State one of these explicitly:

• Temporary artifacts were kept until the final response was drafted, then deleted after validation.
• Temporary artifacts were kept at <path> because the user asked to keep them.
• Temporary artifacts were kept at <path> because they are needed for follow-up analysis.

Even if artifacts were deleted, retain a short run summary such as:

• Probe command or runner shape.
• Runtime context summary such as commit, Python executable, Python version, or model.
• Artifact path and final status.

For benchmark or repeat-heavy probes, keeping artifacts for follow-up is often the right default even when the immediate report is done.

Optional Implementation Note

Include this only when one clear defect was isolated and a short implementation hypothesis or minimal repro direction would help. Keep it brief. Do not turn the report into a broader next-step plan unless the user asked for that.

Compact Template

Use this outline when you need a fast structure:

Findings:

• <finding 1>

held constant: <prompt/tool/state settings kept the same, if comparative> scope: <commit/model/python/live-local/repeat-mode> confidence: <high|medium|low>

• <finding 2>

held constant: <prompt/tool/state settings kept the same, if comparative> scope: <commit/model/python/live-local/repeat-mode> confidence: <high|medium|low>

Validation approach:

• Surface: <what was exercised>
• Probe code: <brief overview>
• Coverage: <success, edge, error, repeat-sensitive, and quality categories>
• Execution modes: <single-shot|repeat-N|warm-up + repeat-N>
• Comparison parity: <what was held constant and what varied, if comparative>
• Docs source: <MCP or official-docs fallback, if relevant>

Case summary:

Artifact status and brief run summary:

• Temporary artifacts were kept until the final response was drafted, then deleted.
• Summary: <command/runtime-context/artifact-status summary>

Optional implementation note:

• <brief hypothesis or minimal repro direction>

Adjust the format to the task, but preserve the ordering.

Validation Matrix

Use the matrix to decide what to probe before writing scripts. The goal is not exhaustive combinatorics; the goal is high-value coverage that is visible, explainable, and likely to reveal runtime surprises. The matrix should make the real news easy to scan.

Minimum Columns

Use these columns unless the task clearly needs more:

• case_id: Stable identifier such as S1, E3, or R2.
• scenario: Short description of the behavior under test.
• mode: single-shot, repeat-N, or warm-up + repeat-N.
• question: The concrete runtime uncertainty this case is answering.
• setup: Inputs, environment, or preconditions required for the case.
• observation_summary: A compact summary of what actually happened.
• result_flag: unexpected, negative, expected, or blocked.
• evidence: Path, log reference, or deleted.

Add these columns when they materially improve the investigation:

• comparison_basis: The baseline, docs, or prior behavior you are comparing against.
• variable_under_test: The single factor that is intentionally changing in a comparison case.
• held_constant: Prompt shape, tool setup, model settings, or state rules that were intentionally kept the same.
• output_constraint: Any schema, length, or response-shape constraint used to keep the comparison fair.
• status: Use pass, fail, unexpected-pass, unexpected-fail, or blocked only when there is a credible comparison basis or control.
• confidence: high, medium, or low.
• state_setup: Fresh or reused state, cache strategy, unique IDs, and cleanup checks.
• repeats: Number of measured runs.
• warm_up: Whether a warm-up run was used and why.
• variance: Any useful spread or instability note across repeated runs.
• usage_note: Token, usage, or output-length note when it materially affects interpretation.
• control: Known-good comparison point for regression or behavior-change questions.
• risk_profile: read-only, mutating, or costly for live probes.
• env_vars: Exact environment-variable names the case plans to read.
• approval: not-needed, pending, or approved for cases that need user permission before execution.

Use result_flag as the fast scan field. It is what makes unexpected or negative findings jump out before the reader studies the full report.

Use status only when you have a real comparison basis. If the case is exploratory and there is no trustworthy baseline, prefer a strong observation_summary plus result_flag and confidence instead of pretending the result is a clean pass or fail.

Choosing Execution Mode

Pick an execution mode before you run the case:

• Use single-shot for deterministic, one-run checks.
• Use repeat-N automatically when the question involves cache behavior, retries, streaming, interruptions, rate limiting, concurrency, or other run-to-run-sensitive behavior.
• Use warm-up + repeat-N when the first run is likely to include cold-start effects such as container provisioning, import caches, or prompt-cache population.

Use these defaults unless the task clearly needs something else:

• repeat-3 for a quick screen of a repeat-sensitive question.
• warm-up + repeat-10 for decision-grade latency comparisons or release-facing recommendations.
• For costly live probes, start at repeat-3 and expand only if the answer is still unclear.

If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe.

Phase The Matrix

When the question is comparative or benchmark-like, do not jump straight to the largest matrix.

Start with a pilot:

• One control.
• One or two highest-signal success cases.
• The smallest repeat count that can disqualify a weak candidate quickly.

Expand only when:

• The candidate survives the pilot.
• The results are close enough that more samples matter.
• A major runtime surface is still uncovered.
• The user explicitly wants decision-grade evidence.

Coverage Categories

Try to cover at least one case from each relevant category:

• success: Normal behavior that should work.
• control: Known-good comparison such as origin/main, the latest release, or the same request without the suspected option.
• boundary: Size, count, or parameter limits near a plausible edge.
• invalid: Bad inputs or unsupported combinations.
• misconfig: Missing key, wrong endpoint, bad permissions, or incompatible local setup.
• transient: Timeout, temporary server issue, network breakage, or rate limiting.
• recovery: Retry behavior, partial completion, duplicate submission, or cleanup.
• concurrency: Overlapping operations when shared state, ordering, or isolation may matter.
• quality: A harder or more open-ended sample when the user is asking about model intelligence, not just workflow parity.

If time is limited, prioritize categories in this order:

1. Known-good control when the question implies regression or drift. 2. Highest-risk success case. 3. Most plausible user-facing failure. 4. Most likely edge case with ambiguous behavior. 5. Cleanup or retry semantics. 6. Lower-probability extremes.

Matrix Template

Use this compact template:

Recording Results

Keep question unchanged after execution. Put the actual behavior in observation_summary, then mark the scan-friendly result_flag.

Use these result_flag values consistently:

• unexpected: The result diverged from the best current understanding in a surprising way.
• negative: The result exposed a user-relevant failure, risk, or sharp edge.
• expected: The result matched the current understanding and did not reveal new risk.
• blocked: The case did not produce a trustworthy observation.

Only fill status when there is a credible comparison basis. Otherwise use observation_summary, result_flag, and confidence to communicate what was learned without over-claiming certainty.

For comparison cases, use observation_summary and the final report to say whether the evidence supports pattern parity only or a broader quality claim.

If a case reveals a new branch of behavior, add a follow-up case instead of overloading the original one.

Evidence Discipline

Treat a case as incomplete when:

• The observed output omits the key result you were testing.
• The script mixed multiple questions and the result is ambiguous.
• Hidden state, cache behavior, or previous runs may have influenced the result and were not controlled or documented.
• The question is whether behavior changed, but the case has no credible control or baseline to compare against.
• The case plans to read environment variables, but the exact variable names were not approved by the user before execution.
• The case was repeat-sensitive, but it ran only once without a clear rationale.

When this happens, narrow the probe and rerun. A smaller script with a cleaner result is better than a more complicated script that is hard to trust.