Rigorous Experiments
- 36 installs
- 339 repo stars
- Updated August 4, 2026
- glebis/claude-skills
Design, run, validate, and audit statistical experiments on personal or observational time-series data with pre-registration, exact permutation tests, and FDR discipline.
About
Enforces rigorous n-of-1 statistical practice (pre-registration, exact permutation, fixed-family BH, stationarity and Simpson checks, adversarial review) across design, conduct, validate, and audit modes. A developer uses it to test whether a correlation in health, behavioral, or self-tracking data is real.
- Exact permutation over the full calendar, never sampled on small n
- Honest statuses: confirmed vs lead vs null vs descriptive
Rigorous Experiments by the numbers
- 36 all-time installs (skills.sh)
- Ranked #1,042 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/glebis/claude-skills --skill rigorous-experimentsAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 36 |
|---|---|
| repo stars | ★ 339 |
| Last updated | August 4, 2026 |
| Repository | glebis/claude-skills ↗ |
What it does
Design, run, validate, and audit statistical experiments on personal or observational time-series data with pre-registration, exact permutation tests, and FDR discipline.
Files
Rigorous Experiments
Run statistical experiments on observational/personal time-series data that survive scrutiny. Distilled from a 54-experiment n-of-1 program in which sampled permutation tests, missing-data artifacts, app-categorization bugs and collinear mechanisms repeatedly manufactured — and then destroyed — "findings". Every rule here exists because its absence once produced a wrong conclusion.
Modes
Pick the mode matching the request; chain them for a full study.
| Mode | When | Reference |
|---|---|---|
| design | New hypothesis or study | references/design.md |
| conduct | Implementing + running the experiment | references/statistics.md |
| validate-data | Before trusting ANY new data source | references/data-validation.md |
| cross-validate | Findings worth defending; code review; external model review (e.g. GPT Pro) | references/cross-validation.md |
| investigate-leads | A sweep/run produced leads (p<0.06, not FDR-confirmed) | references/lead-investigation.md |
| audit | Re-examining past claims, registries of findings | references/statistics.md §Audit |
Non-negotiable core (all modes)
1. Pre-register before computing. Hypotheses, exact tests, family size m, and the acceptance threshold go in the script docstring BEFORE the first run. Post-hoc tests are reported as descriptive, never promoted. 2. Exact permutation, never sampled, on small n. A session sequence of n=19 has 18 circular shifts: the minimum honest p is ~1/19≈0.05. Sampling 2000 shifts with replacement fabricates precision (this killed a flagship "q=0.028" finding). Use scripts/perm_stats.py. 3. Permute over the full calendar, not the compressed series. Shifting a gap-compressed series breaks the timeline; keep missingness as NaN masks re-applied per shift. Event indicators must be pure 0/1 with no gaps — missingness lives only in the outcome series. 4. BH with FIXED family size m, a LITERAL CONSTANT declared at design time — never len(tests) (that defeats pre-registration; the linter rejects it). Assert the run matches the declared m. Confirmatory families small and separate from exploratory sweeps; pooling everything into one BH buries true effects, cherry-picking families manufactures them. Plain BH assumes independent/positively-dependent tests; for strongly dependent lag families use BH-Yekutieli or maxT resampling. 5. Stationarity check before correlating trending series. Exact circular shift on a trending series is "exactly, reproducibly wrong": report prewhitened-r (AR1 residuals) and stationary bootstrap alongside. 6. Stratify before pooling (Simpson check): within group (e.g. therapy/coaching) and within regime (pre/post known breaks). A pooled r=−0.25 once hid therapy −0.64 vs coaching +0.53. 7. Controls can re-describe a finding, not just kill it. When a control collapses an effect, check collinearity of control and predictor — r(self-focus, session-length)=0.79 meant "mechanism ambiguous", not "effect fake". Report the decomposition. 8. Honest statuses: confirmed (q<0.10 exact) ≠ lead (p<0.06) ≠ null ≠ descriptive. Status flips are recorded, never silently edited. Nulls with adequate power are findings. Robust ≠ significant: a lead surviving leave-one-out at small n is still underpowered — a candidate for prospective test, not a finding. 8b. Series scope is part of the test. A lagged "[t+1]" means the next unit in the series the hypothesis is about, not the next pooled row; define scope before lagging (it once flipped a sign). When recomputing a prior result, reproduce a stored artifact on that scope first. 9. Privacy: raw text/audio never enters output files or external uploads — statistics, rates and embedding-derived scores only. 10. Plain-language reporting: every statistic carries its practical meaning inline; define r/p/q/n once per report; no untranslated jargon calques. Narrative first, numbers as support.
Workflow (full study)
1. validate-data gate on any new source (see reference — the checklist has caught: zero-vs-missing conflation, dedup semantics, substring category bugs, rolling purge windows, timezone conventions). 2. design: pre-registered hypotheses + family + power sanity. 3. conduct: implement with scripts/perm_stats.py; run; write results JSON with tests, statuses, and caveats including known limitations. 4. cross-validate: adversarial code review (e.g. Codex read-only) BEFORE trusting results; fix findings; re-run. For major claims, external model review with a privacy-screened archive. 5. investigate-leads on anything that surfaced as a lead (not at the same scale — the triage battery: LOO, directionality, detrend-vs-step, within-cycle, prewhiten+bootstrap; consolidate same-direction leads into one composite). Mark diagnostic runs descriptive_only: true. 6. Verdicts in honest prose (mixed/rejected allowed); report; registry update with status provenance.
Viewing results
Launch the bundled explorer over any directory of results JSONs:
python3 scripts/explorer.py <results_dir> [--port 8799] [--pattern "exp*.json"] [--sort newest|oldest]Generates explorer.html in the directory, starts (or reuses) a loopback http server on the port, and opens the browser: experiment list with confirmed/lead badges, filter, sortable test tables color-coded by status, verdicts, caveats, raw JSON. The page fetches result files live — re-running experiments updates the view; re-run the script only when new result files appear. Serve over localhost, never file:// (CDN fonts) and never on a non-loopback interface (results may contain personal statistics).
Evals
Run python3 evals/run_evals.py (from the skill directory) to lint an experiment script/results pair against the standards (pre-registration present, fixed literal m, exact perm usage, caveats, no raw text in outputs). A diagnostic/triage run that intentionally mints no new tests sets descriptive_only: true in its results JSON to satisfy the "has tests" check. Eval cases in evals/cases/ document expected pass/fail examples.
"""Demo experiment without discipline."""
import numpy as np
def perm_p(x, y, reps=2000):
rng = np.random.default_rng(0)
count = 0
for _ in range(reps):
k = int(rng.integers(1, len(y)))
yr = np.roll(y, k)
count += abs(np.corrcoef(x, yr)[0, 1]) >= 0.3
return count / reps
{"experiment": "demo-bad",
"tests": [{"h": "T1", "r": 0.62}],
"per_session": [{"quote": "And then I told my therapist about the whole situation with the apartment and how my mother reacted when she found out about the move and everything that happened afterwards during that long difficult conversation we had on the phone late at night"}]}
[{"name": "disciplined experiment passes clean",
"script": "good_exp.py", "results": "good_results.json",
"expect_codes": []},
{"name": "sampled perm + no prereg + leaky results all flagged",
"script": "bad_exp.py", "results": "bad_results.json",
"expect_codes": ["L1", "L3", "L3b", "L4", "L5", "L6"]}]
"""Demo experiment.
PRE-REGISTERED (m=2): T1 x<->y same day; T2 x->y next day.
Exact circular permutation over the full calendar; BH m=2.
"""
from perm_stats import bh, exact_circ_p
def main(x, y):
tests = []
r, p, n = exact_circ_p(x, y)
tests.append({"h": "T1", "r": r, "p": p, "n": n})
r, p, n = exact_circ_p(x, y[1:] + [None])
tests.append({"h": "T2", "r": r, "p": p, "n": n})
bh(tests, m=2)
return tests
{"experiment": "demo", "hypothesis": "x relates to y",
"tests": [{"h": "T1", "r": 0.1, "p": 0.4, "q": 0.5, "n": 100}],
"caveats": ["small n", "volume collider not controlled"]}
#!/usr/bin/env python3
"""Evals for the rigorous-experiments skill.
Two uses:
python3 evals/run_evals.py # run bundled eval cases
python3 evals/run_evals.py <script.py> <results.json> # lint real files
The linter checks the standards the skill enforces:
L1 pre-registration present in the script docstring
L2 BH called with an explicit fixed family size (m=...)
L3 exact permutation used; sampled-permutation patterns flagged
L4 results JSON has non-empty caveats
L5 results tests carry p and n fields
L6 privacy: no sentence-length quoted data outside prose keys
"""
from __future__ import annotations
import ast
import json
import os
import re
import sys
PROSE_KEYS = {"hypothesis", "method", "caveats", "desc", "description",
"interpretation", "goal", "note", "verdict", "label",
"approaches", "status"}
def lint_script(path):
src = open(path, encoding="utf-8").read()
findings = []
try:
tree = ast.parse(src)
doc = ast.get_docstring(tree) or ""
except SyntaxError as e:
return [("L0", f"script does not parse: {e}")]
prereg = re.search(r"pre-?registered", doc, re.I)
if not prereg or re.search(r"not\s+pre-?registered", doc, re.I):
findings.append(("L1", "no pre-registration block in module "
"docstring (write hypotheses/tests/m BEFORE "
"running)"))
elif not re.search(r"\bm\s*=\s*\d|\bfamily\b", doc, re.I):
findings.append(("L1b", "pre-registration block lacks a declared "
"family size (m=N)"))
# L2: bh() must receive a CONSTANT m (AST), not len(tests)/variables
for node in ast.walk(tree):
if isinstance(node, ast.Call) and \
getattr(node.func, "id", getattr(node.func, "attr", "")) \
== "bh":
mkw = next((k for k in node.keywords if k.arg == "m"), None)
marg = (mkw.value if mkw else
node.args[1] if len(node.args) > 1 else None)
if marg is None:
findings.append(("L2", "bh() called without family size"))
elif not isinstance(marg, ast.Constant):
findings.append(("L2", "bh() family size is computed, not "
"a declared constant (m=len(tests) "
"defeats pre-registration)"))
sampled = re.search(
r"for\s+_?\w*\s+in\s+range\s*\(\s*(reps|2000|5000|1000)\b"
r"[\s\S]{0,200}?np\.roll", src)
if sampled:
findings.append(("L3", "sampled circular-shift permutation "
"detected (range(reps)+np.roll): use exact "
"all-shifts enumeration"))
if not re.search(r"exact_circ_p|exact_event_diff|break_diff"
r"|all\s+n-1 shifts|for\s+k\s+in\s+range\s*\(\s*1?\s*,"
r"\s*len", src):
findings.append(("L3b", "no exact permutation machinery found "
"(import perm_stats or enumerate shifts)"))
return findings
def _walk_strings(obj, keypath=()):
if isinstance(obj, dict):
for k, v in obj.items():
yield from _walk_strings(v, keypath + (str(k).lower(),))
elif isinstance(obj, list):
for v in obj:
yield from _walk_strings(v, keypath)
elif isinstance(obj, str):
yield keypath, obj
def lint_results(path):
findings = []
try:
d = json.load(open(path, encoding="utf-8"))
except Exception as e:
return [("R0", f"results JSON unreadable: {e}")]
if not d.get("caveats"):
findings.append(("L4", "results JSON has no caveats — every "
"experiment has known limitations; list "
"them"))
tests = d.get("tests") or []
if not tests and not d.get("descriptive_only"):
findings.append(("L5", "no tests in results (set "
"descriptive_only: true if intentional)"))
elif tests and not all("p" in t and "n" in t for t in tests):
findings.append(("L5", "tests missing p/n fields"))
for keypath, s in _walk_strings(d):
if len(s) > 240 and s.count(" ") > 25 \
and not (set(keypath) & PROSE_KEYS):
findings.append(("L6", f"sentence-length string outside prose "
f"keys at {'/'.join(keypath)}: possible "
f"raw-text leak"))
break
return findings
def lint(script, results):
return lint_script(script) + lint_results(results)
def run_cases():
here = os.path.dirname(os.path.abspath(__file__))
cases = json.load(open(os.path.join(here, "cases", "cases.json"),
encoding="utf-8"))
failures = 0
for c in cases:
f = lint(os.path.join(here, "cases", c["script"]),
os.path.join(here, "cases", c["results"]))
got = sorted({code for code, _ in f})
want = sorted(c["expect_codes"])
ok = got == want
print(f" {'PASS' if ok else 'FAIL'} {c['name']}: "
f"expected {want}, got {got}")
if not ok:
failures += 1
print(f"{len(cases) - failures}/{len(cases)} eval cases pass")
return failures
if __name__ == "__main__":
if len(sys.argv) == 3:
fs = lint(sys.argv[1], sys.argv[2])
for code, msg in fs:
print(f" {code}: {msg}")
print("CLEAN" if not fs else f"{len(fs)} findings")
sys.exit(1 if fs else 0)
sys.exit(1 if run_cases() else 0)
Mode: cross-validate
Two layers: adversarial code review (every experiment) and external model review (major claims). Both earned their place by finding killers that internal passes missed.
Layer 1 — adversarial code review (mandatory per experiment batch)
Use an independent code-review agent (e.g. Codex CLI read-only):
echo "<review prompt>" | codex exec --skip-git-repo-check --sandbox read-only -C <repo> 2>/dev/nullPrompt template: name the files; direct focus to (1) statistical correctness — permutation universe, lag alignment through data gaps, BH family handling, residualization validity; (2) data handling — timezones, zero-vs-missing, cache staleness, category matching, silent drops; (3) selection bias. Ask for numbered findings with severity and one-line fixes; forbid edits.
Triage discipline: fix mechanical bugs; for methodological findings that do not flip conclusions (e.g. anti-conservative approximations under a null), add a disclosure note to the results JSON instead of re-architecting. Re-run affected experiments after fixes. Track findings-fixed counts in the report footer.
Layer 2 — external model review (major claims / program reviews)
An external frontier model (e.g. ChatGPT Pro via browser automation, or any strong model with code execution) reviews the program with FULL methods and AGGREGATE results — never raw personal text/audio.
Archive packaging rules:
- include: experiment scripts, results JSONs (statistics), verdicts,
pre-registration prompts, a dense pipeline summary (read-first file);
- exclude: transcripts, dictation/diary text, audio, embeddings caches;
- scan before upload: flag any string value >240 chars with >25 spaces
(sentence detector) — methodology prose is fine, quoted data is not.
Ask for four deliverables in order: (1) adversarial review naming fragile claims and uncontrolled confounds; (2) literature mapping to computable measures; (3) analyses it can run itself on the included aggregates; (4) N prioritized, concretely executable follow-up goals (hypothesis, data, method, power, priority each).
Treat its kills seriously: an external exact recompute of a flagship result (impossible p=0.0005 at n=19) is what triggered the audit layer.
Layer 3 — convergence as validation
The strongest evidence in observational self-data is CROSS-CHANNEL convergence: two independent instruments (different sources, different methods) agreeing on the same construct (e.g. diary embedding axis ↔ dictation lexicon, r≈0.21, q≈0.03). Prefer designs that admit a convergence test; a channel that converges with nothing after ~200 shared days is suspect.
Mode: validate-data
Gate every new data source through this checklist BEFORE designing against it. Each item is a bug class that actually occurred and silently corrupted results until caught.
The checklist
1. Zero vs missing. A day with zero marker hits is a real 0.0 rate, not a missing day. Dict comprehensions over hit-counters silently drop zero-keys (n collapsed from 234 to 35 once). Initialize all expected keys explicitly. 1b. Degenerate units / leverage points. A unit with a near-zero denominator (empty/failed transcript, 1-token session) makes every per-1k or ratio read exactly 0 — and then sits at a corner of every scatter as a maximum-leverage point that can carry a whole correlation. One n_tokens=1 session inflated a headline lag from r=−0.32 (n.s.) to r=−0.64 (p=0.03); removing it collapsed the "central finding." Filter units below a sane size threshold BEFORE any per-unit analysis, and eyeball the scatter for single points pinned to an axis. Also: a "size" variable that does not vary in reality (e.g. transcript length when all sessions are a fixed 60 min — it measures recording completeness, not dose) must NOT be used as a covariate or alternative mechanism. Confirm a variable is real before controlling for it. 2. Dedup semantics. Know what one row means. A UNIQUE(url, device) constraint makes "visits/day" actually "NEW unique URLs/day" — revisits invisible. Rename the measure accordingly. 3. Category matching. Substring rules over identifiers are booby-trapped: "code" matched ru.keepcoder.Telegram and silently reclassified 1,094 human messages as coding (audience analysis ran on n=8 until a review caught it). Match on specific tokens; order rules by specificity; verify per-category counts against raw counts. 4. Timezone & day boundaries. Establish the storage convention (UTC? local-at-write? offset column with JS sign convention local=utc−offset?) and the DST behavior of fallbacks before computing hour-of-day or day-boundary features. 5. Retention windows. Check how long the source keeps data (a dictation app kept audio only ~2 weeks — rolling purge). If data expires, build the harvester FIRST, analysis later. 6. Coverage map per field. Fields appear/disappear with app versions (speechDuration on 102/379 days; corrections on 49). Print per-field monthly coverage before designing tests on them. 7. Format/granularity strata. Mixed export formats (merged turns vs utterance-level) make raw counts incomparable — z-score within format or analyze within stratum. 8. Cache invalidation. Caches keyed by age go stale wrong; key them by source mtime + schema version + date window. Thread --refresh through every consumer. 9. Silent exclusions. Count and report every dropped row by reason (bad timestamp, empty text, short clip). "Processed 0 entries" looked like success in a sync log for six months. 10. Instrumentation breaks vs life breaks. Before interpreting a regime change, check for device non-wear, app switches, schema changes, sync outages at the same date. A channel whose "break" coincides with an instrumentation change is flagged, not interpreted. 11. Identifier drift. Hostnames, spellings of names, app bundle ids drift over time (mDNS rename broke a sync silently; a coach's name had three spellings). Match leniently, log what matched. 12. Missingness mechanism. Test whether missingness itself tracks time, outcomes or predictors (missing-not-at-random): correlate the observed/missing indicator with the key series. If days are missing because of the state being studied (e.g. no dictation on bad days), complete-case correlations are biased — report the sensitivity. 13. Positive control. Find one event the source MUST see (a documented move, a vacation). If it can't detect that, its nulls are void.
Validation output
A short validation memo: rows, window, per-field coverage, dedup semantics, timezone convention, retention, known strata, exclusion counts, positive-control result, and the list of measures that are SAFE to design against.
Mode: design
Produce a pre-registered experiment plan BEFORE any data contact beyond coverage checks.
Output of this mode
A docstring-ready block containing:
1. Hypothesis — directional, mechanism-flavored, falsifiable. State what a positive AND a null would mean (if a null teaches nothing, redesign). 2. Tests — exact list, each with: predictor, outcome, lag structure, test statistic, permutation scheme. 5–10 tests max per family. 3. Family size m — fixed now, as a LITERAL CONSTANT, never len(tests). Declare it; assert the run matches; pass the literal to bh(). Computing m from the result count defeats pre-registration and the linter rejects it. Idiom:
FAMILY_M = {"A": 4, "B": 4} # pre-registered, by hand
got = {f: len(v) for f, v in fams.items()}
assert got == FAMILY_M, f"generation drift: {got} != {FAMILY_M}"
bh(fams["A"], m=4) # literal, not len()Confirmatory family (1–5 tests, things to be believed) separate from exploratory family (the sweep). 4. Thresholds — confirmatory: exact q<0.10; lead: exact p<0.06. 5. Power sanity — n available; smallest detectable r at 80% power (~0.21 at n=180, ~0.15 at n=365 for daily series — iid two-sided α=0.05 approximations; autocorrelation, missingness and multiplicity all reduce effective power below these). For session sequences of n≈20–30 only |r|>0.5 is detectable and permutation resolution is 1/n — say so. 6. Positive control where possible — one test that MUST fire if the instrument works (e.g. "apartment-browsing share must change around a documented move"). If the positive control fails, nulls are uninterpretable. 7. Negative controls / placebo outcomes — one outcome the predictor must NOT move (and/or placebo dates). A predictor that "works" on the negative control is measuring confounding or instrumentation, not the mechanism.
Blind sweeps cannot beat FDR at small n
A large undirected sweep (e.g. every marker × every outcome) at session scale (n≈20–60) will produce leads at exactly the chance rate and none will survive FDR — a generation burned to relearn this. Two honest designs instead:
- Small theory-tagged confirmatory families (this is the default):
each test derived from a named theory, m fixed and tiny.
- An explicitly EXPLORATORY sweep whose deliverable is the *p-value
distribution* (is the leftmost bin above the uniform line?), NOT a lead list. Register it as exploratory; survivors are candidates, never confirmed; downstream is investigate-leads, not publication.
Design heuristics (earned the hard way)
- Three methods beat one: session-level correlation, within-unit
event study, and multivariate prediction answer different questions and fail differently. An event study over thousands of within-unit events has real power when n_sessions=20 does not.
- Lagged designs: pair adjacent-by-date units; record gap length as a
covariate. "t → t+1" over irregular spacing weakens interpretation.
- Name the collider up front: volume/usage of a channel usually
correlates with workload and with the markers computed from it.
- Anticipation matters for life events: plannable events (moves, job
changes) show preparation signatures months before the date — design windows around the preparation period, not just the event date.
- Prospective protocols: one primary predictor, one primary outcome,
test ONCE at a pre-committed n; include a kill/decision rule and an adherence criterion for interventions. Randomize from a recorded seed and paste the literal generated schedule into the protocol.
- Audience/register strata: text markers differ by who the text is
for (AI vs humans vs self). If the corpus mixes registers, plan a stratified or within-day paired contrast.
Mode: investigate-leads
After an exploratory sweep or any run produces "leads" (p<0.06, not FDR-confirmed), investigate them — do NOT list them as findings and do NOT re-test them at the same scale. The goal is triage: separate the few candidates worth prospective testing from small-n and regime-step artifacts.
The core distinction
Robust ≠ significant. A lead that survives leave-one-out at n≈26–31 is still underpowered for confirmation. Most leads from a large sweep are "robust-but-underpowered" — honest candidates, not findings. Only prospective replication (a frozen protocol, tested once at a pre-committed n) confirms.
The battery (pre-register which apply per lead)
Run scripts/triage.py for the project-agnostic checks; add the trend/daily checks inline.
1. Leave-one-out robustness. Drop each unit once. FRAGILE if the sign flips OR |r| more than halves on any single deletion — a small-n artifact. (Necessary, not sufficient: passing LOO ≠ significant.) 2. Directionality (lag leads). The reverse direction must be weaker. If reverse |r| ≥ forward |r|, the causal reading dies. 3. Detrend vs regime-step (trend leads). A "gradual trend" whose two halves have opposite-sign slopes is a STEP, not a trend (this killed an r=−0.78 lead). Require same-sign slope in both halves; ols-detrend and confirm the residual no longer trends. 4. Within-stratum / Simpson (cross-group or cross-cycle trends). Re-fit the slope WITHIN each stratum; a cross-stratum trend whose within-stratum slopes disagree is a between-regime artifact. 5. Prewhiten + bootstrap (daily/trending cross-series). Re-test on AR1 residuals and under stationary block bootstrap. A lead that holds after prewhitening is a real relationship, not shared trend — this is how a lead gets genuinely UPGRADED, not just survived.
Consolidation (the high-value move)
Several weak leads pointing the same way are often ONE construct. Combine them into a single z-scored composite and test that — it usually has more power than any component. Example: three separate therapy trends (absolutist↑, negation↑, hedge↓) consolidated into one z(absolutist)+z(negation)−z(hedge) certainty/rigidity index at r=0.66, stronger than any of its parts. A composite is a cleaner prospective target than three fragile correlations — and it is theory-shaped.
Output
A triage table with one verdict per lead:
- known — re-confirms an established effect (directional, robust);
- strengthened — survives prewhitening/bootstrap → upgrade;
- candidate — robust but underpowered → prospective protocol;
- artifact — small-n / regime-step / Simpson; killed.
Mint NO new "confirmed" here — the battery is diagnostic. Mark the results file descriptive_only: true. Promote candidates only via a prospective protocol.
Mode: conduct (+ §Audit)
Implement with scripts/perm_stats.py (battle-tested; copy into the project). API: exact_circ_p(x, y) — Pearson r with exact circular-shift permutation over full-calendar series with None gaps (rolls y; pass the COMPLETE series as y so missingness stays attached to its own series); exact_event_diff(indicator, values, step=1) — SD-unit event contrast, indicator strictly 0/1 (assert enforced), step=7 preserves weekday structure; break_diff(values, cut_idx, min_side=30) — level step at a known date vs ALL non-wrapping placebo cuts; bh(tests, m) — BH with fixed family size.
Permutation correctness checklist
- Universe size: session sequence n → n−1 shifts → min p = 1/n. If the
guard returns p=None (universe <12), the test is unpowered — report it as such, never substitute a sampled test.
- Daily series: build the CONTIGUOUS calendar from first to last day;
gaps as None. d+1 must mean the next calendar day, not the next row.
- Break tests: circularly rolled "before/after" indicators wrap around
the calendar and are invalid cutpoints — enumerate non-wrapping cuts.
- Event studies near weekly-structured events: placebo shifts in
multiples of 7.
- Residualize-then-permute is a Freedman–Lane approximation
(anti-conservative). Acceptable when the conclusion is a null (a fortiori); for positive claims refit per permutation.
- Sign-flip permutation for paired within-unit contrasts is fine sampled
(5000) at n≥100; the exactness rule binds at small n.
Beyond circular shifts (trending/structured series)
- Prewhiten (AR1 residuals) and stationary bootstrap alongside exact
perm for any cross-series claim; report all three. A claim that dies under prewhitening was a shared trend.
- Stratify by group and regime before pooling (Simpson). Run within-kind
and within-regime; pool only if signs agree.
- Granger/lead-lag on differenced series, not levels.
Series scope is part of the test definition
A lagged "[t+1]" means the next unit in the series the hypothesis is about — not the next row of a pooled frame. Computing "next-session valence" over a POOLED therapy+coaching sequence once flipped the sign vs the therapy-only definition (+0.12 p=0.77 → −0.585 p=0.033): coaching sessions interleaved between therapy ones broke the offsets. Define the scope before lagging. When recomputing a prior result, FIRST reproduce a stored artifact (an offset value, a count) on the chosen scope as a sanity check; only then run the test.
Results JSON conventions
{experiment, hypothesis, method, n_*, tests: [{h, desc, r, p, q, n}], caveats: [...]} — statistics only, no raw text. For every effect that will be DEFENDED (confirmed/lead), include an uncertainty interval (bootstrap or permutation CI), not just p. Round for display but keep BH on full precision when families are large. Always include the caveats list; every known limitation goes in.
§Audit (re-examining past claims)
1. Build/refresh a findings registry: every test as {exp, family, id, effect, p, q, n, status} with statuses: confirmed / lead / null / descriptive. 2. Impossible-p detector: for sampled-permutation session tests, reported p < 1/n is unattainable exactly → flag and recompute. 2b. Sweep diagnostic: for any exploratory sweep, the honest summary is the p-value HISTOGRAM, not the lead list. Mostly-null → ~uniform, leftmost bin at the chance line; real signal → leftmost bin far above it. Report the lead count against 0.06 × n_tests (expected by chance) — if they match, you found nothing, and say so. 3. Recompute exactly from stored per-unit data where possible; BH with the ORIGINAL family size; record status flips with provenance (status_original, audited fields) — never silently edit. 4. When a control collapses an effect, decompose: re-run with each control alone, then check r(predictor, killing control). High collinearity → "mechanism ambiguous", not "artifact". 5. External recomputation (another model/person, aggregate data only) is the strongest audit — it found what three internal reviews missed.
#!/usr/bin/env python3
"""Experiment results explorer — launchable viewer over a directory of
results JSONs. Part of the rigorous-experiments skill.
Usage:
python3 explorer.py <results_dir> [--port 8799] [--pattern "exp*.json"]
[--sort newest|oldest] [--no-open] [--no-serve]
What it does:
1. Scans <results_dir> for experiment results files (default pattern
exp*.json, verdict files excluded) and builds a manifest with
confirmed/lead counts per experiment (q<0.10 / p<0.06). "Creation
date" = st_birthtime on macOS; on filesystems without birthtime it
falls back to mtime (re-runs reorder the list there).
2. Links full-text reports: every *.html in the directory is scanned
for experiment ids; matching reports are linked from each
experiment's detail view.
3. Writes explorer.html into <results_dir>. The page fetches result
files live (same origin), so re-running experiments updates the view
without regenerating; regenerate only when NEW files appear.
4. Ensures a local http server is serving <results_dir> on --port
(starts one bound to 127.0.0.1 if the port is free; reuses a server
only after verifying it serves THIS directory) and opens the browser.
UI: system sans-serif; resizable sidebar (drag the divider, width
persisted); star experiments (★, persisted in localStorage per
directory) and filter by starred; every scalar/dict/list field of a
results JSON is rendered (facts table, definition lists like
"approaches", bullet lists) — nothing meaningful hides in raw JSON only.
LOCAL-ONLY: serves on loopback. Results files following the skill's
conventions ({experiment, hypothesis, method, tests:[{h,desc,r,p,q,n}],
caveats}) render richest; unknown shapes degrade gracefully.
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import re
import socket
import subprocess
import sys
import webbrowser
HTML = """<!DOCTYPE html>
<html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Experiment explorer</title>
<style>
:root{--ink:#1a1a1a;--bg:#fdfbf7;--muted:#6b6b6b;--rule:#d8d2c4;
--green:#2a7a5a;--amber:#c89000;--red:#a02a2a;--purple:#5a5aaa}
*{box-sizing:border-box}
body{font-family:-apple-system,BlinkMacSystemFont,'Helvetica Neue',Helvetica,Arial,sans-serif;
background:var(--bg);color:var(--ink);margin:0;font-size:15px;line-height:1.5}
.layout{display:grid;grid-template-columns:var(--sidew,420px) 6px 1fr;height:100vh}
.side{border-right:1px solid var(--rule);overflow-y:auto;padding:1rem;min-width:240px}
#drag{cursor:col-resize;background:transparent}
#drag:hover{background:rgba(90,90,170,.15)}
.main{overflow-y:auto;padding:1.4rem 2rem}
h1{font-size:1.15rem;margin:.2rem 0 .8rem;font-weight:600}
input#q{width:100%;font:inherit;padding:.4rem .6rem;border:1px solid var(--rule);
border-radius:6px;background:#fff;margin-bottom:.5rem}
select#sortsel{width:100%;font:inherit;font-size:.8rem;margin-bottom:.45rem;
padding:.3rem;border:1px solid var(--rule);border-radius:6px;background:#fff}
#chips{display:flex;flex-wrap:wrap;gap:.25rem;margin-bottom:.6rem}
#chips button{font:inherit;font-size:.72rem;padding:.12rem .55rem;
border:1px solid var(--rule);border-radius:10px;background:#fff;cursor:pointer}
.exp{padding:.5rem .6rem;border-radius:7px;cursor:pointer;margin:.18rem 0;
border:1px solid transparent;position:relative}
.exp:hover{background:#f6f2e8}
.exp.active{border-color:var(--purple);background:#fff}
.exp .id{font-family:ui-monospace,Menlo,monospace;font-size:.7rem;color:var(--muted)}
.exp .t{font-size:.82rem;line-height:1.3;display:block;margin-top:.1rem}
.star{position:absolute;top:.35rem;right:.4rem;cursor:pointer;font-size:.9rem;
color:#c8c2b4;user-select:none}
.star.on{color:var(--amber)}
.badge{display:inline-block;font-family:ui-monospace,Menlo,monospace;font-size:.65rem;
border-radius:8px;padding:0 .4rem;margin-left:.25rem;color:#fff}
.b-c{background:var(--green)}.b-l{background:var(--amber)}
h2{font-size:1.3rem;margin:.2rem 0 .5rem;font-weight:650}
h3{font-size:.95rem;margin:1.4rem 0 .3rem;font-weight:650}
.meta{font-size:.86rem;color:#444;margin:.4rem 0 .8rem}
.meta b{color:var(--ink)}
table{border-collapse:collapse;width:100%;font-size:.82rem;
margin:1.4rem 0 1.6rem}
th{text-align:left;border-bottom:2px solid var(--ink);padding:.35rem .5rem;
cursor:pointer;user-select:none;white-space:nowrap;font-size:.75rem;
text-transform:uppercase;letter-spacing:.04em;color:#444}
td{border-bottom:1px solid var(--rule);padding:.35rem .5rem}
td.num{font-family:ui-monospace,Menlo,monospace;font-size:.76rem;text-align:right;
white-space:nowrap}
tr.confirmed td{background:rgba(42,122,90,.16);font-weight:600}
tr.confirmed td:first-child{border-left:4px solid var(--green)}
tr.lead td{background:rgba(200,144,0,.10)}
tr.lead td:first-child{border-left:4px solid var(--amber)}
tr.null td{color:#777}
tr.null td:first-child{border-left:4px solid transparent}
.st-confirmed{color:var(--green);font-weight:700}
.st-lead{color:var(--amber);font-weight:600}
.st-null{color:#999}
.caveats{border-left:3px solid var(--amber);background:#fffaf0;
padding:.6rem 1rem;font-size:.83rem;margin:1.4rem 0}
.verdict{border-left:3px solid var(--purple);background:#fff;
padding:.6rem 1rem;font-size:.88rem;margin:.8rem 0}
.kv{display:grid;grid-template-columns:max-content 1fr;gap:.15rem .9rem;
font-size:.84rem;margin:.5rem 0 1rem}
.kv dt{font-family:ui-monospace,Menlo,monospace;font-size:.74rem;
color:var(--muted);padding-top:.1rem}
.kv dd{margin:0}
.facts{display:flex;flex-wrap:wrap;gap:.3rem;margin:.6rem 0 1rem}
.fact{font-size:.74rem;border:1px solid var(--rule);border-radius:6px;
background:#fff;padding:.15rem .5rem}
.fact b{font-family:ui-monospace,Menlo,monospace;font-weight:600}
.reports{margin:.5rem 0 .9rem}
.reports a{display:inline-block;font-size:.76rem;border:1px solid var(--purple);
color:var(--purple);border-radius:10px;padding:.1rem .6rem;margin:0 .3rem .3rem 0;
text-decoration:none}
.reports a:hover{background:var(--purple);color:#fff}
ul.plain{margin:.3rem 0 1rem;padding-left:1.2rem;font-size:.84rem}
details{margin:1.4rem 0}
summary{cursor:pointer;font-size:.85rem;color:var(--muted)}
pre{background:#fff;border:1px solid var(--rule);border-radius:6px;
padding:.8rem;font-size:.72rem;overflow-x:auto;max-height:50vh}
.note{color:var(--muted);font-size:.88rem}
</style></head><body>
<div class="layout" id="layout">
<div class="side">
<h1 id="title">Experiments <span style="font-size:.75rem;color:var(--muted)" id="count"></span></h1>
<div id="viewtoggle" style="display:flex;gap:.25rem;margin-bottom:.5rem">
<button data-v="exp" style="flex:1;font:inherit;font-size:.75rem;padding:.25rem;border:1px solid var(--rule);border-radius:6px;cursor:pointer">experiments</button>
<button data-v="tests" style="flex:1;font:inherit;font-size:.75rem;padding:.25rem;border:1px solid var(--rule);border-radius:6px;cursor:pointer">all tests</button>
</div>
<input id="q" placeholder="filter…">
<select id="sortsel">
<option value="newest">newest first</option>
<option value="oldest">oldest first</option>
<option value="expnum">by experiment №</option>
<option value="tests">most tests</option>
<option value="confirmed">most confirmed</option>
<option value="leads">most leads</option>
</select>
<div id="chips"></div>
<div id="list"></div>
</div>
<div id="drag"></div>
<div class="main" id="main"><p class="note">← pick an experiment.
Green badge — confirmed (q<0.10), amber — lead (p<0.06).
★ stars an experiment; the «starred» chip filters to your stars.</p></div>
</div>
<script>
const M=__DATA__;
const esc=v=>String(v??'').replace(/&/g,'&').replace(/</g,'<')
.replace(/>/g,'>').replace(/"/g,'"');
const list=document.getElementById('list'),main=document.getElementById('main');
let active=null;
// ---- resizable sidebar (width persisted per directory)
const WKEY='explorer-width-'+M.dir_token;
const saved=localStorage.getItem(WKEY);
if(saved)document.getElementById('layout').style.setProperty('--sidew',saved+'px');
(()=>{const drag=document.getElementById('drag');let on=false;
drag.addEventListener('mousedown',()=>{on=true;document.body.style.userSelect='none';});
window.addEventListener('mousemove',e=>{if(!on)return;
const w=Math.max(240,Math.min(700,e.clientX));
document.getElementById('layout').style.setProperty('--sidew',w+'px');});
window.addEventListener('mouseup',()=>{if(on){on=false;
document.body.style.userSelect='';
const w=getComputedStyle(document.getElementById('layout'))
.getPropertyValue('--sidew').trim().replace('px','');
localStorage.setItem(WKEY,w);}});})();
// ---- stars (persisted per directory)
const SKEY='explorer-stars-'+M.dir_token;
let stars=new Set(JSON.parse(localStorage.getItem(SKEY)||'[]'));
function toggleStar(file){
stars.has(file)?stars.delete(file):stars.add(file);
localStorage.setItem(SKEY,JSON.stringify([...stars]));
render(document.getElementById('q').value);
}
let dir=M.sort||'newest';
document.getElementById('sortsel').value=dir;
document.getElementById('sortsel').onchange=e=>{dir=e.target.value;
render(document.getElementById('q').value);};
const CHIPS=[
['all','all',e=>true],
['starred','★ starred',e=>stars.has(e.file)],
['tests','with tests',e=>e.n_tests>0],
['confirmed','confirmed ✓',e=>e.n_confirmed>0],
['leads','leads',e=>e.n_leads>0],
['nulls','nulls only',e=>e.n_tests>0&&!e.n_confirmed&&!e.n_leads],
['notests','no tests',e=>e.n_tests===0]];
let chip='all';
const chipsEl=document.getElementById('chips');
for(const [id,label] of CHIPS.map(c=>[c[0],c[1]])){
const b=document.createElement('button');
b.textContent=label;b.dataset.id=id;
b.onclick=()=>{chip=id;render(document.getElementById('q').value);};
chipsEl.appendChild(b);
}
function expnum(e){const m=e.exp.match(/\\d+/);return m?+m[0]:1e9;}
const SORTS={
newest:(a,b)=>b.created-a.created,
oldest:(a,b)=>a.created-b.created,
expnum:(a,b)=>expnum(a)-expnum(b),
tests:(a,b)=>b.n_tests-a.n_tests,
confirmed:(a,b)=>b.n_confirmed-a.n_confirmed||b.n_leads-a.n_leads,
leads:(a,b)=>b.n_leads-a.n_leads||b.n_confirmed-a.n_confirmed};
function render(filter){
list.innerHTML='';
for(const b of chipsEl.children)
b.style.background=b.dataset.id===chip?'var(--purple)':'#fff',
b.style.color=b.dataset.id===chip?'#fff':'inherit';
const f=(filter||'').toLowerCase();
const pred=CHIPS.find(c=>c[0]===chip)[2];
const items=[...M.manifest].sort(SORTS[dir]||SORTS.newest);
let shown=0;
for(const e of items){
if(!pred(e))continue;
if(f && !(e.file+' '+e.title).toLowerCase().includes(f))continue;
shown++;
const div=document.createElement('div');
div.className='exp'+(active===e.file?' active':'');
div.innerHTML=`<span class="id">${e.exp} · ${e.created_h} · ${e.n_tests} tests`+
(e.n_confirmed?`<span class="badge b-c">${e.n_confirmed}</span>`:'')+
(e.n_leads?`<span class="badge b-l">${e.n_leads}</span>`:'')+
`</span><span class="t">${esc(e.title)}</span>`+
`<span class="star${stars.has(e.file)?' on':''}" data-f="${esc(e.file)}">`+
(stars.has(e.file)?'★':'☆')+`</span>`;
div.onclick=ev=>{
if(ev.target.classList.contains('star')){toggleStar(e.file);
ev.stopPropagation();return;}
show(e);};
list.appendChild(div);
}
document.getElementById('count').textContent='('+shown+'/'+M.manifest.length+')';
}
document.getElementById('q').oninput=e=>render(e.target.value);
// mirrors Python extract_tests(): p/q-like keys, lists + standalone dicts
const P_AL=new Set(['p','p_band','perm_p','exact_p','pval','p_value']);
const Q_AL=new Set(['q','q_value','q_bh','fdr_q']);
function pq(x){
let p=null,q=null,has=false;
for(const [k,v] of Object.entries(x)){
if(typeof v!=='number'&&v!==null)continue;
if(Q_AL.has(k)){q=v;has=true;}
else if(P_AL.has(k)){p=v;has=true;}
}
return {has,p,q};
}
function findTests(o,path,acc){
acc=acc||[];path=path||[];
if(Array.isArray(o)){
const dicts=o.filter(x=>x&&typeof x==='object'&&!Array.isArray(x));
const hits=dicts.filter(x=>pq(x).has);
if(hits.length){
for(const x of hits){const r=pq(x);
acc.push({item:x,p:r.p,q:r.q,path:path.join('.'),top:path[0]||''});}
return acc;
}
for(const x of o)findTests(x,path,acc);
}else if(o&&typeof o==='object'){
const r=pq(o);
if(r.has&&!Object.values(o).some(v=>v&&typeof v==='object')){
acc.push({item:o,p:r.p,q:r.q,path:path.join('.'),top:path[0]||''});
return acc;
}
for(const [k,v] of Object.entries(o))
findTests(v,path.length<6?path.concat(k):path,acc);
}
return acc;
}
const NUMK=new Set(['r','p','q','n','effect','beta','F','tau','d','slope']);
function testDesc(w){
const t=w.item;
const named=t.desc??t.label??t.relation;
if(named)return String(named);
if(t.from!==undefined&&t.to!==undefined)return t.from+' \\u2192 '+t.to;
const parts=[];
for(const [k,v] of Object.entries(t))
if(typeof v==='string'&&!NUMK.has(k)&&k!=='h'&&k!=='id')parts.push(v);
if(parts.length)return parts.join(' \\u00b7 ');
return w.path||'(unnamed test)';
}
function testEffect(w){
const t=w.item;
for(const k of ['r','effect','d','beta','F','tau','slope'])
if(t[k]!==undefined&&t[k]!==null)return k+'='+t[k];
return '';
}
function status(w){
if(w.q!=null&&w.q<0.10)return 'confirmed';
if(w.p!=null&&w.p<0.06)return 'lead';
return w.p!=null?'null':'desc';
}
// ---- generic field rendering: nothing meaningful hides in raw JSON
const HANDLED=new Set(['experiment','title','hypothesis','goal','method',
'verdict','verdict_summary','caveats','tests']);
function renderExtras(d,testTops){
let h='';
const facts=[];
for(const [k,v] of Object.entries(d)){
if(HANDLED.has(k)||testTops.has(k))continue;
if(v===null)continue;
if(typeof v==='number'||typeof v==='boolean'||
(typeof v==='string'&&v.length<=80)){
facts.push(`<span class="fact"><b>${esc(k)}</b>: ${esc(v)}</span>`);
}
}
if(facts.length)h+=`<h3>Facts</h3><div class="facts">${facts.join('')}</div>`;
for(const [k,v] of Object.entries(d)){
if(HANDLED.has(k)||testTops.has(k))continue;
if(typeof v==='string'&&v.length>80){
h+=`<h3>${esc(k)}</h3><p class="meta">${esc(v)}</p>`;
}else if(Array.isArray(v)&&v.length&&v.every(x=>typeof x==='string')){
h+=`<h3>${esc(k)}</h3><ul class="plain">`+
v.map(x=>`<li>${esc(x)}</li>`).join('')+'</ul>';
}else if(v&&typeof v==='object'&&!Array.isArray(v)){
const ent=Object.entries(v);
if(ent.length&&ent.every(([kk,vv])=>typeof vv==='string'||
typeof vv==='number'||typeof vv==='boolean')){
// dicts like "approaches": {A: "...", B: "..."} -> definition list
h+=`<h3>${esc(k)}</h3><dl class="kv">`+
ent.map(([kk,vv])=>`<dt>${esc(kk)}</dt><dd>${esc(vv)}</dd>`).join('')+
'</dl>';
}else if(ent.length){
h+=`<h3>${esc(k)}</h3><p class="note">nested object `+
`(${ent.length} keys) — see raw JSON below</p>`;
}
}else if(Array.isArray(v)&&v.length){
h+=`<h3>${esc(k)}</h3><p class="note">${v.length} records — `+
`see raw JSON below</p>`;
}
}
return h;
}
let sortKey=null,sortAsc=true,reqToken=0;
function show(e,keepSort){
active=e.file;render(document.getElementById('q').value);
const tok=++reqToken;
fetch(encodeURIComponent(e.file)+'?'+Date.now()).then(r=>r.json()).then(d=>{
if(tok!==reqToken)return; // stale response from a faster earlier click
if(!keepSort){sortKey=null;}
const tests=findTests(d);
const testTops=new Set(tests.map(w=>w.top).filter(Boolean));
if(sortKey)tests.sort((a,b)=>{
const get=w=>sortKey==='p'?w.p:sortKey==='q'?w.q:
sortKey==='n'?w.item.n:sortKey==='status'?status(w):
sortKey==='effect'?testEffect(w):sortKey==='desc'?testDesc(w):
(w.item.h??w.item.id);
const av=get(a),bv=get(b);
if(av==null)return 1;if(bv==null)return -1;
return (av<bv?-1:av>bv?1:0)*(sortAsc?1:-1);});
let h=`<h2>${esc(e.exp)} · ${esc(d.title||d.experiment||e.file)}</h2>`;
if(e.reports&&e.reports.length)
h+='<div class="reports">Reports: '+e.reports.map(r=>
`<a href="${esc(r)}" target="_blank">${esc(r.replace('.html',''))}</a>`)
.join('')+'</div>';
if(d.hypothesis)h+=`<div class="meta"><b>Hypothesis:</b> ${esc(d.hypothesis)}</div>`;
else if(d.goal)h+=`<div class="meta"><b>Goal:</b> ${esc(d.goal)}</div>`;
if(d.method)h+=`<div class="meta"><b>Method:</b> ${esc(d.method)}</div>`;
const verd=d.verdict||d.verdict_summary;
if(verd)h+=`<div class="verdict"><b>Verdict:</b> ${
esc(typeof verd==='string'?verd:JSON.stringify(verd))}</div>`;
if(tests.length){
h+='<table><tr>';
for(const k of ['h','desc','effect','p','q','n','status'])
h+=`<th data-k="${k}">${k}${sortKey===k?(sortAsc?' \\u2191':' \\u2193'):''}</th>`;
h+='</tr>';
const SLBL={confirmed:'\\u2713 confirmed',lead:'lead','null':'null',desc:'\\u2014'};
for(const w of tests){
const st=status(w);
h+=`<tr class="${st}"><td class="num">${esc(w.item.h??w.item.id??'')}</td>`+
`<td>${esc(testDesc(w))}</td>`+
`<td class="num">${esc(testEffect(w))}</td>`+
`<td class="num">${w.p??''}</td><td class="num">${w.q??''}</td>`+
`<td class="num">${w.item.n??''}</td>`+
`<td class="num st-${st}">${SLBL[st]}</td></tr>`;
}
h+='</table>';
}else{
h+='<p class="note">No inferential tests detected in this results '+
'file \\u2014 descriptive layer. See fields and raw JSON below.</p>';
}
h+=renderExtras(d,testTops);
if(d.caveats&&d.caveats.length)
h+='<div class="caveats"><b>Caveats:</b><br>'+
d.caveats.map(c=>'\\u2022 '+esc(c)).join('<br>')+'</div>';
h+=`<details><summary>raw JSON (${esc(e.file)})</summary><pre>${
JSON.stringify(d,null,1).replace(/&/g,'&').replace(/</g,'<')
.slice(0,200000)}</pre></details>`;
main.innerHTML=h;
main.querySelectorAll('th').forEach(th=>th.onclick=()=>{
const k=th.dataset.k;
if(sortKey===k)sortAsc=!sortAsc;else{sortKey=k;sortAsc=true;}
show(e,true);});
}).catch(err=>{main.innerHTML='<p>failed to load '+esc(e.file)+': '+esc(err)+'</p>'});
}
// ---- ALL-TESTS view: flatten every test from every file into one table
let view='exp', allTests=null, tSortKey='p', tSortAsc=true;
async function loadAllTests(){
if(allTests)return allTests;
main.innerHTML='<p class="note">loading all tests…</p>';
const out=[];
await Promise.all(M.manifest.map(e=>
fetch(encodeURIComponent(e.file)+'?'+Date.now()).then(r=>r.json())
.then(d=>{for(const w of findTests(d))
out.push({exp:e.exp,file:e.file,desc:testDesc(w),
effect:testEffect(w),p:w.p,q:w.q,n:w.item.n,st:status(w)});})
.catch(()=>{})));
allTests=out; return out;
}
function renderTests(){
const f=(document.getElementById('q').value||'').toLowerCase();
const pred=({all:()=>1,starred:t=>stars.has(t.file),
tests:()=>1,confirmed:t=>t.st==='confirmed',
leads:t=>t.st==='lead',nulls:t=>t.st==='null',
notests:()=>0})[chip]||(()=>1);
let rows=allTests.filter(t=>pred(t)&&
(!f||(t.exp+' '+t.desc).toLowerCase().includes(f)));
const g=t=>tSortKey==='p'?(t.p==null?2:t.p):tSortKey==='q'?(t.q==null?2:t.q):
tSortKey==='n'?(t.n||0):tSortKey==='effect'?(parseFloat((t.effect||'').split('=')[1])||0):
tSortKey==='exp'?t.exp:tSortKey==='st'?t.st:t.desc;
rows.sort((a,b)=>{const x=g(a),y=g(b);return (x<y?-1:x>y?1:0)*(tSortAsc?1:-1);});
const SL={confirmed:'✓ confirmed',lead:'lead','null':'null',desc:'—'};
let h=`<p class="note">${rows.length} tests across ${M.manifest.length} experiments</p>`;
h+='<table><tr>';
for(const[k,lbl]of[['exp','exp'],['desc','test'],['effect','effect'],
['p','p'],['q','q'],['n','n'],['st','status']])
h+=`<th data-k="${k}">${lbl}${tSortKey===k?(tSortAsc?' ↑':' ↓'):''}</th>`;
h+='</tr>';
for(const t of rows)
h+=`<tr class="${t.st}"><td class="num" style="cursor:pointer" data-f="${esc(t.file)}">${esc(t.exp)}</td>`+
`<td>${esc(t.desc)}</td><td class="num">${esc(t.effect)}</td>`+
`<td class="num">${t.p??''}</td><td class="num">${t.q??''}</td>`+
`<td class="num">${t.n??''}</td><td class="num st-${t.st}">${SL[t.st]}</td></tr>`;
h+='</table>';
main.innerHTML=h;
main.querySelectorAll('th').forEach(th=>th.onclick=()=>{
const k=th.dataset.k;
if(tSortKey===k)tSortAsc=!tSortAsc;else{tSortKey=k;tSortAsc=true;}
renderTests();});
main.querySelectorAll('td[data-f]').forEach(td=>td.onclick=()=>{
const e=M.manifest.find(m=>m.file===td.dataset.f);
if(e){setView('exp');show(e);}});
}
function setView(v){
view=v;
for(const b of document.querySelectorAll('#viewtoggle button'))
b.style.background=b.dataset.v===v?'var(--purple)':'#fff',
b.style.color=b.dataset.v===v?'#fff':'inherit';
document.getElementById('list').style.display=v==='exp'?'':'none';
document.getElementById('sortsel').style.display=v==='exp'?'':'none';
document.getElementById('title').firstChild.textContent=
v==='exp'?'Experiments ':'All tests ';
if(v==='tests'){loadAllTests().then(renderTests);}
else{main.innerHTML='<p class="note">← pick an experiment.</p>';}
}
for(const b of document.querySelectorAll('#viewtoggle button'))
b.onclick=()=>setView(b.dataset.v);
// route filter + chips to the active view
const _origInput=document.getElementById('q').oninput;
document.getElementById('q').oninput=e=>{
if(view==='tests')renderTests();else render(e.target.value);};
const _chipClick=id=>{chip=id;
if(view==='tests')renderTests();else render(document.getElementById('q').value);};
for(const b of chipsEl.children)b.onclick=()=>_chipClick(b.dataset.id);
setView('exp');
render('');
</script>
</body></html>
"""
# EXACT alias whitelist — broad q_*/p_* matching once swallowed feature
# columns like q_rate (question rate) and fabricated q-values.
P_ALIASES = {"p", "p_band", "perm_p", "exact_p", "pval", "p_value"}
Q_ALIASES = {"q", "q_value", "q_bh", "fdr_q"}
def _pq(x):
"""(p, q) values of a dict via alias keys, else (None-marker)."""
p = q = None
has = False
for k, v in x.items():
if not isinstance(v, (int, float, type(None))):
continue
if k in Q_ALIASES:
q, has = v, True
elif k in P_ALIASES:
p, has = v, True
return has, p, q
def extract_tests(d):
"""Universal test discovery, mirrored EXACTLY by findTests() in the
embedded JS: (a) every list whose dict items carry a p/q-like key
(p, q, p_band, perm_p, exact_p, ...); (b) standalone dicts carrying
a p-like key (single pre-registered tests like primary_partial),
named by their JSON path."""
found = []
def walk(o, path):
if isinstance(o, list):
dicts = [x for x in o if isinstance(x, dict)]
hits = [x for x in dicts if _pq(x)[0]]
if hits:
for x in hits:
has, p, q = _pq(x)
found.append({"_p": p, "_q": q, "raw": x,
"path": path})
return
for x in o:
walk(x, path)
elif isinstance(o, dict):
has, p, q = _pq(o)
if has and not any(isinstance(v, (dict, list))
for v in o.values()):
found.append({"_p": p, "_q": q, "raw": o, "path": path})
return
for k, v in o.items():
walk(v, path + [k] if len(path) < 6 else path)
walk(d, [])
return found
def scan_reports(results_dir, exp_ids):
"""Map exp id -> [report html files mentioning it]. Whole-word match
(exp21 must not match exp210)."""
links = {e: [] for e in exp_ids}
for path in sorted(glob.glob(os.path.join(results_dir, "*.html"))):
name = os.path.basename(path)
if name == "explorer.html":
continue
try:
text = open(path, encoding="utf-8", errors="ignore").read()
except OSError:
continue
for e in exp_ids:
if re.search(re.escape(e) + r"(?![0-9a-z])", text):
links[e].append(name)
return links
def build_manifest(results_dir, pattern):
if os.path.isabs(pattern) or ".." in pattern.split(os.sep):
sys.exit("--pattern must be relative without parent traversal")
manifest = []
for path in sorted(glob.glob(os.path.join(results_dir, pattern))):
if os.path.commonpath([os.path.abspath(path), results_dir]) \
!= results_dir:
continue
name = os.path.basename(path)
if "verdicts" in name or name == "explorer.html":
continue
try:
d = json.load(open(path, encoding="utf-8"))
except Exception:
continue
if not isinstance(d, dict):
continue
tests = extract_tests(d)
n_sig = sum(1 for t in tests
if t["_q"] is not None and t["_q"] < 0.10)
n_lead = sum(1 for t in tests
if t["_p"] is not None and t["_p"] < 0.06
and (t["_q"] is None or t["_q"] >= 0.10))
m = re.match(r"([A-Za-z]+\d+[a-z]?)", name)
title = str(d.get("title") or d.get("hypothesis") or d.get("goal")
or d.get("verdict_summary") or d.get("experiment")
or name)[:140]
st = os.stat(path)
created = getattr(st, "st_birthtime", st.st_mtime)
import datetime as _dt
manifest.append({
"file": name, "exp": m.group(1) if m else name,
"title": title,
"n_tests": len(tests), "n_confirmed": n_sig,
"n_leads": n_lead, "created": created,
"created_h": _dt.datetime.fromtimestamp(created)
.strftime("%Y-%m-%d %H:%M"),
})
# link full-text reports to experiments
links = scan_reports(results_dir, sorted({e["exp"] for e in manifest}))
for e in manifest:
e["reports"] = links.get(e["exp"], [])
return manifest
def port_in_use(port):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
return s.connect_ex(("127.0.0.1", port)) == 0
def dir_token(rd):
import hashlib
return hashlib.sha1(rd.encode()).hexdigest()[:12]
def serves_this_dir(port, rd):
"""True if the listener on `port` serves OUR explorer for `rd`."""
import urllib.request
try:
with urllib.request.urlopen(
f"http://127.0.0.1:{port}/explorer.html", timeout=2) as r:
return dir_token(rd) in r.read(100000).decode("utf-8", "ignore")
except Exception:
return False
def main():
ap = argparse.ArgumentParser()
ap.add_argument("results_dir")
ap.add_argument("--port", type=int, default=8799)
ap.add_argument("--pattern", default="exp*.json")
ap.add_argument("--sort", choices=["newest", "oldest"],
default="newest",
help="initial sidebar order by file creation date")
ap.add_argument("--no-open", action="store_true")
ap.add_argument("--no-serve", action="store_true")
a = ap.parse_args()
rd = os.path.abspath(a.results_dir)
if not os.path.isdir(rd):
sys.exit(f"not a directory: {rd}")
manifest = build_manifest(rd, a.pattern)
if not manifest:
sys.exit(f"no experiment results matching {a.pattern} in {rd}")
out = os.path.join(rd, "explorer.html")
# </script>-breakout-safe embedding: escape < > & line separators
payload = json.dumps({"manifest": manifest, "sort": a.sort,
"dir_token": dir_token(rd)},
ensure_ascii=False)
payload = (payload.replace("&", "\\u0026").replace("<", "\\u003c")
.replace(">", "\\u003e").replace("
", "\\u2028")
.replace("
", "\\u2029"))
with open(out, "w", encoding="utf-8") as f:
f.write(HTML.replace("__DATA__", payload))
n_linked = sum(1 for e in manifest if e["reports"])
print(f"wrote {out} ({len(manifest)} experiments, "
f"{n_linked} linked to reports)")
port = a.port
if not a.no_serve:
# reuse ONLY a server that provably serves this directory
# (token embedded in explorer.html); otherwise find a free port
while port_in_use(port) and not serves_this_dir(port, rd):
print(f"port {port} busy with something else — trying "
f"{port + 1}")
port += 1
if port_in_use(port):
print(f"port {port} already serving this directory — reusing")
else:
subprocess.Popen(
[sys.executable, "-m", "http.server", str(port),
"--bind", "127.0.0.1", "--directory", rd],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print(f"started http.server on 127.0.0.1:{port}")
url = f"http://localhost:{port}/explorer.html"
if not a.no_open:
webbrowser.open(url)
print(url)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Shared exact permutation statistics for time-series experiments.
Calendar-aligned series with None gaps; EXACT circular-shift permutation
(all N-1 shifts of the full calendar, joint missingness mask re-applied
per shift) — two hard-won lessons: sampled permutation under-estimates
p on small n, and shifting a gap-compressed series breaks the timeline.
"""
from __future__ import annotations
import numpy as np
def masked_r(x, y):
"""Pearson r over jointly observed entries of two float arrays w/ NaN."""
m = ~(np.isnan(x) | np.isnan(y))
n = int(m.sum())
if n < 8:
return None, n
xs, ys = x[m], y[m]
if xs.std() == 0 or ys.std() == 0:
return None, n
return float(np.corrcoef(xs, ys)[0, 1]), n
def to_arr(series):
return np.array([np.nan if v is None else v for v in series], float)
def exact_circ_p(x, y):
"""r + exact circular-shift permutation p over full calendar series."""
x, y = to_arr(x), to_arr(y)
r_obs, n = masked_r(x, y)
if r_obs is None:
return None, None, n
count, total = 0, 0
for k in range(1, len(y)):
r, _ = masked_r(x, np.roll(y, k))
if r is None:
continue
total += 1
if abs(r) >= abs(r_obs) - 1e-12:
count += 1
# guard: a tiny shift universe cannot produce a meaningful p.
# 12 keeps short session sequences (n=13..20, resolution 1/13..1/20)
# testable while refusing degenerate universes; prior results with
# universes >=20 are unaffected.
if total < 12:
return r_obs, None, n
return r_obs, (count + 1) / (total + 1), n
def exact_event_diff(indicator, values, step=1):
"""Mean(values | indicator==1) - mean(values | indicator==0), in SD
units, with exact circular-shift permutation of the indicator.
indicator must be a PURE 0/1 list over the full calendar (no None) —
missingness lives only in `values`, so shifting the indicator moves
event labels without changing the observation sample (audit fix:
None-carrying indicators leaked missingness into placebo samples).
step=7 restricts placebo shifts to weekday-preserving offsets.
"""
ind = np.asarray(indicator, float)
assert not np.isnan(ind).any(), "indicator must be 0/1 with no gaps"
val = to_arr(values)
obs_mask = ~np.isnan(val)
def diff(i):
a = val[obs_mask & (i == 1)]
b = val[obs_mask & (i == 0)]
if len(a) < 5 or len(b) < 5:
return None, len(a)
sd = val[obs_mask].std() or 1.0
return float((a.mean() - b.mean()) / sd), len(a)
d_obs, n1 = diff(ind)
if d_obs is None:
return None, None, n1
count, total = 0, 0
for k in range(step, len(ind), step):
d, _ = diff(np.roll(ind, k))
if d is None:
continue
total += 1
if abs(d) >= abs(d_obs) - 1e-12:
count += 1
if total < 20:
return d_obs, None, n1
return d_obs, (count + 1) / (total + 1), n1
def break_diff(values, cut_idx, min_side=30):
"""Mean(after cut) - mean(before cut) in SD units, with placebo
distribution from ALL non-wrapping cut points having >= min_side
observed days on each side (audit fix: circularly rolled break
indicators wrap around the calendar and are not valid cutpoints)."""
val = to_arr(values)
obs = ~np.isnan(val)
sd = val[obs].std() or 1.0
def diff(c):
a = val[:c][obs[:c]]
b = val[c:][obs[c:]]
if len(a) < min_side or len(b) < min_side:
return None
return float((b.mean() - a.mean()) / sd)
d_obs = diff(cut_idx)
if d_obs is None:
return None, None, 0
placebo = [diff(c) for c in range(len(val))]
placebo = [d for d in placebo if d is not None]
if len(placebo) < 20:
return d_obs, None, len(placebo)
count = sum(1 for d in placebo if abs(d) >= abs(d_obs) - 1e-12)
return d_obs, count / len(placebo), len(placebo)
def bh(tests, m):
"""Benjamini-Hochberg with fixed family size m; adds 'q' in place.
q is stored at FULL precision (round only for display). m must be at
least the number of valid tests — a smaller m is anti-conservative.
Note: plain BH controls FDR under independence/PRDS; for strongly
dependent families use BH-Yekutieli or a maxT resampling scheme.
"""
valid = [t for t in tests if t.get("p") is not None]
if m < len(valid):
raise ValueError(f"family size m={m} < {len(valid)} valid tests "
"(anti-conservative); declare the true family")
qs = {}
for rank, t in enumerate(sorted(valid, key=lambda t: t["p"]), 1):
qs[id(t)] = min(1.0, t["p"] * m / rank)
prev = 1.0
for t in sorted(valid, key=lambda t: -t["p"]):
prev = min(prev, qs[id(t)])
t["q"] = prev
#!/usr/bin/env python3
"""Lead-triage battery — project-agnostic parts (numpy only).
After a sweep produces leads, triage each one. These three checks need no
project dependencies; add prewhiten/bootstrap (statistics.md) for daily
cross-series leads. See references/lead-investigation.md.
loo_robustness(x, y) -> small-n artifact detector
directionality(x, y) -> forward-vs-reverse for lag leads
half_split_gradual(y) -> trend-vs-regime-step for trend leads
consolidate(*series) -> z-composite of same-direction leads
Verdict helper combines them; mint NO new "confirmed" — diagnostic only.
"""
from __future__ import annotations
import numpy as np
def _clean(x, y):
a, b = [], []
for u, v in zip(x, y):
if u is None or v is None:
continue
if (isinstance(u, float) and np.isnan(u)) or \
(isinstance(v, float) and np.isnan(v)):
continue
a.append(float(u))
b.append(float(v))
return np.array(a), np.array(b)
def loo_robustness(x, y):
"""Sign-stability and magnitude-fragility under leave-one-out.
FRAGILE (artifact) if sign flips or |r| more than halves on any drop.
NOTE: passing LOO is necessary, not sufficient — not significance."""
xa, ya = _clean(x, y)
if len(xa) < 8 or xa.std() == 0 or ya.std() == 0:
return None
r_full = float(np.corrcoef(xa, ya)[0, 1])
rs = []
for i in range(len(xa)):
m = np.arange(len(xa)) != i
if xa[m].std() and ya[m].std():
rs.append(float(np.corrcoef(xa[m], ya[m])[0, 1]))
return {
"r_full": round(r_full, 3),
"loo_min": round(min(rs), 3), "loo_max": round(max(rs), 3),
"sign_stable": all((r > 0) == (r_full > 0) for r in rs),
"magnitude_fragile": any(abs(r) < abs(r_full) / 2 for r in rs),
}
def directionality(x, y):
"""Forward vs reverse Pearson r (caller supplies already-lagged
pairs). Causal reading dies if |reverse| >= |forward|."""
xa, ya = _clean(x, y)
if len(xa) < 8 or xa.std() == 0 or ya.std() == 0:
return None
fwd = float(np.corrcoef(xa, ya)[0, 1])
rev = float(np.corrcoef(ya, xa)[0, 1]) # symmetric for same pairing;
# real direction test needs the caller to pass the reverse lag pairs.
return {"forward_r": round(fwd, 3), "reverse_r": round(rev, 3),
"note": "pass reverse-lag pairs explicitly for a real test"}
def half_split_gradual(y):
"""Trend vs regime-step: a gradual trend has same-sign slope in both
halves; opposite signs mean a STEP masquerading as a trend."""
ya = np.array([v for v in y if v is not None
and not (isinstance(v, float) and np.isnan(v))], float)
h = len(ya) // 2
if h < 3 or len(ya) - h < 3:
return None
s1 = float(np.polyfit(np.arange(h), ya[:h], 1)[0])
s2 = float(np.polyfit(np.arange(len(ya) - h), ya[h:], 1)[0])
return {"slope_first_half": round(s1, 4),
"slope_second_half": round(s2, 4),
"gradual": bool(s1 * s2 > 0)}
def consolidate(named_series):
"""z-composite of same-direction leads. `named_series` is a dict
{label: (series, sign)} where sign is +1/-1 for the hypothesized
direction. Returns the composite array (NaN-aware z per component)."""
comps = []
for label, (s, sign) in named_series.items():
a = np.array([np.nan if v is None else v for v in s], float)
z = (a - np.nanmean(a)) / (np.nanstd(a) + 1e-9)
comps.append(sign * z)
return np.nansum(np.vstack(comps), axis=0)
def verdict(loo, gradual=None, direction=None, prewhiten_holds=None):
"""Combine checks into a triage verdict. Diagnostic only."""
if loo is None or not loo["sign_stable"] or loo["magnitude_fragile"]:
return "artifact"
if gradual is not None and not gradual["gradual"]:
return "artifact"
if prewhiten_holds:
return "strengthened"
if direction and abs(direction.get("reverse_r", 0)) >= \
abs(direction.get("forward_r", 1)):
return "artifact"
return "candidate" # robust but underpowered — prospective only
Related skills
Microsoft FoundryDeploy, evaluate, and continuously improve Microsoft Foundry agents from a single agent interface.490k1.3k
Ai Research ReproductionOrchestrate trustworthy, auditable reproduction of deep learning repositories directly from their READMEs.176k511Run TrainSafely execute selected deep learning training commands with standardized evidence capture.176k511Explore RunSafely run isolated exploratory experiments with clear recording and conservative selection before committing changes.176k511Paper Context ResolverFetch precise reproduction-critical details like dataset splits, preprocessing steps, or evaluation protocols from the original academic paper when the repo README leav141k511Repo Intake And PlanScan unfamiliar AI research repositories and receive a minimal, trustworthy reproduction target before investing significant time.140k511