Marketplace Health Check

Marketplace Health Check

Run a comprehensive, evidence-based health check of this Claude Code skills marketplace repo using a parallel fan-out Dynamic Workflow. Six independent inspectors cover, in parallel:

1. Code & script safety — dangerous deletes, NO-FALLBACK secret leaks, hardcoded real paths, bare except, injection, missing shebangs 2. Documentation / SSOT consistency — version coherence across marketplace.json / README×2 / CHANGELOG / git release, skill & plugin counts, broken references, derived-value drift 3. Security / PII — keyword-free leaks gitleaks can't catch (real names, private domains), the .security-scan-passed marker gap, case-file audits 4. Open-PR triage — classify every PR (worth-merging / needs-changes / decline-as-promotion) 5. Open-issue triage — real bugs vs skill-requests vs promotion, plus the broken-install-command bug class 6. Marketplace-manifest integrity — check_marketplace.sh + check_doc_skill_lists.py, orphans, suite registration

Then YOU verify the serious findings and report by priority. The bundled script (scripts/repo-health-check.workflow.js) is the proven, ready-to-run workflow; this file is how to run and interpret it.

Why a workflow — and why it MUST run inline

The six dimensions are independent, so fanning them out across six parallel agents is far faster than one agent sweeping serially, and each inspector stays focused on one concern with its own structured output.

This skill must run inline (no `context: fork`). It orchestrates parallel agents through the Workflow tool, and a forked subagent cannot spawn subagents or launch a workflow — running it forked would silently break the fan-out. The Workflow tool also requires explicit user opt-in; a user asking to "run the health check" IS that opt-in, so proceed.

How to run

Step 1 — Scout the current scale (one quick pass, shared by all six agents)

The workflow script takes an args object so all agents share one accurate snapshot instead of each re-discovering it. Gather:

gh repo view --json nameWithOwner,stargazerCount,isPrivate | jq -c .
echo "skills: $(find . -name SKILL.md -not -path '*-workspace/*' | wc -l | tr -d ' ')"
echo "open PRs: $(gh pr list --state open --json number | jq length)"
echo "open issues: $(gh issue list --state open --json number | jq length)"
grep -A1 '"metadata"' .claude-plugin/marketplace.json | grep -oE '"version": "[^"]*"' | head -1
git rev-parse --short HEAD; gh release view --json tagName -q .tagName 2>/dev/null

Confirm isPrivate: false before treating PII as a publishing risk — the whole point is that this is a PUBLIC repo.

Step 2 — Launch the workflow

Read the bundled script and launch it inline via the `script` parameter (pass its contents, so there's no dependency on where the skill is installed):

Workflow({
  script: <full contents of scripts/repo-health-check.workflow.js>,
  args: { repo: "<owner/name>", scale: "<one-line summary from Step 1>" }
})

It runs the six inspectors in parallel (~15-20 min, ~400-500k output tokens — tell the user the cost up front) and returns { checks: [...] }, one structured result per dimension: health + summary + findings[] (each with severity / title / detail / location / recommendation) + stats.

While it runs you can do other useful prep, but don't start editing files the inspectors are reading.

Step 3 — Counter-Review the serious findings BEFORE reporting

Agent findings are HYPOTHESES, not conclusions. Never relay them verbatim. For every high/critical finding, verify it yourself with a quick command — grep the leaked value, sed -n the broken line, gh repo view the claimed state — confirming it's (a) real, (b) located where the agent says, and (c) not over-reach. This catches false alarms AND, just as important, agent recommendations that are actively wrong. (In the session this skill was distilled from, a security inspector recommended adding the real private domains into the public .gitleaks.toml — an anti-target move that had to be rejected; see the methodology reference.)

Filter every finding through four questions: probability (does it really happen?), cost (fix vs ignore), real scenario (does it bite in practice?), verifiable (can a 1-line command confirm or refute it?).

Report format

Lead with the table, then layer by priority. Classify — don't dump:

• One-line verdict + a 6-dimension health table (good / minor-issues / needs-attention / critical per dimension)
• 🔴 Must-fix — each VERIFIED high/critical, with exact location + a concrete fix
• 🟠 Backlog — PR/issue triage outcomes, scan-marker gaps (decisions, often outward-facing — flag that they affect external contributors)
• 🟢 Optional — low/info nits, one line each
• 💡 Key insights — the meta-findings worth surfacing (a structural blind spot in tooling, a recurring bug class)

Tag each surfaced item ✅ real / ⚠️ partly / ❌ false-alarm. Most raw agent output is noise; your job is to surface the real risks the owner didn't already know, not to forward 25 findings for them to sift.

Judgment principles

Apply these when interpreting findings and proposing fixes. Full reasoning + the real failure cases behind each are in references/health-check-methodology.md — read it before acting on PII or PR/issue findings.

• Anti-target: never "fix" a PII leak by listing the real value in a public allowlist (e.g. the repo's own .gitleaks.toml) — a public list enumerating real assets is itself a leak. Sanitize the value in place; detection rules for real private values belong in the owner's private global guard, not in this public repo.
• History note: sanitizing the working copy cleans the current version, but a pre-existing leak still sits in git history. Flag the history exposure honestly; a history rewrite (force-push) is a separate high-risk decision that affects every fork — never do it unprompted.
• Scan marker = necessary-not-sufficient: a .security-scan-passed marker means "no known-format secret was found", NOT "sanitized". It is blind to keyword-free leaks, so pair it with a human/semantic read of any skill shipping real-data examples.
• Mandatory version bump: any change to a skill's files requires bumping that skill's version in marketplace.json (and a CHANGELOG entry). External-contributor PRs almost always miss this — flag it, don't merge without it.
• Promotion is declined by default: third-party directory / tool / marketplace promotion PRs and issues are declined — the repo is a personal curated marketplace, not an ecosystem directory. The decline-policy template is a reference doc at the repo root (outside this skill's bundle).

Bundled resources

• scripts/repo-health-check.workflow.js — the six-inspector Dynamic Workflow. Run it via the Workflow tool's script param (Step 2). Edit it when you add/retire an inspector dimension.
• references/health-check-methodology.md — the Counter-Review filter, reporting discipline, and the anti-target / history / scan-marker / decline rules, each with the real failure case that motivated it.

Next step

After delivering the report, the typical follow-ups are owner decisions, not automated actions — fixing the verified HIGHs (sanitize PII, correct broken commands), or triaging the PR/issue backlog. Surface them as options; don't auto-fix or auto-comment on PRs/issues without the user's go-ahead, since those are outward-facing and affect external contributors.

Health-Check Methodology

The reasoning and real failure cases behind the judgment principles in SKILL.md. Read this before acting on PII or PR/issue findings — the rules look obvious in the abstract, but each one exists because the obvious-looking move was wrong in a real run. (Cases are described abstractly on purpose: naming the real leaked values here would re-leak them, which is the anti-target rule below.)

Counter-Review: agent findings are hypotheses, not conclusions

A fan-out of six inspectors is fast and broad, but breadth comes with noise. Each inspector is good at finding things and bad at weighing them — it lists every theoretically-possible problem without distinguishing a 0.1% edge case from a live one. So the inspectors' output is a risk list, not a verdict. Reporting it verbatim floods the user with noise and, worse, can launder a wrong fix into an action.

Before reporting any high/critical finding, run it through four questions:

1. Probability — does this really happen, or is it a fictional edge case? 2. Cost — what does fixing vs ignoring each cost? 3. Real scenario — does it actually bite in this repo's real usage? 4. Verifiable — can a one-line command confirm or refute it right now?

Then verify the survivors with a direct command (grep the value, sed -n the line, gh repo view the state). Two failure modes this catches:

• False alarms — a flagged "broken reference" that's actually an illustrative example; a "secret" that's a documented public placeholder; a version line inside a skill that teaches about versioning.
• Wrong recommendations — the more dangerous case, where an inspector correctly finds a real problem and proposes a fix that makes things worse.

Case: the anti-target recommendation (rejected)

In the run this skill was distilled from, the security inspector correctly found real private values in shipped examples — and then recommended adding those exact values into the repo's own public .gitleaks.toml so future commits would be caught. That is backwards. .gitleaks.toml lives in the PUBLIC repo, so a list enumerating the owner's real private domains/handles is itself a published target map. The correct fix is two-part: sanitize the value in place, AND put the detection rule in the owner's PRIVATE global guard, which already scans future diffs. Verifying that global guard's coverage (it already had the domains; it lacked the names) is what converted a plausible-but-wrong recommendation into the right action. Relayed verbatim, it would have shipped a target list.

Anti-target: never publish the thing you're hiding

A public allowlist/denylist that names real private assets defeats itself. This applies to .gitleaks.toml, to "forbidden domains" comments, to any in-repo file in a public repo. In the public repo, only ever remove the value (replace with example.com, a neutral placeholder, <user>); never enumerate it. Detection rules for real private values belong in a private guard outside the repo. The same logic is why this very methodology file describes its cases abstractly.

History: working-copy sanitization is not a history scrub

Sanitizing a file cleans the current version. But a leak that was committed earlier (pre-existing) still sits in git history — git show <old-commit>:<file> reveals it. The honest report says: "current version clean; the value remains in history." A history rewrite (git filter-repo + force-push) is a separate, high-risk decision — it rewrites every commit hash and breaks every fork's sync, and the value is likely already forked/cached anyway, so the benefit is limited. Never rewrite history unprompted; present it as an option with its full cost, and let the owner decide (in practice they usually keep history and accept the residual exposure).

Scan marker = necessary, not sufficient

A .security-scan-passed marker records that the bundled security_scan.py (gitleaks + regex) found no KNOWN-FORMAT secret. It is structurally blind to keyword-free leaks: a real personal name in another language, a real private domain that matches no secret pattern, a verbatim transcript line. In the distillation run, two skills with GREEN markers nonetheless shipped a real private domain and a real personal handle — both invisible to the scanner, both caught only by a human semantic read. So: treat a missing marker as "unscanned, worth fixing", but never treat a present marker as "sanitized". Always pair it with a read-through for any skill shipping real-data examples (debugging case studies, transcript fixtures, financial samples).

Operational note: security_scan.py MUTATES the marker file when it runs. During a read-only audit, git checkout the marker afterward so the audit leaves no trace in the working tree.

The broken-install-command bug class

A recurring, high-leverage doc bug in a suite-based marketplace: install instructions that name a SUITE MEMBER as if it were a standalone plugin. claude plugin install <member>@<marketplace> FAILS — only the suite plugin is installable; members are invoked as <suite>:<member>. It bites hardest on the flagship skill's front-door command (the first thing a visitor copy-pastes), so a broken one is HIGH severity, not cosmetic. The check is mechanical: for every install command, confirm the named plugin is a top-level marketplace.json entry, NOT a name that only appears inside a suite's skills[] array. Internal inconsistency (one section correct, another wrong) is the tell that it's an oversight, not a design choice — and a sign the fix is doc-only, not a re-architecture.

Mandatory version bump + CHANGELOG

Any change to a skill's files requires bumping that skill's version in marketplace.json plus a CHANGELOG entry — that is how the marketplace tracks what changed, and it gates the install/update flow. External-contributor PRs almost always miss it (they don't know the convention). Flag such PRs as needs-changes; don't merge without it. For suite members, the version lives on the suite plugin, so a member-file change bumps the suite version, not a member version.

Promotion is declined by default

Third-party promotion — "add my external directory", "list my tool", revenue-share backlinks, a new "Community Skills" section — is declined per the repo's standing promotion-decline policy — a reference doc at the repo root, outside this skill's bundle. The repo is a personal curated marketplace, not an ecosystem directory. Decline politely with the policy's template; the contributor's own repo/marketplace already works standalone. This is the dominant pattern in the open-PR/issue queue, so triaging it correctly is what clears most of the backlog — but it's outward-facing, so surface the decline list to the owner rather than commenting on PRs/issues directly.