Deep Brainstorming

Deep Brainstorming

Research-hardened brainstorming that catches biases agents naturally introduce. Collapses what would otherwise be 3-4 sessions of iterative discovery into one disciplined session.

REQUIRED: Run a standard brainstorming pass first (e.g., superpowers:brainstorming or equivalent). This skill augments brainstorming output — it does not replace it.

When to Use

digraph when {
  "Starting brainstorming?" [shape=diamond];
  "Multiple valid tech choices?" [shape=diamond];
  "Wrong choice costly to reverse?" [shape=diamond];
  "Use regular brainstorming" [shape=box];
  "Use deep brainstorming" [shape=box, style=bold];

  "Starting brainstorming?" -> "Multiple valid tech choices?";
  "Multiple valid tech choices?" -> "Use regular brainstorming" [label="no"];
  "Multiple valid tech choices?" -> "Wrong choice costly to reverse?";
  "Wrong choice costly to reverse?" -> "Use regular brainstorming" [label="no — easy to swap"];
  "Wrong choice costly to reverse?" -> "Use deep brainstorming" [label="yes"];
}

Intensity Selector

Pick intensity based on stakes and time budget. Default to Standard unless stakes clearly call for Quick or Thorough.

Process Overview

Phase 1: Sanitize the Vision

Extract the client brief into a clean requirements document. This is the anchor for all research.

Strip: All tool names, frameworks, architecture patterns, vendor references. Keep: What it does, who uses it, hard constraints (language, platform, compliance). Save as: <project-docs-dir>/<project>-vision.md

Scoping discipline: Only include requirements the client actually stated. If a requirement wasn't in the brief, it doesn't belong in the vision. Agents inject phantom requirements from training data (compliance frameworks, accessibility standards, monitoring stacks). Challenge every requirement: "Did the client ask for this, or did an agent add it?"

Non-Requirements section: Explicitly list what the client did NOT ask for in the vision doc. This is the primary defense against phantom requirements — research agents that recommend solutions to Non-Requirements get flagged immediately.

Phase 2: Research with Clean Prompts

Prompt rules:

• Zero vendor/tool names — only functional requirements + vision doc
• "What's the best way to achieve Y?" not "Is X good for Y?"
• Include: "Verify non-library claims via web search. Do not rely on training data."
• Attach the sanitized vision document to every prompt

Agent discipline (mandatory for ALL researcher agents):

• Synthesize first: Every researcher prompt MUST include: "SYNTHESIZE FIRST — write your findings report before doing more searches. You can always search more after writing what you know." Without this, agents spend 80%+ of their token budget on searches and never produce output. (Evidence: 60% researcher agent failure rate observed from token exhaustion on web-search-heavy tasks.)
• Max 5 questions per agent. Split larger research across parallel agents. One agent with 10 questions will exhaust tokens on the first 5 and never synthesize. Each agent gets 3-5 focused questions.
• Structured doc tools for library docs. Prefer structured documentation sources (e.g., Context7 MCP, official API docs) over generic web search for library API verification. Structured docs return focused content at lower token cost than web search chains.
• Relaunch failed agents. When a researcher fails to synthesize (returns partial output like "Let me search for more..."), relaunch with a shorter prompt. Doing the research inline burns 5-10x more main context.

Round structure:

• Split research by domain (e.g., backend, frontend, data pipeline, infrastructure, NLP)
• One agent per domain per round, all running in parallel
• Round 1: broad discovery — "What are the top approaches for [functional need]?"
• Round 2: verification with fresh prompts that do NOT reference Round 1 findings. Ask the same functional questions differently. Compare convergence.
• Round 3+ (Thorough only): targeted investigation of divergences between rounds

Consensus guard: If all agents in a round converge on the same tool, that's a red flag — not validation. Force dissent: "What's the strongest alternative to [consensus pick] and under what conditions would it win?"

Prompt review cycle: Before launching each round, present prompts to the user for review and revision. Agents internalize biased framing invisibly — the user catches phrasing that anchors research. Specific things to check: did any functional description smuggle in a tool name? Did "agentic orchestration" imply a specific architecture? Soften loaded terms to neutral descriptions.

Full reset option (Thorough): For the final research round, discard all prior findings: "All previous research counts as zero. Start from the requirements only." This prevents anchoring to earlier rounds' conclusions and surfaces genuinely different approaches.

Convergence table: After each round, produce a cross-round comparison table showing every decision layer with what each round recommended. Items that converge across rounds get high confidence. Items that diverge get "benchmark to decide" status. This table is the primary decision tool — not any single round's output.

For detailed prompt templates, see references/research-prompts.md.

Phase 3: Bias Audit

Check after EVERY research round. Audit the full output, not just the recommendations.

Retroactive sweep: Bias found in one section means the same bias likely exists in ALL sections. Audit every previous section for the same pattern before proceeding.

Phase 4: Independent Verification

Do not trust agent output for factual claims. The orchestrator verifies directly — not via more agents. The orchestrator has full synthesis context and can cross-reference claims against actual leaderboard URLs in a way a fresh agent cannot.

Positive claims (assertions):

• Benchmark numbers → find the original leaderboard or paper
• "#1" / "best" claims → verify against the cited source (not a blog post about it)
• Version numbers → check official release pages
• "Supports X" claims → check official docs, not third-party posts
• Pricing/licensing → check the vendor's current pricing page

Negative claims ("X doesn't exist") — NEVER accept from a single agent:

• Agent searches can miss things. "X doesn't exist" requires LOCAL verification:
• Model existence → HuggingFace API query (not pip — pip checks packages, not model registries)
• Package existence → pip index versions X or PyPI search
• Import path → python -c "from X import Y"
• Version → pip show X or official docs
• Evidence: agent said "Model X not publicly available" (HuggingFace API showed it exists). Another agent said "Library v5 doesn't exist" (official docs confirmed v5). Both premature dismissals accepted from single agents, reversed by local checks.
• Rule: Only accept negative claims after local verification produces the same conclusion.

Verification chain termination — three types of evidence that break the loop: 1. Direct local (highest): pip show, pip index versions, python -c, HuggingFace API 2. Multi-source convergence (good): 2+ independent agents agree via different methods 3. Implementation (ultimate): if it imports and runs, it works

Agents verifying agents is infinite recursion. Only local verification produces ground truth.

Synthesis robustness audit: Before moving from research to design, categorize EVERY recommendation into three tiers: 1. Robust — independently verified from 2+ sources across multiple rounds 2. Single-source — one round found it, or only one source confirms it 3. Unverified — agent-generated, no independent confirmation found

Only Robust items enter the spec with confidence. Single-source items enter with caveats. Unverified items are dropped or flagged for benchmarking.

Phase 5: Claim Provenance

For every cited number, benchmark, or attributed claim, record:

Mark unverified claims explicitly in the spec. Never let an unverified claim propagate into planning.

Phase 6: Progressive File Capture

Write each design section to a file as it's approved. Assemble into one spec at the end.

• One file per section: <project-docs-dir>/<project>-section-N-<topic>.md
• Each file includes its provenance records
• Assemble final spec from approved section files — not from memory
• Final spec replaces section files (they were scaffolding)

This protects against context compaction losing approved work.

Phase 7: Three-Phase Review

After assembling the spec, run the review phases required by your selected intensity level. Each phase is a separate agent with NO context from prior phases.

Phase R1 — Verify-fix: Standard review for consistency, completeness, formatting. Fix issues, verify fixes are clean.

Phase R2 — Adversarial (MANDATORY for Standard+): "You are reviewing a spec that passed standard review. Your job is to find what the standard review missed. Don't trust the previous approval. Challenge every architectural claim, every tool selection, every assumption. Find contradictions between sections."

Phase R3 — Security/Operations: "Focus exclusively on: security vulnerabilities, scalability bottlenecks, error handling gaps, deployment complexity, testing gaps, operational burden. Ignore formatting and style."

Each phase must end with a clean verification pass before proceeding to the next.

Anti-skip rule: "The plan already went through N agents" is the EXACT rationalization this rule targets. Different phases catch different classes of issues. R1 catches consistency. R2 catches architectural contradictions and YAGNI. R3 catches security/ops gaps. Evidence: adversarial review has caught 7+ YAGNI violations and multiple critical issues invisible to standard review and to deepening agents in real use. Skipping R2 because "agents already reviewed" is the single most expensive mistake in this process.

Finding engagement depth: For each finding from any review phase, READ the actual code cited (not the reviewer's summary), VERIFY the claim is real, ESTIMATE fix cost, and FIX if cheaper than documenting a deferral. "Tracked as deferred" is not engagement — it's categorization theater. If the fix is 1-5 lines, just fix it.

For exact review prompts, see references/review-protocols.md.

Phase 8: "Objectively Best?" Quality Gate

After all reviews pass, challenge the entire spec one final time.

Protocol: 1. Re-read the spec from scratch (not from memory of writing it) 2. For each major decision: "Is this the objectively best choice, or did we settle?" 3. For each claim: "Is this verified, or did we trust an agent?" 4. For each omission: "Did we skip this because it doesn't matter, or because it was hard?"

This gate has surfaced real issues in every single use. It is not optional.

When the gate surfaces issues: Fix the issue, then re-run the relevant review phase (not just the gate). The gate is a detector, not a fixer — changes made in response need the same review rigor as the original spec.

Self-assessment is not verification. Evaluating your own work against your own criteria is circular. Re-read actual source artifacts fresh.

Transition to Planning

After the spec is approved:

1. Spec code is illustrative, not verified. Examples in the spec are for communication — they may use stale APIs or wrong syntax. Dispatch research agents to verify spec code against current API docs BEFORE writing plan code. 2. Each plan section needs its own research agent to verify the spec's assumptions against current reality. 3. The brainstorm spec is the plan's input, not its source of truth. Plans must cite current documentation, not the spec's examples.

The 9 Bias Types (Quick Reference)

Common Mistakes

Red Flags — STOP and Re-examine

• Research prompt mentions a specific tool by name
• All agents in a round converge on the same popular tool
• Benchmark number cited without original source
• All candidates are one license type only (OSS-only or commercial-only)
• Standard review passed but no adversarial or security review done
• Bias found in section 3 but sections 1, 2, 4-6 not re-audited
• Agent output trusted without independent web verification
• A requirement appears that the client never stated
• "Objectively best?" gate skipped or answered with "yes, I'm confident" without re-reading
• Agent says "X doesn't exist" and you accept it without local verification
• Researcher agent returns partial output ("Let me search for more...") and you move on
• "The plan already went through N agents" used to skip a review phase
• Review findings categorized but not actually read (batch disposition without code reading)

Checklist

Items marked (S) = Standard+Thorough only. Items marked (T) = Thorough only. Unmarked = all intensities.

• [ ] Vision sanitized (zero tool names, zero phantom requirements, Non-Requirements section)
• [ ] User reviewed and revised research prompts before launch
• [ ] Research Round 1 (clean prompts, parallel agents, vision attached)
• [ ] Bias audit on Round 1 (Quick: spot-check only; Standard+: all 9 types + retroactive sweep)
• [ ] (S) Research Round 2 (fresh prompts, no Round 1 references)
• [ ] (S) Convergence table (Round 1 vs Round 2 comparison)
• [ ] (S) Bias audit on Round 2
• [ ] (T) Research Round 3+ (targeted divergences or full reset)
• [ ] Consensus guard applied (forced dissent if unanimous)
• [ ] All researcher prompts include "SYNTHESIZE FIRST" and max 5 questions
• [ ] Structured doc sources used for library verification (not just web search)
• [ ] Independent verification of key claims (orchestrator directly, not more agents)
• [ ] Negative claims verified locally (pip, HF API, python -c) — never trust single agent
• [ ] Synthesis robustness audit (every item: robust / single-source / unverified)
• [ ] Claim provenance recorded (source, quote, status)
• [ ] Design sections approved + written to files
• [ ] Spec assembled from section files
• [ ] Review R1: verify-fix → clean
• [ ] (S) Review R2: adversarial → clean
• [ ] (T) Review R3: security/ops → clean
• [ ] Retroactive consistency verified across all sections
• [ ] "Objectively best?" quality gate passed (re-read fresh, not from memory)
• [ ] Transition notes for planning (spec code marked as illustrative)
• [ ] User reviews final spec

Research Prompt Templates

Reference file for deep-brainstorming skill. Load when structuring research rounds.

Agent Discipline (inject into ALL researcher prompts)

Every researcher prompt MUST include these instructions:

SYNTHESIZE FIRST — write your findings report before doing more searches.
You can always search more after writing what you know. Do not spend more
than 60% of your effort on searching — the remaining 40% must go to synthesis.

For library documentation, prefer structured doc sources (e.g., Context7 MCP, official API docs)
over web search where available. They return focused content at lower token cost.

Max 5 questions per agent. If you have more questions, split across parallel agents.

Round 1: Broad Discovery

One agent per domain. Each agent gets the same vision doc but a different functional area.

Prompt Template (Round 1)

You are researching [DOMAIN AREA] solutions for a project.

SYNTHESIZE FIRST — write your findings report before doing more searches.
You can always search more after writing what you know.

For library API docs, prefer structured documentation sources (official docs, doc tools)
over web search. Use web search for benchmarks, comparisons, and non-library topics.

## Requirements
[Paste sanitized vision document here]

## Your Task (max 5 questions — focus on these)
Research the best approaches for the [DOMAIN AREA] requirements above.

1. Identify the top 3-5 approaches (include both open-source and commercial options)
2. For each approach, provide:
   - What it does and how it addresses the requirements
   - Verified benchmark data (cite the original source, not a blog post)
   - Known limitations for this specific use case
   - Production readiness (who uses it at scale?)
3. Rank approaches by quality-of-fit for THESE specific requirements
4. Note any requirements that NO current tool fully addresses

Verify non-library claims via web search. Do not rely on training data for version numbers,
benchmarks, or feature availability. If you cannot verify a claim, mark it as UNVERIFIED.

Do NOT recommend tools because they are popular. Recommend them because they are
the best fit for these specific requirements.

Domain Split Examples

Round 2: Fresh Verification

Critical: Round 2 prompts must NOT reference Round 1 findings. Ask the same functional questions with different framing.

Prompt Template (Round 2)

You are independently evaluating approaches for [DOMAIN AREA].

SYNTHESIZE FIRST — write your findings report before doing more searches.
You can always search more after writing what you know.

For library API docs, prefer structured documentation sources (official docs, doc tools)
over web search. Use web search for benchmarks, comparisons, and non-library topics.

## Requirements
[Paste same sanitized vision document]

## Your Task (max 5 questions — focus on these)
Without assuming any prior research has been done:

1. What are the strongest solutions for [DOMAIN AREA] given these requirements?
2. For each solution:
   - Current version and release date (verify via official source)
   - Performance characteristics verified from independent benchmarks (not vendor)
   - Integration complexity with [related domains from vision]
   - Licensing and pricing (verify current, not cached)
3. What would you recommend if cost were no object? What about for a constrained budget?
4. What's the biggest risk with the most popular option in this space?

Verify non-library claims via web search. Mark anything you cannot independently confirm as UNVERIFIED.

Comparing Round 1 and Round 2

After both rounds complete:

Round 3+: Targeted Investigation (Thorough Only)

Only needed when Rounds 1 and 2 diverge significantly.

Prompt Template (Round 3)

Two independent research rounds produced different recommendations for [DOMAIN AREA]:
- Round A recommends: [OPTION A] because [EVIDENCE A]
- Round B recommends: [OPTION B] because [EVIDENCE B]

SYNTHESIZE FIRST — write your analysis before doing more searches.

Your job is to determine which recommendation is better supported:

1. Verify each piece of evidence independently
2. Find cases where [OPTION A] outperforms [OPTION B] and vice versa
3. Identify the specific conditions under which each option is superior
4. Recommend which option is better FOR THESE SPECIFIC REQUIREMENTS, with evidence

Do not default to the more popular option. The popular option needs to EARN the
recommendation with better evidence, not better marketing.

Consensus Guard Prompt

Use when all agents in a round converge on the same tool.

All research agents recommended [TOOL X] for [DOMAIN].

Your job is adversarial: find the strongest case AGAINST [TOOL X] and FOR an alternative.

1. What are [TOOL X]'s biggest weaknesses for these specific requirements?
2. What's the strongest alternative, and under what conditions would it win?
3. Are there use cases similar to ours where [TOOL X] failed or was replaced?
4. Is the recommendation based on benchmarks or on blog post volume?

If after this investigation [TOOL X] is genuinely best, say so with evidence.
But your default assumption should be skepticism, not confirmation.

Claim Verification Prompt

Use for independent verification of specific claims.

Verify the following claim:
"[EXACT CLAIM TEXT]"

SYNTHESIZE FIRST — write what you find before doing more searches.

1. Find the original source (not a secondary reference)
2. What does the source actually say? (Direct quote if possible)
3. Is this from an independent source or the vendor?
4. Has this been replicated or confirmed by others?
5. What's the date of this information? Is it current?

Report: VERIFIED (2+ independent sources), SINGLE-SOURCE, VENDOR-ONLY, or UNVERIFIED.

Negative Claim Verification

Use when an agent reports "X doesn't exist" or "X is not available." NEVER accept from a single agent.

Do NOT dispatch another agent. Verify locally:

# Model existence (HuggingFace models, not PyPI packages)
# HuggingFace API: curl -s https://huggingface.co/api/models/<org>/<model> | head
# or: hf models info <model-name>

# Package existence
pip index versions <package-name>
# or: PyPI web search

# Import path verification
python -c "from X import Y"

# Version check
pip show <package-name>

Evidence: In real use, agents prematurely dismissed existing models and library versions as "not available." Local verification (HuggingFace API, official docs) reversed both. This pattern recurs — agents confidently report non-existence for things that exist.

Only accept negative claims after local verification produces the same conclusion.

Review Phase Protocols

Reference file for deep-brainstorming skill. Load when running the three-phase review.

Key Principle

Each review phase must be a separate agent with no context from prior phases. The agent receives only the spec document and its review instructions. This prevents confirmation bias from earlier approvals.

Phase R1: Verify-Fix

Standard review for consistency, completeness, and correctness.

R1 Prompt

Review this architecture specification for:

1. **Internal consistency** — Do sections contradict each other?
2. **Completeness** — Are there requirements from the vision that aren't addressed?
3. **Factual accuracy** — Are version numbers, API references, and benchmarks current?
4. **Clarity** — Would a developer understand exactly what to build?
5. **Claim verification** — Are provenance records present for cited numbers?
   Flag any claim marked UNVERIFIED or VENDOR-ONLY.

For each issue found:
- Cite the exact section and line
- Explain what's wrong
- Suggest a specific fix

Categorize issues as: CRITICAL (blocks implementation), HIGH (causes problems), LOW (style/clarity).

R1 Process

1. Agent reviews spec → produces findings 2. Fix all CRITICAL and HIGH issues 3. Re-run R1 agent on the fixed spec 4. Continue until R1 returns clean (no CRITICAL or HIGH)

Phase R2: Adversarial (MANDATORY for Standard+)

Challenges every architectural decision. Specifically looks for what standard review misses.

This phase is NOT optional for Standard intensity. Evidence: adversarial review has caught 7+ YAGNI violations and multiple critical issues invisible to standard review and to deepening agents. "The plan already went through N agents" is the EXACT rationalization this phase counters.

R2 Prompt

You are reviewing a specification that has already passed standard review.
Your job is to find what the standard reviewers missed.

DO NOT trust the previous approval. Assume the spec has unfound problems.

Focus on:
1. **Architectural contradictions** — Does the system design actually work end-to-end?
   Trace a request from user input to database and back. Where does it break?
2. **Phantom requirements** — Are there requirements that the client never asked for?
   Compare every spec requirement against the vision document. Flag additions.
3. **Tool selection justification** — For each chosen tool, can you find a stronger alternative?
   The burden of proof is on the chosen tool, not the alternatives.
4. **Assumption audit** — What assumptions does the spec make about:
   - Scale (concurrent users, data volume)
   - Integration (APIs, services that "support X")
   - Performance (latency, throughput)
   Are these assumptions verified or hoped?
5. **Missing failure modes** — What happens when:
   - The primary database is unreachable
   - An external API changes its contract
   - Traffic exceeds the stated capacity by 10x
   - A dependency is deprecated

For each finding, assign a confidence level:
- CERTAIN: Demonstrably wrong (cite evidence)
- LIKELY: Strong reason to believe it's wrong
- WORTH INVESTIGATING: Suspicious but not confirmed

Only CERTAIN and LIKELY findings need fixing before proceeding.

R2 Process

1. Agent reviews spec with adversarial lens → produces findings 2. For CERTAIN findings: fix immediately 3. For LIKELY findings: verify independently, then fix or dismiss with evidence 4. Re-run R2 agent on the fixed spec 5. R2 is clean when no CERTAIN or LIKELY findings remain

Phase R3: Security/Operations

Production readiness review. Ignores formatting and style entirely.

R3 Prompt

Review this architecture specification for production readiness.
Ignore formatting, style, and documentation quality — focus only on:

1. **Security vulnerabilities**
   - Authentication/authorization gaps
   - Data exposure risks (PII, credentials, tokens)
   - Input validation gaps (injection, XSS, SSRF)
   - Secret management approach
   - Encryption at rest and in transit

2. **Scalability bottlenecks**
   - Single points of failure
   - Connection pool limits
   - Database query patterns under load
   - Caching strategy gaps
   - Background job queue saturation

3. **Error handling gaps**
   - What happens when external services fail?
   - Retry and circuit breaker strategies
   - Data consistency during partial failures
   - User-facing error messages (information leakage?)

4. **Deployment complexity**
   - How many moving parts to deploy?
   - Rollback procedure
   - Database migration safety
   - Zero-downtime deployment feasibility

5. **Testing gaps**
   - What's untestable in the current design?
   - Integration test complexity
   - Mock/stub requirements for CI

6. **Operational burden**
   - Monitoring and alerting coverage
   - Log aggregation approach
   - On-call complexity (how many systems to understand?)
   - Cost estimation accuracy

For each finding:
- Severity: CRITICAL (must fix before implementation) / HIGH (fix during implementation) / MEDIUM (track as TODO)
- Specific recommendation

R3 Process

1. Agent reviews spec for ops/security → produces findings 2. CRITICAL findings: fix before proceeding to planning 3. HIGH findings: add to implementation plan as requirements 4. MEDIUM findings: track in TODO for later phases 5. Re-run R3 until no CRITICAL findings remain

"Objectively Best?" Final Gate

After all required review phases for your selected intensity pass, apply this final quality gate.

Protocol

This is NOT a review phase — it's a re-read of the entire spec with fresh eyes.

1. Re-read the spec from the beginning — not from memory of writing it 2. For each major technology choice, ask:

• "Is this the objectively best option, or the most familiar one?"
• "What evidence supports this over the alternatives?"
• "If we removed the tool name and described only the requirements, would research agents converge on this same choice?"

3. For each cited number or benchmark:

• "Is this verified from an independent source?"
• "Is the provenance record complete?"

4. For each section:

• "Would a developer new to this project understand exactly what to build?"
• "Is anything here because an agent suggested it rather than because the requirements demand it?"

Quality Gate Pass Criteria

The spec passes the quality gate when:

• [ ] Every technology choice is backed by verified evidence (not popularity)
• [ ] Every cited number has a complete provenance record
• [ ] No phantom requirements remain (everything traces to the client brief)
• [ ] A fresh reader could implement from this spec without ambiguity
• [ ] The user has reviewed and approved the final spec

If any of these fail, fix the issue and re-run the relevant review phase.