Testing Strategies

Testing Strategies

Use this skill when the main question is "what validation packet do we trust, what confidence level do we actually need, and what should the team do next?"

The job is not to dump a generic pyramid/trophy manifesto or write test code. The job is to: 1. normalize the current policy packet, 2. name which gate is actually being decided, 3. choose one primary policy mode, 4. name the smallest convincing layer mix, 5. separate local / PR / release / scheduled expectations, 6. state exception and flake rules honestly, 7. route implementation, debugging, release execution, accessibility, game launch, or performance work out immediately.

Read references/intake-packets-and-route-outs.md before handling an unfamiliar policy packet. Read references/gate-truth-and-release-handovers.md when the ambiguity is really about branch blockers vs release-only or platform-launch proof. Read references/validation-matrix.md when choosing the minimum convincing layer mix. Read references/handoff-boundaries.md when deciding whether testing-strategies, backend-testing, debugging, code-review, performance-optimization, or web-accessibility should own the next step.

When to use this skill

• The team needs one change-based validation brief instead of vague “add more tests” advice
• A developer or reviewer is asking what should run locally, on PR, before release, or on scheduled/nightly cadence
• You need to decide whether unit, integration, contract, smoke, exploratory, or manual checks are actually required
• A flaky or expensive suite problem is really a gate-policy problem
• An escaped bug or incident needs the right regression ratchet without blindly expanding broad E2E coverage
• Release-readiness or QA-signoff work needs to be tied back to the real change risk

When not to use this skill

• The main task is implementing API/service/database/browser tests, fixtures, mocks, or testcontainers → use backend-testing or the stack-specific implementation skill
• The main task is reproducing a failure, isolating why a test is red, or debugging flaky behavior → use debugging
• The main task is judging one specific PR, diff, or merge request → use code-review
• The main task is accessibility-heavy verification or visual review policy → use web-accessibility or web-design-guidelines
• The dominant risk is performance benchmarking, load testing, or frame-budget policy → use performance-optimization or game-performance-profiler
• There is no real change or decision point yet; first define what changed and what confidence decision must be made

Instructions

Step 1: Start from the policy packet already in hand

Use references/intake-packets-and-route-outs.md.

Normalize the current packet into one of these shapes:

• change-risk-packet — a feature, bugfix, migration, API change, auth change, UI flow change, config/deploy change, or incident follow-up
• gate-design-packet — the team is arguing about what belongs in local, PR, release, or scheduled gates
• flake-cost-packet — the suite is slow, noisy, brittle, or expensive and policy is unclear
• release-readiness-packet — staging smoke, signoff, rollout, or checklist work is present but layer ownership is fuzzy
• incident-ratchet-packet — an escaped bug or outage fix needs the smallest lasting regression protection

Capture the minimum useful frame:

Packet: change-risk-packet
Change type: API contract + DB migration
Hotspots: compatibility, migration safety, permissions
Decision point: PR + release
Current evidence: unit tests only

Rule: start from the packet the user already has. Do not demand an ideal QA template before doing useful work.

Step 2: Name the gate truth before choosing layers

Say which decision point is real right now:

• merge-gate-truth — branch-blocking evidence, required status checks, or review-blocking proof
• release-gate-truth — staging smoke, rollout safety, rollback notes, permissions, packaging, or human signoff
• scheduled-breadth-truth — nightly/cron/matrix coverage that improves confidence but should not block every PR

Rules:

• Do not let protected-branch tooling masquerade as the whole test strategy; it only enforces the blocking subset.
• Do not let release/platform checklists silently expand the PR gate when they are really launch or rollout ownership.
• If more than one gate is present, name the primary gate and the follow-up gate explicitly.

Step 3: Choose one primary policy mode

Pick exactly one primary mode:

• layer-selection — what validation layers prove the changed behavior?
• gate-shaping — what belongs in local vs PR vs release vs scheduled loops?
• flake-and-cost-policy — what should block, quarantine, move, or become informational?
• incident-regression-ratchet — what is the lowest layer that would have caught the escaped bug?
• release-confidence — what final smoke, checklist, or rollout proof is still honestly needed?

Optional: name one secondary mode, but do not flatten every testing conversation into the same checklist.

Step 4: Classify the risk tier and critical path

Use a small risk model:

• Tier 0 — low risk: docs, comments, dead code deletion, isolated rename, obvious config metadata
• Tier 1 — ordinary product change: routine feature or refactor with limited blast radius
• Tier 2 — high risk: public API, migration, auth, billing, external integrations, state machines, concurrency, background jobs
• Tier 3 — release-critical / incident-linked: escaped bugs, outage fixes, rollback-sensitive deploy paths, or critical customer journeys

Capture:

• critical paths and users affected
• failure cost: annoyance, feature break, trust damage, data loss, rollout risk, security risk
• evidence already present: tests, screenshots, previews, logs, contract notes, rollout notes, checklists
• the decision point: local confidence, PR/merge confidence, release confidence, or long-running scheduled breadth

If the change spans multiple tiers, plan for the highest one.

Step 5: Choose the smallest convincing layer mix

Use references/validation-matrix.md.

Default layer choices:

• Unit / component / service when logic, validation, branching, or mapping is the main risk
• Integration when wiring, DB semantics, middleware, serialization, jobs, or real dependency behavior matters
• Contract / API-level when response shapes, schemas, events, or cross-service/client boundaries changed
• Smoke / selective E2E when multiple layers must prove one critical end-to-end journey together
• Manual exploratory / checklist validation when visual nuance, device variation, operational edge cases, or human signoff is still the honest answer

Always say both:

• what is required now
• what is intentionally out of scope for now and why

Examples:

• “integration + contract now; no broad browser E2E because the user journey is unchanged”
• “release smoke plus rollout checklist; no new unit tests because the only risk lives in staging config and deployment behavior”

Step 6: Separate local, PR, release, and scheduled expectations

A useful policy brief does not pretend one suite fits every loop.

Define the smallest truthful gate split:

• Local — fast, cheap, developer-loop proof
• PR / merge — changed-surface confidence for risky paths
• Release — production-facing smoke, migration safety, rollout checks, or manual signoff items
• Scheduled / nightly — broad matrices, expensive combinations, compatibility sweeps, long-running suites

Rules of thumb:

• if a suite is too slow or flaky for PRs, move it deliberately instead of silently rerunning it forever
• if branch protection / required status checks are in play, name only the checks that truly must block merge
• if a release checklist exists, tie it back to the specific risk that still needs human proof
• if store/platform launch checklists are now the dominant work, route to the launch or delivery owner instead of stuffing them back into merge coverage
• if the packet is really just release coordination, say so instead of pretending every item is a test-layer choice

Step 7: Write explicit exception and flake rules

This step is where strategy becomes operational.

State:

• which checks are blocking vs informational
• when a flaky test should be quarantined, fixed, moved to scheduled, or removed from the gate
• what explanation is required when no new regression coverage is added
• whether coverage percentages matter here or are just background reporting
• whether manual verification is temporary, release-only, or an intentional long-term choice

Good defaults:

• repeated flake is a policy problem, not just a rerun ritual
• quarantining can reduce CI noise, but must keep owner + timeout + follow-up visible
• “coverage went up” is not proof that the risky scenario is protected
• escaped bugs should ratchet in the lowest-layer regression that would actually have caught them

Step 8: Route the next owner immediately

This skill owns policy and confidence decisions, not all downstream work.

Typical route-outs:

• backend-testing — write or repair the chosen API/service/database/fixture/contract tests
• debugging — investigate why a suite is red, flaky, or environment-specific right now
• code-review — judge whether one diff’s current evidence is good enough to approve
• deployment-automation — own rollout execution, staging/prod verification sequencing, rollback steps, or release runbooks once the gate is chosen
• game-ci-cd-pipeline — own engine/build pipeline implementation or stabilization when the problem is a game CI/CD surface, not policy selection
• steam-store-launch-ops — own Steam-specific launch/store/runbook work when the remaining proof is release checklist, page readiness, or launch timing rather than merge confidence
• web-accessibility / web-design-guidelines — handle accessibility-heavy or visual-governance validation packets
• performance-optimization / game-performance-profiler — handle benchmark, load, latency, or frame-budget policy when performance is the actual dominant risk

If the user asks “what should we test?” stay here. If they ask “how do we write or stabilize those tests?” route out.

Step 9: Produce one concise validation brief

Preferred format:

# Validation Strategy Brief

## Policy packet
- Packet:
- Gate truth:
- Primary mode:
- Risk tier:
- Decision point:

## Required validation now
1. [Layer] ... because ...
2. [Layer] ... because ...
3. [Manual / release check] ... because ...

## Gate split
- Local:
- PR:
- Release:
- Scheduled:

## Out of scope for now
- ...

## Exception / flake policy
- ...

## Next owner
- `backend-testing` / `debugging` / `code-review` / other

A short, explicit brief beats a giant testing manifesto. If the honest answer is “do less, but at the right layer,” say that directly.

Output format

Always return a validation strategy brief, gate-shaping memo, or regression-ratchet brief.

Required qualities:

• identify the packet already in hand
• name the real gate being decided before expanding into more layers
• choose one primary policy mode
• classify risk and critical path explicitly
• separate local, PR, release, and scheduled expectations when relevant
• explain intentional exclusions instead of hand-waving them away
• route implementation, debugging, review, accessibility, or performance work to the correct neighboring skill

Examples

Example 1: API + migration packet

Input

This PR changes an API contract and adds a DB migration. What validation should be required before merge?

Output sketch

• Packet: change-risk-packet
• Gate truth: merge-gate-truth
• Primary mode: layer-selection
• Risk tier: 2
• Required validation:

1. integration test for migration read/write path 2. contract/API check for response compatibility 3. release smoke on the highest-value consumer path

• Out of scope: broad browser E2E because the user flow is unchanged
• Next owner: backend-testing

Example 2: Flaky browser suite packet

Input

Our Playwright suite is slow and keeps flaking. What testing strategy should this repo adopt?

Output sketch

• Packet: flake-cost-packet
• Gate truth: merge-gate-truth with scheduled-breadth-truth follow-up
• Primary mode: flake-and-cost-policy
• Required change:

1. narrow PR browser coverage to critical journeys only 2. move broader combinations to scheduled runs 3. define quarantine/fix rules for repeated flake 4. shift confidence to lower-level integration/component checks where honest

• Next owner: backend-testing for implementation, debugging for current flake root cause

Example 3: Release-readiness packet

Input

Engineering says tests are green, but should we require anything else before this release?

Output sketch

• Packet: release-readiness-packet
• Gate truth: release-gate-truth
• Primary mode: release-confidence
• Required validation:

1. targeted staging smoke for the changed customer journey 2. migration / rollback checklist item if deploy shape changed 3. explicit note that no new broad regression sweep is required beyond scheduled coverage

• Route-out: accessibility-specific signoff to web-accessibility if the change is UI-state heavy

Example 4: Game launch checklist packet

Input

Our build passed CI, but we still have Steam release checklist items and packaging work. Does this stay here?

Output sketch

• Packet: release-readiness-packet
• Gate truth: release-gate-truth
• Primary mode: release-confidence
• Required validation: targeted final smoke plus the minimum proof that launch/build checklist items are satisfied
• Out of scope: expanding branch-blocking PR checks just because store/platform launch work remains
• Next owner: steam-store-launch-ops and/or game-ci-cd-pipeline

Best practices

1. Start from the packet already in hand, not from a favorite testing slogan. 2. Prefer the cheapest layer that still proves the risky behavior. 3. Keep local, PR, release, and scheduled loops distinct. 4. Treat flaky tests as a policy smell, not just a rerun inconvenience. 5. Tie release checklists back to actual risk instead of treating them as a separate universe. 6. State intentional exclusions so residual risk is visible. 7. Use escaped bugs to ratchet in the lowest-layer regression that would have caught them. 8. Make merge blockers, release-only proof, and scheduled breadth explicit instead of blending them together. 9. Route implementation to backend-testing, diagnosis to debugging, rollout execution to deployment-automation, platform/game launch work to steam-store-launch-ops or game-ci-cd-pipeline, and accessibility-heavy validation to web-accessibility. 10. Use manual validation when it is the honest answer, not as a shameful fallback. 11. One concise validation brief is more reusable than a giant testing manifesto.

References

• Martin Fowler — Test Pyramid
• Martin Fowler — The Practical Test Pyramid
• Google Testing Blog — Just Say No to More End-to-End Tests
• Selenium — Test Practices
• Playwright — Best Practices
• Pact Docs

Gate Truth and Release Handovers

Use this note when the core ambiguity is which gate is really being shaped rather than which test type sounds sophisticated.

1. Merge gate truth

Use this when the team is deciding what must block a PR or merge queue.

Typical signals:

• protected branches or required status checks are in play
• reviewers are asking what is mandatory before approval
• flaky or slow suites are slowing merges

Good output:

• one explicit blocking set
• one explicitly informational set
• one scheduled/non-blocking follow-up set if needed

Reminder: GitHub protected-branch rules enforce the blocking subset; they do not define the whole validation strategy for you.

2. Release gate truth

Use this when tests are mostly green but the remaining risk is rollout, launch, permissions, packaging, migrations, or human signoff.

Typical signals:

• staging smoke or rollback proof is missing
• deployment/runbook sequencing matters more than another unit test
• store/platform release checklists still have open items
• the team is asking “is this actually ready to ship?” rather than “can we merge?”

Good output:

• narrow release-only proof
• named launch/deploy owner
• explicit statement that PR blockers should not grow just because launch work remains

Typical handoffs:

• deployment-automation for rollout execution, rollback steps, post-deploy verification order, and release runbooks
• steam-store-launch-ops for Steam release checklist, store page, launch timing, or launch-ops proof
• game-ci-cd-pipeline for engine/build pipeline implementation or stabilization when the remaining work is pipeline-owned

3. Scheduled breadth truth

Use this when the useful coverage is real but too expensive, flaky, or broad to block every change.

Typical signals:

• matrix combinations, browser/device sweeps, or compatibility passes
• long-running data/job scenarios
• quarantine candidates that still need visibility

Good output:

• scheduled cadence
• ownership for follow-up
• explicit reason the suite is non-blocking

4. Quick tests for the boundary

Ask these questions: 1. If this evidence failed, should the PR be blocked right now? 2. If the answer is no, is the evidence release-only or scheduled-breadth proof? 3. If the remaining proof is platform/store/deploy execution, is testing-strategies still the owner?

If questions 2 or 3 dominate, route out.

5. Anti-patterns

• Expanding merge blockers because launch work exists elsewhere.
• Treating “required status checks” as proof that the right checks were chosen.
• Keeping flaky broad suites in PR gates just because no one named a scheduled owner.
• Using release checklists as a vague bucket instead of tying them to one remaining risk.

Handoff Boundaries

Use this file when the request is drifting away from testing policy and into implementation, diagnosis, or review.

Keep work in testing-strategies when

• the user asks what validation layers are required
• the repo needs merge / release / nightly gate policy
• the team needs a risk-based coverage recommendation
• the problem is over-broad, flaky, or expensive validation policy
• the question is whether a change deserves unit vs integration vs contract vs smoke coverage

Route to backend-testing when

• the policy is already chosen and the user now needs to write or repair API/service/database tests
• fixture/factory/seed/reset strategy is the main work
• mock vs fake vs containerized dependency decisions need stack-specific implementation detail
• the team needs concrete test-runner patterns or examples for backend code

Route to debugging when

• a test is already failing and the main question is why
• the suite is flaky because of an unknown root cause
• the task is reproducing, isolating, or verifying a failure rather than choosing policy

Route to code-review when

• a specific PR or diff needs reviewer judgment
• the question is whether the submitted evidence is convincing enough to approve or request changes
• the task is classifying review findings by severity

Route to performance-optimization when

• performance, throughput, latency, memory, or scale is the dominant risk
• the team needs benchmarking/load-test strategy more than functional validation policy

Route to web-accessibility or web-design-guidelines when

• accessibility, visual QA, responsive behavior, or design-system compliance is the main validation surface
• screenshots, keyboard navigation, or assistive-tech evidence dominate the review

Simple heuristic

• "What should we test?" → testing-strategies
• "How do we implement those tests?" → backend-testing
• "Why is this test failing?" → debugging
• "Is this PR ready?" → code-review

Intake Packets and Route-outs

Start from the validation packet the user already has. Do not force every question through a generic pyramid lecture.

change-risk-packet

Use when the request is tied to one concrete feature, bugfix, migration, contract change, auth change, UI flow change, or config/deploy change.

Capture:

• what changed
• who or what path is exposed to risk
• the current decision point: local, PR, release, or mixed
• what evidence already exists

Common outputs:

• one layer-mix recommendation
• one gate split
• one named next owner

Route out when:

• implementation of the chosen tests is now the main work → backend-testing
• the packet is really one diff approval decision → code-review
• the change is dominated by performance benchmarking → performance-optimization

gate-design-packet

Use when the team is debating what belongs in local, PR, release, and scheduled gates.

Capture:

• current gate pain: slow, flaky, expensive, noisy, or missing confidence
• which checks are blocking vs informational today
• where evidence arrives too late

Look for:

• one suite that should move to scheduled/nightly
• one missing release-only check
• one place where PR gates are broader than the risk justifies

Route out when:

• the next task is implementing or stabilizing a specific suite → backend-testing or debugging

flake-cost-packet

Use when the suite is noisy, slow, brittle, or frequently rerun.

Capture:

• which suite or layer is causing pain
• whether failures are known flake, unknown flake, or real regressions
• who owns quarantine/fix follow-up
• whether the suite is in the wrong gate

Good policy outcomes:

• blocking vs informational rules
• quarantine with owner + timeout + follow-up expectation
• moving broad coverage to scheduled runs
• lowering confidence to the smallest honest blocking gate

Route out when:

• the task becomes reproducing why one flaky test is failing now → debugging
• the task becomes tool-specific CI implementation → implementation or infra skill

release-readiness-packet

Use when a change is near deploy and the team needs final confidence, signoff, staging smoke, rollback notes, or checklist cleanup.

Capture:

• changed customer path or operational path
• rollout sensitivity: migration, feature flag, config, secret, queue, background job, dependency
• what production-facing proof is still missing

Look for:

• narrow smoke coverage
• migration / rollback / rollback-proof items
• explicit manual signoff when human judgment is still honest

Route out when:

• the work is now release execution rather than validation policy → deployment-automation
• the review is mostly UI/accessibility signoff → web-accessibility or web-design-guidelines

incident-ratchet-packet

Use when an escaped bug, outage fix, or severe regression needs lasting protection.

Capture:

• what escaped
• what layer could have caught it earliest
• whether the previous gap was missing automation, missing gate placement, or missing release checklist proof

Good outcome:

• the lowest-layer regression that would have caught the issue
• a small gate change if needed
• explicit avoidance of unrelated coverage expansion

Route out when:

• the system still needs root-cause investigation → debugging
• the next job is implementing the regression test → backend-testing

Quick route-out table

Rule of thumb

A good packet is the smallest artifact set that lets you make one honest confidence decision. If the packet cannot support that decision yet, ask for one missing artifact or route to the neighboring skill that owns the next step.

Validation Matrix

Use this matrix when choosing the minimum convincing validation plan.

By change type

By decision point

Risk reminders

• Tier 0: docs / dead code / isolated rename — light verification is often enough.
• Tier 1: normal feature or refactor — choose the smallest layer mix that proves the behavior.
• Tier 2: API, migration, auth, billing, integrations, concurrency — require stronger-than-unit evidence.
• Tier 3: escaped bug, outage fix, release-critical journey — add explicit regression and release confidence.

Anti-patterns

• Using browser E2E as the default proof for every change.
• Saying "coverage increased" instead of naming the protected scenario.
• Running slow flaky suites on every PR because nobody defined a better gate.
• Treating manual release checks as shameful when they are the honest answer for some risks.
• Confusing policy questions with test-implementation questions.