Qa Expert

QA Expert

Establish world-class QA testing processes for any software project using proven methodologies from Google Testing Standards and OWASP security best practices.

When to Use This Skill

Trigger this skill when:

• Setting up QA infrastructure for a new or existing project
• Writing standardized test cases (AAA pattern compliance)
• Executing comprehensive test plans with progress tracking
• Implementing security testing (OWASP Top 10)
• Filing bugs with proper severity classification (P0-P4)
• Generating QA reports (daily summaries, weekly progress)
• Calculating quality metrics (pass rate, coverage, gates)
• Preparing QA documentation for third-party team handoffs
• Enabling autonomous LLM-driven test execution

Quick Start

One-command initialization:

python scripts/init_qa_project.py <project-name> [output-directory]

What gets created:

• Directory structure (tests/docs/, tests/e2e/, tests/fixtures/)
• Tracking CSVs (TEST-EXECUTION-TRACKING.csv, BUG-TRACKING-TEMPLATE.csv)
• Documentation templates (BASELINE-METRICS.md, WEEKLY-PROGRESS-REPORT.md)
• Master QA Prompt for autonomous execution
• README with complete quickstart guide

For autonomous execution (recommended): See references/master_qa_prompt.md - single copy-paste command for 100x speedup.

Core Capabilities

1. QA Project Initialization

Initialize complete QA infrastructure with all templates:

python scripts/init_qa_project.py <project-name> [output-directory]

Creates directory structure, tracking CSVs, documentation templates, and master prompt for autonomous execution.

Use when: Starting QA from scratch or migrating to structured QA process.

2. Test Case Writing

Write standardized, reproducible test cases following AAA pattern (Arrange-Act-Assert):

1. Read template: assets/templates/TEST-CASE-TEMPLATE.md 2. Follow structure: Prerequisites (Arrange) → Test Steps (Act) → Expected Results (Assert) 3. Assign priority: P0 (blocker) → P4 (low) 4. Include edge cases and potential bugs

Test case format: TC-[CATEGORY]-[NUMBER] (e.g., TC-CLI-001, TC-WEB-042, TC-SEC-007)

Reference: See references/google_testing_standards.md for complete AAA pattern guidelines and coverage thresholds.

3. Test Execution & Tracking

Ground Truth Principle (critical):

• Test case documents (e.g., 02-CLI-TEST-CASES.md) = authoritative source for test steps
• Tracking CSV = execution status only (do NOT trust CSV for test specifications)
• See references/ground_truth_principle.md for preventing doc/CSV sync issues

Manual execution: 1. Read test case from category document (e.g., 02-CLI-TEST-CASES.md) ← always start here 2. Execute test steps exactly as documented 3. Update TEST-EXECUTION-TRACKING.csv immediately after EACH test (never batch) 4. File bug in BUG-TRACKING-TEMPLATE.csv if test fails

Autonomous execution (recommended): 1. Copy master prompt from references/master_qa_prompt.md 2. Paste to LLM session 3. LLM auto-executes, auto-tracks, auto-files bugs, auto-generates reports

Innovation: 100x faster vs manual + zero human error in tracking + auto-resume capability.

4. Bug Reporting

File bugs with proper severity classification:

Required fields:

• Bug ID: Sequential (BUG-001, BUG-002, ...)
• Severity: P0 (24h fix) → P4 (optional)
• Steps to Reproduce: Numbered, specific
• Environment: OS, versions, configuration

Severity classification:

• P0 (Blocker): Security vulnerability, core functionality broken, data loss
• P1 (Critical): Major feature broken with workaround
• P2 (High): Minor feature issue, edge case
• P3 (Medium): Cosmetic issue
• P4 (Low): Documentation typo

Reference: See BUG-TRACKING-TEMPLATE.csv for complete template with examples.

5. Quality Metrics Calculation

Calculate comprehensive QA metrics and quality gates status:

python scripts/calculate_metrics.py <path/to/TEST-EXECUTION-TRACKING.csv>

Metrics dashboard includes:

• Test execution progress (X/Y tests, Z% complete)
• Pass rate (passed/executed %)
• Bug analysis (unique bugs, P0/P1/P2 breakdown)
• Quality gates status (✅/❌ for each gate)

Quality gates (all must pass for release):

6. Progress Reporting

Generate QA reports for stakeholders:

Daily summary (end-of-day):

• Tests executed, pass rate, bugs filed
• Blockers (or None)
• Tomorrow's plan

Weekly report (every Friday):

• Use template: WEEKLY-PROGRESS-REPORT.md (created by init script)
• Compare against baseline: BASELINE-METRICS.md
• Assess quality gates and trends

Reference: See references/llm_prompts_library.md for 30+ ready-to-use reporting prompts.

7. Security Testing (OWASP)

Implement OWASP Top 10 security testing:

Coverage targets: 1. A01: Broken Access Control - RLS bypass, privilege escalation 2. A02: Cryptographic Failures - Token encryption, password hashing 3. A03: Injection - SQL injection, XSS, command injection 4. A04: Insecure Design - Rate limiting, anomaly detection 5. A05: Security Misconfiguration - Verbose errors, default credentials 6. A07: Authentication Failures - Session hijacking, CSRF 7. Others: Data integrity, logging, SSRF

Target: 90% OWASP coverage (9/10 threats mitigated).

Each security test follows AAA pattern with specific attack vectors documented.

Day 1 Onboarding

For new QA engineers joining a project, complete 5-hour onboarding guide:

Read: references/day1_onboarding.md

Timeline:

• Hour 1: Environment setup (database, dev server, dependencies)
• Hour 2: Documentation review (test strategy, quality gates)
• Hour 3: Test data setup (users, CLI, DevTools)
• Hour 4: Execute first test case
• Hour 5: Team onboarding & Week 1 planning

Checkpoint: By end of Day 1, environment running, first test executed, ready for Week 1.

Autonomous Execution (⭐ Recommended)

Enable LLM-driven autonomous QA testing with single master prompt:

Read: references/master_qa_prompt.md

Features:

• Auto-resume from last completed test (reads tracking CSV)
• Auto-execute test cases (Week 1-5 progression)
• Auto-track results (updates CSV after each test)
• Auto-file bugs (creates bug reports for failures)
• Auto-generate reports (daily summaries, weekly reports)
• Auto-escalate P0 bugs (stops testing, notifies stakeholders)

Benefits:

• 100x faster execution vs manual
• Zero human error in tracking
• Consistent bug documentation
• Immediate progress visibility

Usage: Copy master prompt, paste to LLM, let it run autonomously for 5 weeks.

Adapting for Your Project

Small Project (50 tests)

• Timeline: 2 weeks
• Categories: 2-3 (e.g., Frontend, Backend)
• Daily: 5-7 tests
• Reports: Daily summary only

Medium Project (200 tests)

• Timeline: 4 weeks
• Categories: 4-5 (CLI, Web, API, DB, Security)
• Daily: 10-12 tests
• Reports: Daily + weekly

Large Project (500+ tests)

• Timeline: 8-10 weeks
• Categories: 6-8 (multiple components)
• Daily: 10-15 tests
• Reports: Daily + weekly + bi-weekly stakeholder

Reference Documents

Access detailed guidelines from bundled references:

• `references/day1_onboarding.md` - 5-hour onboarding guide for new QA engineers
• `references/master_qa_prompt.md` - Single command for autonomous LLM execution (100x speedup)
• `references/llm_prompts_library.md` - 30+ ready-to-use prompts for specific QA tasks
• `references/google_testing_standards.md` - AAA pattern, coverage thresholds, fail-fast validation
• `references/ground_truth_principle.md` - Preventing doc/CSV sync issues (critical for test suite integrity)

Assets & Templates

Test case templates and bug report formats:

• `assets/templates/TEST-CASE-TEMPLATE.md` - Complete template with CLI and security examples

Scripts

Automation scripts for QA infrastructure:

• `scripts/init_qa_project.py` - Initialize QA infrastructure (one command setup)
• `scripts/calculate_metrics.py` - Generate quality metrics dashboard

Common Patterns

Pattern 1: Starting Fresh QA

1. python scripts/init_qa_project.py my-app ./
2. Fill in BASELINE-METRICS.md (document current state)
3. Write test cases using assets/templates/TEST-CASE-TEMPLATE.md
4. Copy master prompt from references/master_qa_prompt.md
5. Paste to LLM → autonomous execution begins

Pattern 2: LLM-Driven Testing (Autonomous)

1. Read references/master_qa_prompt.md
2. Copy the single master prompt (one paragraph)
3. Paste to LLM conversation
4. LLM executes all 342 test cases over 5 weeks
5. LLM updates tracking CSVs automatically
6. LLM generates weekly reports automatically

Pattern 3: Adding Security Testing

1. Read references/google_testing_standards.md (OWASP section)
2. Write TC-SEC-XXX test cases for each OWASP threat
3. Target 90% coverage (9/10 threats)
4. Document mitigations in test cases

Pattern 4: Third-Party QA Handoff

1. Ensure all templates populated
2. Verify BASELINE-METRICS.md complete
3. Package tests/docs/ folder
4. Include references/master_qa_prompt.md for autonomous execution
5. QA team can start immediately (Day 1 onboarding → 5 weeks testing)

Success Criteria

This skill is effective when:

• ✅ Test cases are reproducible by any engineer
• ✅ Quality gates objectively measured
• ✅ Bugs fully documented with repro steps
• ✅ Progress visible in real-time (CSV tracking)
• ✅ Autonomous execution enabled (LLM can execute full plan)
• ✅ Third-party QA teams can start testing immediately

Test Case Template

Use this template for writing standardized, reproducible test cases.

---

Template Structure

### TC-[CATEGORY]-[NUMBER]: [Descriptive Title]

**Priority**: P0/P1/P2/P3/P4
**Type**: Unit/Integration/E2E/Security/Performance
**Estimated Time**: [X] minutes

**Prerequisites**:
- Prerequisite 1 (specific and verifiable)
- Prerequisite 2

**Test Steps**:
1. Step 1 (exact command or action)
2. Step 2 (specific input data)
3. Step 3 (verification action)

**Expected Result**:
✅ Specific outcome with example output
✅ Measurable verification criteria

**Pass/Fail Criteria**:
- ✅ PASS: All verification steps succeed
- ❌ FAIL: Any error, missing data, or deviation

**Potential Bugs to Watch For**:
- Known edge case or vulnerability
- Related security concern

---

Example: CLI Install Test

### TC-CLI-001: Install Skill from GitHub Repository

**Priority**: P0
**Type**: Integration
**Estimated Time**: 5 minutes

**Prerequisites**:
- CLI installed globally (`which ccpm` returns path)
- Internet connection active
- `~/.claude/skills/` directory exists or can be created
- No existing installation of `cli-demo-generator`

**Test Steps**:
1. Open terminal
2. Run: `ccpm install cli-demo-generator`
3. Observe success message
4. Run: `ls ~/.claude/skills/`
5. Verify directory exists
6. Run: `cat ~/.claude/skills/cli-demo-generator/package.json`
7. Verify valid JSON with name field

**Expected Result**:
✅ Terminal shows: "Successfully installed cli-demo-generator"
✅ Directory created: `~/.claude/skills/cli-demo-generator/`
✅ package.json exists with valid content
✅ No errors in terminal output

**Pass/Fail Criteria**:
- ✅ PASS: All 4 verification criteria met, exit code 0
- ❌ FAIL: Any error message, missing directory, or malformed package.json

**Potential Bugs to Watch For**:
- Path traversal vulnerability (test with `../../../etc/passwd`)
- Network timeout with no retry logic
- Incorrect permissions on `~/.claude` directory
- Race condition if multiple installs concurrent

---

Example: Security Test

### TC-SEC-001: SQL Injection Protection - Login Form

**Priority**: P0
**Type**: Security
**Estimated Time**: 3 minutes

**Prerequisites**:
- Application running on http://localhost:8080
- Test user account exists: `test@example.com` / `password123`
- Database seeded with sample data

**Test Steps**:
1. Navigate to login page
2. Enter username: `admin' OR '1'='1`
3. Enter password: `anything`
4. Click "Login" button
5. Observe response

**Expected Result**:
✅ Login FAILS with error: "Invalid credentials"
✅ SQL injection attempt logged in security_events table
✅ No database data exposed in error message
✅ User NOT authenticated

**Pass/Fail Criteria**:
- ✅ PASS: Login fails, injection logged, no data leak
- ❌ FAIL: Login succeeds, no logging, or SQL error exposed

**Potential Bugs to Watch For**:
- Verbose error messages exposing schema
- Second-order SQL injection in profile fields
- NoSQL injection if using MongoDB
- Timing-based blind SQL injection

---

Guidelines

Writing Clear Prerequisites

❌ Bad: "System running" ✅ Good: "Docker containers healthy (docker ps shows 5 running), port 8080 accessible"

Writing Specific Steps

❌ Bad: "Test the login" ✅ Good: "Enter 'test@example.com' in email field, enter 'Password123!' in password field, click 'Login' button"

Writing Measurable Results

❌ Bad: "It works" ✅ Good: "HTTP 200 response, redirects to /dashboard, session cookie set with 30min expiry"

Estimating Time

• Simple validation: 1-2 min
• API call test: 2-3 min
• E2E workflow: 5-10 min
• Security audit: 3-5 min per test

---

Category Codes

• CLI: Command-line interface tests
• WEB: Web UI tests
• API: Backend API tests
• DB: Database tests
• SEC: Security tests
• PERF: Performance tests
• INT: Integration tests
• E2E: End-to-end tests

---

Priority Assignment Rules

Assign P0 if:

• Prevents core functionality
• Security vulnerability (OWASP Top 10)
• Data loss or corruption
• System crash

Assign P1 if:

• Major feature broken (with workaround)
• Significant UX degradation
• Performance regression >50%

Assign P2 if:

• Minor feature issue
• Edge case failure
• Non-critical bug

Assign P3/P4 for cosmetic or documentation issues.

---

Usage: Copy this template when writing new test cases. Replace all bracketed placeholders with actual values.

Day 1 Onboarding Checklist

Purpose: Complete 5-hour onboarding guide for new QA engineers joining a software testing project.

Time: 5 hours (with breaks)

---

Hour 1: Environment Setup (60 min)

1.1 Clone Repository & Install Dependencies

git clone <repository-url>
cd <project-dir>
pnpm install  # or npm install

1.2 Start Local Database (if using Supabase/PostgreSQL)

npx supabase start  # Wait 2-3 minutes for all containers
docker ps | grep supabase  # Verify 8-11 containers running

1.3 Configure Environment Variables

cp .env.example .env
# Edit .env with local development URLs

1.4 Apply Database Migrations

# Apply all migrations in order
for file in database/migrations/*.sql; do
  docker exec -i <db-container-name> psql -U postgres -d postgres < "$file"
done

1.5 Verify Database Seeded

docker exec <db-container-name> psql -U postgres -d postgres -c "SELECT COUNT(*) FROM <main-table>;"
# Should return expected row count

1.6 Start Development Server

pnpm dev
# Verify: http://localhost:8080 (or configured port)

Checkpoint: ✅ Environment running, database seeded, website loads correctly.

---

Hour 2: Documentation Review (60 min)

2.1 Read Quick Start Guide (30 min)

• Understand project scope (total test cases, timeline)
• Identify test categories (CLI, Web, API, Security)
• Memorize quality gates (pass rate target, P0 bug policy)
• Review execution schedule (Week 1-5 plan)

2.2 Review Test Strategy (30 min)

• Understand AAA pattern (Arrange-Act-Assert)
• Learn bug classification (P0-P4 severity levels)
• Study test case format (TC-XXX-YYY numbering)
• Review OWASP security coverage target

Checkpoint: ✅ Strategy understood, test case format memorized.

---

Hour 3: Test Data Setup (60 min)

3.1 Create Test Users (20 min)

Via UI (if auth page available): 1. Navigate to /auth or /signup 2. Create 5 regular test users 3. Create 1 admin user 4. Create 1 moderator user

Via SQL (assign roles):

INSERT INTO user_roles (user_id, role)
SELECT id, 'admin'
FROM auth.users
WHERE email = 'admin@test.com';

3.2 Install CLI for Testing (20 min)

# Global installation (for testing `ccpm` command directly)
cd packages/cli
pnpm link --global

# Verify
ccpm --version
ccpm --help

3.3 Configure Browser DevTools (20 min)

• Install React Developer Tools extension
• Set up network throttling presets (Slow 3G, Fast 3G, Fast 4G)
• Configure responsive design mode (Mobile, Tablet, Desktop viewports)
• Test viewport switching

Checkpoint: ✅ Test users created, CLI installed, DevTools configured.

---

Hour 4: Execute First Test Case (60 min)

4.1 Open Test Execution Tracking Spreadsheet (5 min)

• File: tests/docs/templates/TEST-EXECUTION-TRACKING.csv
• Open in Google Sheets, Excel, or LibreOffice Calc
• Find first test case: TC-CLI-001 or equivalent

4.2 Read Full Test Case Documentation (10 min)

• Locate test case in documentation (e.g., 02-CLI-TEST-CASES.md)
• Read: Prerequisites, Test Steps, Expected Result, Pass/Fail Criteria

4.3 Execute TC-001 (20 min)

Example (CLI install command):

# Step 1: Clear previous installations
rm -rf ~/.claude/skills/<skill-name>

# Step 2: Run install command
ccpm install <skill-name>

# Step 3: Verify installation
ls ~/.claude/skills/<skill-name>
cat ~/.claude/skills/<skill-name>/package.json

4.4 Document Test Results (15 min)

Update TEST-EXECUTION-TRACKING.csv:

If test failed: 1. Open BUG-TRACKING-TEMPLATE.csv 2. Create new bug entry (Bug ID: BUG-001, BUG-002, etc.) 3. Fill in: Title, Severity (P0-P4), Steps to Reproduce, Environment 4. Link bug to test case in tracking CSV

4.5 Celebrate! (10 min)

✅ First test executed successfully!

Checkpoint: ✅ First test case executed and documented.

---

Hour 5: Team Onboarding & Planning (60 min)

5.1 Meet the Team (20 min)

Scheduled meeting with:

• QA Lead (your manager)
• QA Engineers (peers)
• Engineering Lead (answers technical questions)
• DevOps Lead (handles infrastructure)

Agenda: 1. Introductions 2. Project overview and goals 3. Your role and responsibilities 4. Q&A and troubleshooting

5.2 Review Week 1 Plan (20 min)

With QA Lead: Review weekly execution schedule.

Example Week 1: CLI Testing (93 test cases)

Discuss:

• Any blockers from today's setup?
• Confident with tools and documentation?
• Adjustments needed?

5.3 Bookmark Critical Resources (10 min)

Create browser bookmarks folder: "Project QA Resources"

Essential links:

• Local website (http://localhost:8080 or configured port)
• Database admin UI (Supabase Studio, phpMyAdmin, etc.)
• GitHub repository
• Test case documents
• Tracking spreadsheets

5.4 Final Q&A (10 min)

Common questions:

Q: What if I find a critical bug (P0) on Day 1? A: Immediately notify QA Lead. Document in bug tracker. P0 bugs block release and must be fixed within 24 hours.

Q: What if I can't complete all test cases in a day? A: Prioritize P0 tests first. Update QA Lead by end of day. Schedule can be adjusted.

Q: Can I run tests in a different order? A: Yes, but follow priority order (P0 → P1 → P2 → P3). Update tracking spreadsheet.

Q: What if a test case is unclear? A: Ask in team Slack/chat. Document question in tracking spreadsheet for future improvement.

Checkpoint: ✅ Team met, Week 1 plan reviewed, all questions answered.

---

Day 1 Completion Checklist

Before leaving for the day, verify all setup complete:

Environment

• [ ] Repository cloned and dependencies installed
• [ ] Database running (Docker containers or hosted instance)
• [ ] .env file configured with correct URLs/keys
• [ ] Database migrations applied and data seeded
• [ ] Development server running and website loads

Tools

• [ ] CLI installed (if applicable) - global and/or local
• [ ] Browser DevTools configured with extensions
• [ ] Network throttling presets added
• [ ] Responsive design mode tested

Test Data

• [ ] Regular test users created (5+ users)
• [ ] Admin user created with role assigned
• [ ] Moderator user created with role assigned (if applicable)

Documentation

• [ ] Quick Start Guide read (understand scope, timeline)
• [ ] Test Strategy reviewed (understand AAA pattern, quality gates)
• [ ] First test case executed successfully
• [ ] Test results documented in tracking spreadsheet

Team

• [ ] Team introductions completed
• [ ] Week 1 plan reviewed with QA Lead
• [ ] Critical resources bookmarked
• [ ] Communication channels joined (Slack, Teams, etc.)

---

Next Steps: Week 1 Testing Begins

Monday Morning Kickoff: 1. Join team standup (15 min) 2. Review any blockers from Day 1 setup 3. Begin Week 1 test execution (follow documented schedule)

Daily Routine:

• Morning: Team standup (15 min)
• Morning session: Test execution (9:15 AM - 12:00 PM)
• Lunch: Break (12:00 PM - 1:00 PM)
• Afternoon session: Test execution (1:00 PM - 5:00 PM)
• End of day: Update tracking, file bugs, status report (5:00 PM - 5:30 PM)

Weekly Deliverable: Friday EOD - Submit weekly progress report to QA Lead.

---

Troubleshooting Common Day 1 Issues

Issue 1: Database Containers Won't Start

Symptoms: Database service fails or containers show "unhealthy"

Fixes: 1. Restart database service (Docker Desktop, systemd, etc.) 2. Check logs: docker logs <container-name> 3. Verify ports not in use: lsof -i :<port-number> 4. Prune old containers (⚠️ caution): docker system prune

Issue 2: Website Shows "Failed to Fetch Data"

Symptoms: Homepage loads but data sections are empty

Fixes: 1. Verify database has seeded data: SELECT COUNT(*) FROM <table>; 2. Check API connection (network tab in DevTools) 3. Verify .env file has correct database URL 4. Restart dev server

Issue 3: CLI Command Not Found After Installation

Symptoms: <command>: command not found after installation

Fixes: 1. Check installation path: which <command> or pnpm bin -g 2. Add to PATH: export PATH="$(pnpm bin -g):$PATH" 3. Make permanent: Add to ~/.bashrc or ~/.zshrc 4. Reload shell: source ~/.bashrc

Issue 4: Test Users Not Showing Roles

Symptoms: Admin user can't access admin-only features

Fixes: 1. Verify role insert: SELECT * FROM user_roles WHERE user_id = '<user-id>'; 2. Sign out and sign in again (roles cached in session) 3. Clear browser cookies and local storage (F12 → Application tab)

---

Congratulations! 🎉

You've completed Day 1 onboarding. You're now ready to execute the full test plan.

Questions or Blockers?

• Slack/Teams: #qa-team channel
• Email: qa-lead@company.com
• Escalation: Engineering Lead (for critical bugs)

See you Monday for Week 1 testing! 🚀

Google Testing Standards Reference

Comprehensive guide to Google's testing best practices and standards.

---

AAA Pattern (Arrange-Act-Assert)

Every test should follow this structure:

1. Arrange (Setup)

**Prerequisites**:
- System in known state
- Test data prepared
- Dependencies mocked/configured

2. Act (Execute)

**Test Steps**:
1. Perform action
2. Trigger behavior
3. Execute operation

3. Assert (Verify)

**Expected Result**:
✅ Verification criteria
✅ Observable outcomes
✅ System state validation

---

Test Case Design Principles

1. Test Case ID Convention

TC-[CATEGORY]-[NUMBER]

Examples:
- TC-CLI-001 (CLI tests)
- TC-WEB-042 (Web tests)
- TC-API-103 (API tests)
- TC-SEC-007 (Security tests)

2. Priority Classification

P0 (Blocker) - Must fix before release

• Prevents core functionality
• Security vulnerabilities (SQL injection, XSS)
• Data corruption/loss
• System crashes

P1 (Critical) - Fix within 2 weeks

• Major feature broken with workaround
• Significant UX degradation
• Performance issues

P2 (High) - Fix within 4 weeks

• Minor feature issues
• Edge cases
• Non-critical bugs

P3 (Medium) - Fix when possible

• Cosmetic issues
• Rare edge cases
• Nice-to-have improvements

P4 (Low) - Optional

• Documentation typos
• Minor UI alignment

3. Test Types

Unit Tests:

• Test individual functions/methods
• No external dependencies
• Fast execution (<100ms)
• Coverage: ≥80% statements, 75% branches

Integration Tests:

• Test component interactions
• Real dependencies (database, APIs)
• Moderate execution time
• Coverage: Critical user journeys

E2E Tests:

• Test complete user workflows
• Real browser/environment
• Slow execution
• Coverage: Happy paths + critical failures

Security Tests:

• OWASP Top 10 coverage
• Input validation
• Authentication/authorization
• Data protection

---

Coverage Thresholds

Code Coverage Targets

• ✅ Statements: ≥80%
• ✅ Branches: ≥75%
• ✅ Functions: ≥85%
• ✅ Lines: ≥80%

Test Distribution (Recommended)

• Unit Tests: 70%
• Integration Tests: 20%
• E2E Tests: 10%

---

Test Isolation

Mandatory Principles

1. No Shared State

   ❌ BAD: Tests share global variables
   ✅ GOOD: Each test has independent data

2. Fresh Data Per Test

   beforeEach(() => {
     database.seed(freshData);
   });

3. Cleanup After Tests

   afterEach(() => {
     database.cleanup();
     mockServer.reset();
   });

---

Fail-Fast Validation

Critical Security Pattern

// ❌ BAD: Fallback to mock data
if (error) {
  return getMockData(); // WRONG - hides issues
}

// ✅ GOOD: Fail immediately
if (error || !data) {
  throw new Error(error?.message || 'Operation failed');
}

Input Validation

// Validate BEFORE any operations
function processSkillName(input: string): void {
  // Security checks first
  if (input.includes('..')) {
    throw new ValidationError('Path traversal detected');
  }

  if (input.startsWith('/')) {
    throw new ValidationError('Absolute paths not allowed');
  }

  // Then business logic
  return performOperation(input);
}

---

Test Documentation Standards

Test Case Template

### TC-XXX-YYY: Descriptive Title

**Priority**: P0/P1/P2/P3/P4
**Type**: Unit/Integration/E2E/Security
**Estimated Time**: X minutes

**Prerequisites**:
- Specific, verifiable conditions

**Test Steps**:
1. Exact command or action
2. Specific input data
3. Verification step

**Expected Result**:
✅ Measurable outcome
✅ Specific verification criteria

**Pass/Fail Criteria**:
- ✅ PASS: All verification steps succeed
- ❌ FAIL: Any error or deviation

**Potential Bugs**:
- Known edge cases
- Security concerns

---

Quality Gates

Release Criteria

Daily Checkpoints

Morning Standup:

• Yesterday's progress
• Today's plan
• Blockers

End-of-Day:

• Tests executed
• Pass rate
• Bugs filed
• Tomorrow's plan

Weekly Review

Friday Report:

• Week summary
• Baseline comparison
• Quality gates status
• Next week plan

---

Best Practices Summary

DO:

• ✅ Write reproducible test cases
• ✅ Update tracking after EACH test
• ✅ File bugs immediately on failure
• ✅ Follow AAA pattern
• ✅ Maintain test isolation
• ✅ Document environment details

DON'T:

• ❌ Skip test documentation
• ❌ Batch CSV updates
• ❌ Ignore security tests
• ❌ Use production data in tests
• ❌ Skip cleanup
• ❌ Hard-code test data

---

References:

• Google Testing Blog
• Google SWE Book - Testing
• Test Pyramid Concept

Ground Truth Principle - Preventing Documentation Sync Issues

Purpose: Prevent test suite integrity problems caused by documentation/tracking file mismatches.

Lesson Learned: CCPM project discovered 3.2% consistency rate between CSV and documentation (only 3 out of 93 test IDs matched correctly).

---

The Problem

Common Anti-Pattern

Projects often have multiple sources of truth:

• Test case documentation (e.g., 02-CLI-TEST-CASES.md)
• Execution tracking CSV (e.g., TEST-EXECUTION-TRACKING.csv)
• Bug tracking spreadsheet
• Test automation code

What goes wrong: 1. Documentation updated → CSV not updated 2. CSV auto-generated from old test list → docs finalized separately 3. Tests executed based on CSV → wrong test steps followed 4. Bug reports reference CSV IDs → cannot trace back to correct test

Real Example from CCPM

CSV TC-CLI-012: "Install Non-Existent Skill"

• Steps: Run ccpm install this-skill-does-not-exist-12345
• Expected: Clear error message

Doc TC-CLI-012: "Install Skill Already Installed"

• Steps: Run ccpm install cloudflare-troubleshooting (already installed)
• Expected: Warning message with --force hint

Result: Completely different tests! QA engineer might execute wrong test and report incorrect results.

---

The Ground Truth Principle

Rule #1: Single Source of Truth

Declare one file as authoritative for test specifications:

✅ CORRECT:
Ground Truth: 02-CLI-TEST-CASES.md (detailed test specifications)
Supporting: TEST-EXECUTION-TRACKING.csv (execution status only)

❌ WRONG:
CSV and docs both contain test steps (divergence inevitable)

Rule #2: Clear Role Separation

Rule #3: Explicit References

Always specify which file to use in instructions:

Good:

Execute test case TC-CLI-042:
1. Read full test specification from 02-CLI-TEST-CASES.md (pages 15-16)
2. Follow steps exactly as documented
3. Update TEST-EXECUTION-TRACKING.csv row TC-CLI-042 with result

Bad:

Execute test case TC-CLI-042 (no reference to source document)

---

Prevention Strategies

Strategy 1: Automated ID Validation

Script: validate_test_ids.py (generate this in your project)

#!/usr/bin/env python3
"""Validate test IDs between documentation and CSV"""

import csv
import re
from pathlib import Path

def extract_doc_ids(doc_path):
    """Extract all TC-XXX-YYY IDs from markdown documentation"""
    with open(doc_path, 'r') as f:
        content = f.read()
    pattern = r'TC-[A-Z]+-\d{3}'
    return set(re.findall(pattern, content))

def extract_csv_ids(csv_path):
    """Extract all Test Case IDs from CSV"""
    with open(csv_path, 'r') as f:
        reader = csv.DictReader(f)
        return set(row['Test Case ID'] for row in reader if row['Test Case ID'])

def validate_sync(doc_path, csv_path):
    """Check consistency between doc and CSV"""
    doc_ids = extract_doc_ids(doc_path)
    csv_ids = extract_csv_ids(csv_path)

    matching = doc_ids & csv_ids
    csv_only = csv_ids - doc_ids
    doc_only = doc_ids - csv_ids

    consistency_rate = len(matching) / len(csv_ids) * 100 if csv_ids else 0

    print(f"\n{'='*60}")
    print(f"Test ID Validation Report")
    print(f"{'='*60}\n")
    print(f"✅ Matching IDs:     {len(matching)}")
    print(f"⚠️  CSV-only IDs:     {len(csv_only)}")
    print(f"⚠️  Doc-only IDs:     {len(doc_only)}")
    print(f"\n📊 Consistency Rate: {consistency_rate:.1f}%\n")

    if consistency_rate < 100:
        print(f"❌ SYNC ISSUE DETECTED!\n")
        if csv_only:
            print(f"CSV IDs not in documentation: {sorted(csv_only)[:5]}")
        if doc_only:
            print(f"Doc IDs not in CSV: {sorted(doc_only)[:5]}")
    else:
        print(f"✅ Perfect sync!\n")

    return consistency_rate >= 95

if __name__ == "__main__":
    import sys
    if len(sys.argv) < 3:
        print("Usage: python validate_test_ids.py <doc-path> <csv-path>")
        sys.exit(1)

    doc_path = sys.argv[1]
    csv_path = sys.argv[2]

    valid = validate_sync(doc_path, csv_path)
    sys.exit(0 if valid else 1)

Usage:

python scripts/validate_test_ids.py \
  tests/docs/02-CLI-TEST-CASES.md \
  tests/docs/templates/TEST-EXECUTION-TRACKING.csv

# Output:
# ============================================================
# Test ID Validation Report
# ============================================================
#
# ✅ Matching IDs:     3
# ⚠️  CSV-only IDs:     90
# ⚠️  Doc-only IDs:     0
#
# 📊 Consistency Rate: 3.2%
#
# ❌ SYNC ISSUE DETECTED!

Strategy 2: ID Mapping Document

When mismatch detected, create bridge document:

File: tests/docs/TEST-ID-MAPPING.md

Contents:

# Test ID Mapping - CSV vs. Documentation

## Ground Truth
**Official Source**: 02-CLI-TEST-CASES.md
**Tracking File**: TEST-EXECUTION-TRACKING.csv (execution tracking only)

## ID Mapping Table
| CSV ID | Doc ID | Test Name | Match Status |
|--------|--------|-----------|--------------|
| TC-CLI-001 | TC-CLI-001 | Install Skill by Name | ✅ Match |
| TC-CLI-012 | TC-CLI-008 | Install Non-Existent Skill | ❌ Mismatch |

Strategy 3: CSV Usage Guide

Create explicit instructions for QA engineers:

File: tests/docs/templates/CSV-USAGE-GUIDE.md

Contents:

# TEST-EXECUTION-TRACKING.csv Usage Guide

## ✅ Correct Usage

1. **ALWAYS use test case documentation** as authoritative source for:
   - Test steps
   - Expected results
   - Prerequisites

2. **Use this CSV ONLY for**:
   - Tracking execution status
   - Recording results (PASSED/FAILED)
   - Linking to bug reports

## ❌ Don't Trust CSV for Test Specifications

---

Recovery Workflow

When you discover a sync issue:

Step 1: Assess Severity

# Run ID validation script
python scripts/validate_test_ids.py <doc> <csv>

# Consistency Rate:
#   100%:   ✅ No action needed
#   90-99%: ⚠️  Minor fixes needed
#   50-89%: 🔴 Major sync required
#   <50%:   🚨 CRITICAL - regenerate CSV

Step 2: Create Bridge Documents

# If consistency < 100%, create:
1. TEST-ID-MAPPING.md (maps CSV → Doc IDs)
2. CSV-USAGE-GUIDE.md (instructs QA engineers)

Step 3: Notify Team

Subject: [URGENT] Test Suite Sync Issue - Read Before Testing

Team,

We discovered a test ID mismatch between CSV and documentation:
- Consistency Rate: 3.2% (only 3 out of 93 tests match)
- Impact: Tests executed based on CSV may use wrong steps
- Action Required: Read CSV-USAGE-GUIDE.md before continuing

Ground Truth: 02-CLI-TEST-CASES.md (always trust this)
Tracking Only: TEST-EXECUTION-TRACKING.csv

Bridge: TEST-ID-MAPPING.md (maps IDs)

Step 4: Re-validate Executed Tests

Tests executed before fix may need re-verification:
- TC-CLI-001~003: ✅ Correct (IDs matched)
- TC-CLI-029: ⚠️  Verify against Doc TC-CLI-029
- TC-CLI-037: ⚠️  Verify against Doc TC-CLI-037

Step 5: Long-Term Fix

Option A: Maintain separation (recommended during active testing)

• CSV = execution tracking only
• Doc = test specifications
• Mapping doc bridges gap

Option B: Regenerate CSV from docs (post-testing)

• Risk: Loss of execution history
• Benefit: Perfect sync
• Timeline: After current test cycle

---

Best Practices

DO ✅

1. Declare ground truth upfront in project README 2. Separate concerns: Specs vs. tracking vs. bugs 3. Validate IDs regularly (weekly or before major milestones) 4. Document deviations in mapping file 5. Train QA team on ground truth principle

DON'T ❌

1. ❌ Duplicate test steps in multiple files 2. ❌ Auto-generate tracking files without validation 3. ❌ Execute tests based on CSV alone 4. ❌ Assume "it's just tracking" - IDs matter! 5. ❌ Ignore small mismatches (3% → 50% quickly)

---

Checklist for QA Project Setup

When using init_qa_project.py, ensure:

• [ ] Ground truth declared in README
• [ ] CSV contains ID + tracking fields only (no detailed steps)
• [ ] Test case docs are complete before CSV generation
• [ ] ID validation script added to project
• [ ] CSV usage guide included in templates/
• [ ] QA engineers trained on which file to trust

---

Integration with qa-expert Skill

When initializing a project with qa-expert:

python scripts/init_qa_project.py my-app ./

# This creates:
tests/docs/
  ├── README.md                        (declares ground truth)
  ├── 02-CLI-TEST-CASES.md            (authoritative specs)
  ├── TEST-ID-MAPPING.md              (if needed)
  └── templates/
      ├── TEST-EXECUTION-TRACKING.csv (tracking only)
      ├── CSV-USAGE-GUIDE.md          (usage instructions)
      └── validate_test_ids.py        (validation script)

---

Success Criteria

Your test suite has good integrity when:

• ✅ ID consistency rate ≥ 95%
• ✅ QA engineers know which file to trust
• ✅ Tracking CSV contains status only (no steps)
• ✅ Validation script runs weekly
• ✅ Team trained on ground truth principle

Red flags:

• 🚩 Multiple files contain test steps
• 🚩 CSV test names differ from docs
• 🚩 QA engineers "prefer" CSV over docs
• 🚩 No one knows which file is authoritative
• 🚩 Test IDs diverge over time

---

Document Version: 1.0 Created: 2025-11-10 Based On: CCPM test suite integrity incident (3.2% consistency rate) Priority: 🔴 P0 (Critical for test suite quality)

LLM QA Testing Prompts Library

Purpose: Ready-to-use prompts for directing LLM assistants to execute specific QA tasks.

Last Updated: 2025-11-09

---

Quick Navigation

1. Day 1 Onboarding 2. Weekly Execution 3. Daily Progress 4. Bug Investigation 5. Weekly Reporting 6. Emergency Escalation

---

Day 1 Onboarding

Initial Setup

You are a senior QA engineer with 20+ years of experience at Google. Help me set up the QA testing environment.

CRITICAL: Follow the Day 1 onboarding checklist exactly as documented.

Read and execute: tests/docs/templates/DAY-1-ONBOARDING-CHECKLIST.md

Start with Hour 1 (Environment Setup). Complete each hour sequentially. Do NOT skip any steps. After completing each hour, confirm what you did and ask if you should continue.

Report any blockers immediately.

Verify Setup Completion

Verify that my Day 1 QA onboarding is complete by checking:

1. All database containers are healthy (docker ps)
2. Database has seeded data (SELECT COUNT(*) FROM <table>)
3. Test users created (regular, admin, moderator)
4. CLI installed (if applicable) - global and/or local
5. Dev server running at http://localhost:8080
6. First test case executed successfully

Read tests/docs/templates/DAY-1-ONBOARDING-CHECKLIST.md section "Day 1 Completion Checklist" and verify ALL items are checked.

If anything is missing, tell me what needs to be fixed and how to fix it.

---

Weekly Execution

Week 1: Start Testing

You are a senior QA engineer executing Week 1 of the QA test plan.

CRITICAL: Follow the test plan exactly as documented.

Read: tests/docs/02-CLI-TEST-CASES.md (or appropriate test category document)

Your task today (Monday, Week 1):
- Execute test cases TC-CLI-001 through TC-CLI-015 (15 tests)
- Update tests/docs/templates/TEST-EXECUTION-TRACKING.csv after EACH test
- File bugs in tests/docs/templates/BUG-TRACKING-TEMPLATE.csv for any failures
- Expected time: 5 hours

Execute tests in order. For each test:
1. Read the full test case specification
2. Execute all test steps exactly as documented
3. Record result in TEST-EXECUTION-TRACKING.csv immediately
4. If test fails, create bug report in BUG-TRACKING-TEMPLATE.csv before moving to next test

After completing all 15 tests, give me a summary:
- How many passed/failed/blocked
- Bug IDs filed (if any)
- Any blockers for tomorrow

Daily Continuation

Continue Week [N] testing.

Read: tests/docs/[CATEGORY]-TEST-CASES.md

Today's test cases: TC-[CATEGORY]-[START] through TC-[CATEGORY]-[END] ([N] tests)

Follow the same process as yesterday:
1. Execute each test exactly as documented
2. Update TEST-EXECUTION-TRACKING.csv immediately after each test
3. File bugs for any failures
4. Give me end-of-day summary

Start now.

Friday - Week Completion

Complete Week [N] testing and submit weekly progress report.

Tasks:
1. Execute remaining tests: TC-[CATEGORY]-[START] through TC-[CATEGORY]-[END] ([N] tests)
2. Update TEST-EXECUTION-TRACKING.csv for all completed tests
3. Generate weekly progress report using tests/docs/templates/WEEKLY-PROGRESS-REPORT.md

For the weekly report:
- Calculate pass rate for Week [N] (passed / total executed)
- Summarize all bugs filed this week (by severity: P0/P1/P2/P3)
- Compare against baseline: tests/docs/BASELINE-METRICS.md
- Assess quality gates: Are we on track for 80% pass rate?
- Plan for Week [N+1]

Submit the completed WEEKLY-PROGRESS-REPORT.md.

---

Daily Progress

Morning Standup

Daily standup for QA testing.

Current status:
- Week: [1-5]
- Day: [Monday-Friday]
- Yesterday's progress: [X] tests executed, [Y] passed, [Z] failed
- Blockers: [None / List blockers]

Today's plan:
- Test cases: TC-[XXX]-[YYY] to TC-[XXX]-[ZZZ] ([N] tests)
- Expected time: [X] hours
- Prerequisites: [Any setup needed]

Read today's test cases from the appropriate document and confirm you're ready to start.

Mid-Day Progress Check

Give me a mid-day progress update.

How many test cases have you completed so far today?
How many passed vs failed?
Any bugs filed? (provide Bug IDs)
Any blockers preventing you from continuing?
Are you on track to finish today's test cases?

Update: tests/docs/templates/TEST-EXECUTION-TRACKING.csv with latest results before answering.

End-of-Day Summary

Provide end-of-day summary for QA testing.

Today's results:
- Test cases executed: [X] / [Y] planned
- Pass rate: [Z]%
- Bugs filed: [List Bug IDs with severity]
- Test execution tracking updated: Yes/No
- Bug reports filed: Yes/No

Tomorrow's plan:
- Test cases: TC-[XXX]-[YYY] to TC-[XXX]-[ZZZ]
- Prerequisites: [Any setup needed]
- Estimated time: [X] hours

Blockers:
- [None / List blockers]

If you didn't finish today's test cases, explain why and how you'll catch up.

---

Bug Investigation

Investigate Test Failure

A test case failed. I need you to investigate the root cause.

Test Case: TC-[CATEGORY]-[NUMBER]
Expected Result: [Copy from test case spec]
Actual Result: [What happened]

Your investigation:
1. Re-run the test case exactly as documented
2. Capture detailed logs, screenshots, network traces
3. Check if this is a test environment issue vs real bug
4. Determine severity: P0/P1/P2/P3/P4
5. Search for similar issues in existing bug reports

If confirmed as a bug:
- Create bug report in BUG-TRACKING-TEMPLATE.csv
- Assign unique Bug ID (BUG-XXX)
- Complete ALL fields (Steps to Reproduce, Environment, Screenshots, etc.)
- Update TEST-EXECUTION-TRACKING.csv with Bug ID reference

If NOT a bug (e.g., environment issue):
- Fix the environment issue
- Re-run the test
- Update TEST-EXECUTION-TRACKING.csv with PASS result

Report your findings.

Reproduce Bug from Report

I need you to reproduce a bug to verify it's still an issue.

Bug ID: BUG-[XXX]
Read: tests/docs/templates/BUG-TRACKING-TEMPLATE.csv (find Bug ID BUG-[XXX])

Steps:
1. Read the full bug report (Steps to Reproduce, Environment, etc.)
2. Set up the exact same environment
3. Execute the steps to reproduce exactly as documented
4. Verify you get the same Actual Result

If bug reproduces:
- Confirm "Yes, bug still exists"
- Add verification note to bug report

If bug does NOT reproduce:
- Explain what's different (environment, data, timing, etc.)
- Mark bug as "Cannot Reproduce" or "Fixed"

Report your findings.

Root Cause Analysis

Perform root cause analysis for a critical bug.

Bug ID: BUG-[XXX] (P0 or P1 severity)

Your analysis:
1. Understand the symptom (what the user sees)
2. Trace the data flow (where does the failure occur?)
3. Identify the root cause (what line of code / configuration is wrong?)
4. Assess impact (how many users affected? data loss? security risk?)
5. Propose fix (what needs to change to resolve this?)
6. Estimate fix complexity (hours/days to implement)

Read the relevant codebase files.
Check database state.
Review logs.

Document your findings in the bug report under "Root Cause Analysis" section.

---

Weekly Reporting

Generate Weekly Progress Report

Generate the weekly progress report for Week [1-5].

Read: tests/docs/templates/WEEKLY-PROGRESS-REPORT.md (use this template)

Data sources:
- tests/docs/templates/TEST-EXECUTION-TRACKING.csv (for test execution stats)
- tests/docs/templates/BUG-TRACKING-TEMPLATE.csv (for bug stats)
- tests/docs/BASELINE-METRICS.md (for comparison)

Fill in ALL sections:
1. Executive Summary (tests executed, pass rate, bugs found, blockers, on track status)
2. Test Execution Progress (table by category)
3. Bugs Filed This Week (P0/P1 highlights + summary table)
4. Test Execution Highlights (what went well, challenges, findings)
5. Quality Metrics (pass rate trend, bug discovery rate, test velocity)
6. Environment & Infrastructure (any issues?)
7. Next Week Plan (objectives, deliverables, risks)
8. Resource Needs (blockers, questions)
9. Release Readiness Assessment (quality gates status)

Calculate all metrics from actual data. Do NOT make up numbers.

Save the report as: tests/docs/reports/WEEK-[N]-PROGRESS-REPORT-2025-11-[DD].md

Compare Against Baseline

Compare current QA progress against the pre-QA baseline.

Read:
- tests/docs/BASELINE-METRICS.md (pre-QA state)
- tests/docs/templates/TEST-EXECUTION-TRACKING.csv (current state)
- tests/docs/templates/BUG-TRACKING-TEMPLATE.csv (bugs found)

Analysis:
1. Test execution: Baseline had [X] unit tests passing. How many total tests do we have now?
2. Pass rate: What's our QA test pass rate vs baseline pass rate?
3. Bugs discovered: Baseline started with [X] P0 bugs. How many P0/P1/P2/P3 bugs have we found?
4. Quality gates: Are we on track to meet 80% pass rate, zero P0 bugs policy?
5. Security: Have we maintained OWASP coverage?

Provide a comparison table showing:
- Metric | Baseline (YYYY-MM-DD) | Current (YYYY-MM-DD) | Delta | Status

Are we improving or regressing? What actions are needed?

---

Emergency Escalation

Escalate Critical Bug (P0)

URGENT: A P0 (Blocker) bug has been discovered.

Test Case: TC-[CATEGORY]-[NUMBER]
Bug ID: BUG-[XXX]
Severity: P0 (Blocks release, requires 24-hour fix)

Issue: [Brief description]

Immediate actions:
1. Stop all other testing immediately
2. Create detailed bug report in BUG-TRACKING-TEMPLATE.csv
3. Include:
   - Detailed steps to reproduce
   - Screenshots/videos of the issue
   - Full error logs
   - Environment details (OS, browser, Node version, etc.)
   - Impact assessment (how many users affected?)
   - Proposed workaround (if any)
4. Mark test case as "Blocked" in TEST-EXECUTION-TRACKING.csv
5. Notify:
   - QA Lead
   - Engineering Lead
   - Product Manager

Draft escalation email:

Subject: [P0 BLOCKER] [Brief description]

Body:
- What: [Issue description]
- When: [When discovered]
- Impact: [Severity and user impact]
- Test Case: TC-[XXX]-[YYY]
- Bug ID: BUG-[XXX]
- Next Steps: [What needs to happen to fix]
- ETA: [Expected fix time - must be within 24 hours]

Generate the bug report and escalation email now.

Resolve Blocker

A blocker has been resolved. I need you to verify the fix.

Bug ID: BUG-[XXX] (previously P0 blocker)
Status: Engineering reports "Fixed"
Test Case: TC-[CATEGORY]-[NUMBER] (originally failed)

Verification steps:
1. Read the bug report in BUG-TRACKING-TEMPLATE.csv
2. Understand what was fixed (check git commit if available)
3. Re-run the original test case exactly as documented
4. Verify the Expected Result now matches Actual Result

If fix is verified:
- Update BUG-TRACKING-TEMPLATE.csv:
  - Status: "Closed"
  - Resolution: "Fixed - Verified"
  - Resolved Date: [Today's date]
  - Verified By: [Your name/ID]
  - Verification Date: [Today's date]
- Update TEST-EXECUTION-TRACKING.csv:
  - Result: "PASS"
  - Notes: "Re-tested after BUG-[XXX] fix, now passing"

If fix is NOT verified (bug still exists):
- Update bug status: "Reopened"
- Add comment: "Fix verification failed - bug still reproduces"
- Re-escalate to Engineering Lead

Report verification results.

Environment Issues

The test environment is broken. I need you to diagnose and fix it.

Symptoms: [Describe what's not working]

Diagnostic steps:
1. Check database containers: docker ps | grep <db-name>
   - Are all containers running and healthy?
2. Check database connection: docker exec <container-name> psql -U postgres -d postgres -c "SELECT 1;"
   - Can you connect to the database?
3. Check data: docker exec <container-name> psql -U postgres -d postgres -c "SELECT COUNT(*) FROM <table>;"
   - Does the database still have seeded data?
4. Check dev server: curl http://localhost:8080
   - Is the dev server responding?
5. Check CLI (if applicable): ccpm --version
   - Is the CLI installed and working?

Refer to: tests/docs/templates/DAY-1-ONBOARDING-CHECKLIST.md section "Troubleshooting Common Day 1 Issues"

If you can fix the issue:
- Execute the fix
- Document what was broken and how you fixed it
- Verify the environment is fully operational
- Resume testing

If you cannot fix the issue:
- Document all diagnostic findings
- Escalate to Environment Engineer
- Mark affected tests as "Blocked" in TEST-EXECUTION-TRACKING.csv

Start diagnostics now.

---

Best Practices for Using These Prompts

1. Always Provide Context

Include relevant context before the prompt:

• Current week/day
• Previous day's results
• Known blockers
• Environment status

2. Be Specific

Replace all template variables with actual values:

• [CATEGORY]: CLI / WEB / API / SEC
• [NUMBER]: Test case number (e.g., 001, 015)
• [N]: Number of tests
• [XXX]: Bug ID number

3. Reference Documentation

Always point the LLM to specific documentation files:

• Test case specs: tests/docs/02-CLI-TEST-CASES.md
• Test strategy: tests/docs/01-TEST-STRATEGY.md
• Tracking: tests/docs/templates/TEST-EXECUTION-TRACKING.csv

4. Enforce Tracking

Always require the LLM to update tracking templates immediately after each test. Don't allow batch updates at end of day.

5. Verify Results

Ask the LLM to show you the updated CSV/Markdown files after making changes. Verify the data is correct.

6. Escalate Blockers

If the LLM reports a P0 bug or blocker, stop all other work and focus on that issue first.

---

Common Mistakes to Avoid

❌ Mistake 1: Vague Prompts

Bad: "Do some QA testing for me" Good: "Execute CLI test cases TC-CLI-001 through TC-CLI-015 following tests/docs/02-CLI-TEST-CASES.md. Update TEST-EXECUTION-TRACKING.csv after each test."

❌ Mistake 2: Skipping Tracking

Bad: "Run all the CLI tests and tell me the results" Good: "Execute TC-CLI-001. Update TEST-EXECUTION-TRACKING.csv with result. Then execute TC-CLI-002. Update CSV. Repeat for all tests."

❌ Mistake 3: Not Specifying Documentation

Bad: "Test the install command" Good: "Execute test case TC-CLI-001 from tests/docs/02-CLI-TEST-CASES.md. Follow the exact steps documented."

❌ Mistake 4: Allowing Deviations

Bad: "Test the CLI however you think is best" Good: "Execute ONLY the test cases documented in tests/docs/02-CLI-TEST-CASES.md. Do NOT add your own test cases. Do NOT skip test cases."

❌ Mistake 5: Batching Updates

Bad: "Run 15 tests and then update the tracking CSV" Good: "Execute TC-CLI-001. Update TEST-EXECUTION-TRACKING.csv immediately. Then execute TC-CLI-002. Update CSV immediately. Repeat."

---

Troubleshooting

LLM Not Following Test Plan

CRITICAL: You are deviating from the documented test plan.

STOP all current work.

Re-read: tests/docs/README.md

You MUST:
1. Follow the exact test case specifications
2. Execute test steps in the documented order
3. Update TEST-EXECUTION-TRACKING.csv after EACH test (not in batches)
4. File bugs in BUG-TRACKING-TEMPLATE.csv for any failures
5. NOT add your own test cases
6. NOT skip test cases
7. NOT modify test case priorities without approval

Acknowledge that you understand these requirements and will follow the documented test plan exactly.

Then resume testing from where you left off.

LLM Providing Incorrect Results

The test results you reported do not match my manual verification.

Test Case: TC-[CATEGORY]-[NUMBER]
Your Result: [PASS / FAIL]
My Result: [PASS / FAIL]

Re-execute this test case step-by-step:
1. Read the full test case spec from tests/docs/[DOCUMENT].md
2. Show me each test step as you execute it
3. Show me the actual output/result after each step
4. Compare the actual result to the expected result
5. Determine PASS/FAIL based on documented criteria (not your assumptions)

Be precise. Use exact command outputs, exact HTTP responses, exact UI text. Do NOT paraphrase or summarize.

---

Document Version: 1.0 Last Updated: 2025-11-09 Feedback: If you create new useful prompts, add them to this document.

Master QA Prompt - One Command for Autonomous Execution

Purpose: Single copy-paste prompt that directs LLM to execute entire QA test plan autonomously.

Innovation: 100x speedup vs manual testing + zero human error in tracking + auto-resume capability.

---

⭐ The Master Prompt

Copy this prompt and paste it into your LLM conversation. The LLM will handle everything automatically.

You are a senior QA engineer with 20+ years of experience at Google. Execute the QA test plan.

CRITICAL INSTRUCTIONS:
1. Read tests/docs/QA-HANDOVER-INSTRUCTIONS.md - Master handover guide
2. Read tests/docs/BASELINE-METRICS.md - Understand pre-QA baseline
3. Read tests/docs/templates/TEST-EXECUTION-TRACKING.csv - Check current progress
4. Determine current state:
   - If no tests executed yet → Start Day 1 onboarding (tests/docs/templates/DAY-1-ONBOARDING-CHECKLIST.md)
   - If Day 1 complete → Determine current week/day from TEST-EXECUTION-TRACKING.csv
   - If mid-week → Continue from last completed test case
5. Execute today's test cases:
   - Week 1: CLI tests (tests/docs/02-CLI-TEST-CASES.md)
   - Week 2: Web tests (tests/docs/03-WEB-TEST-CASES.md)
   - Week 3: API tests (tests/docs/04-API-TEST-CASES.md)
   - Week 4: Security tests (tests/docs/05-SECURITY-TEST-CASES.md)
   - Week 5: Regression tests (re-run all P0 tests)
6. For EACH test case:
   - Read full test case specification
   - Execute all steps exactly as documented
   - Update TEST-EXECUTION-TRACKING.csv immediately with result
   - If test fails → File bug in BUG-TRACKING-TEMPLATE.csv before continuing
7. End of day:
   - Provide summary (tests executed, pass rate, bugs filed, blockers)
   - If Friday → Generate WEEKLY-PROGRESS-REPORT.md
8. Compare progress against BASELINE-METRICS.md weekly

MANDATORY RULES:
- DO NOT skip tests
- DO NOT deviate from documented test cases
- DO NOT batch CSV updates (update after EACH test)
- DO NOT modify test priorities without approval
- STOP immediately if P0 bug discovered (escalate)

Ground truth: tests/docs/ directory
Test strategy: tests/docs/01-TEST-STRATEGY.md
Quality gates: ≥80% pass rate, zero P0 bugs, ≥80% coverage

Start now. Tell me what state you're in and what you're going to do today.

---

How It Works

Step 1: Assess Current State (2 minutes)

✅ Reading QA-HANDOVER-INSTRUCTIONS.md...
✅ Reading BASELINE-METRICS.md...
✅ Reading TEST-EXECUTION-TRACKING.csv...

Current State Detected:
- Day 1 onboarding: NOT COMPLETE
- Tests executed: 0 / 342
- Current week: N/A (need to complete Day 1 first)

Decision: Start Day 1 onboarding

Step 2: Execute Day 1 Onboarding (if not done yet)

Reading: tests/docs/templates/DAY-1-ONBOARDING-CHECKLIST.md

Starting Hour 1: Environment Setup
Task 1.1: Clone repository
  Executing: git clone https://github.com/your-org/project.git
  Status: ✅ Complete

Task 1.2: Install dependencies
  Executing: pnpm install
  Status: ✅ Complete

[... continues through all 5 hours ...]

Day 1 Onboarding: ✅ COMPLETE
Ready to start Week 1 testing tomorrow.

Step 3: Execute Daily Tests (if Day 1 done)

Current State Detected:
- Day 1 onboarding: ✅ COMPLETE
- Last completed test: TC-CLI-015
- Current week: Week 1 (CLI Testing)
- Current day: Tuesday
- Today's tests: TC-CLI-016 through TC-CLI-030 (15 tests)

Reading: tests/docs/02-CLI-TEST-CASES.md

Executing TC-CLI-016: Install Skill with --project Flag
  Step 1: Create test project directory
    Executing: mkdir /tmp/test-project && cd /tmp/test-project
    Status: ✅ Complete
  Step 2: Run ccpm install --project cli-demo-generator
    Executing: ccpm install --project cli-demo-generator
    Output: ✓ Skill installed to /tmp/test-project/.claude/skills/cli-demo-generator
    Status: ✅ PASS

Updating TEST-EXECUTION-TRACKING.csv:
  Row: TC-CLI-016
  Status: Completed
  Result: PASS
  Execution Date: 2025-11-08
  Executed By: QA Engineer
  Notes: Skill installed to project directory successfully
  ✅ CSV Updated

Executing TC-CLI-017: Install Skill with Invalid Name
[... continues for all 15 tests ...]

End of Day Summary:
✅ Tests executed: 15 / 15 (100%)
✅ Pass rate: 93% (14 passed, 1 failed)
🐛 Bugs filed: BUG-002 (P2 - Error message unclear for invalid skill name)
📊 TEST-EXECUTION-TRACKING.csv: Updated
📝 BUG-TRACKING-TEMPLATE.csv: Updated

Tomorrow (Wednesday):
- Tests: TC-CLI-031 through TC-CLI-045 (15 tests)
- Estimated time: 5.5 hours

Step 4: Weekly Reporting (if Friday)

Current State Detected:
- Current day: Friday (end of Week 1)
- Week 1 tests: 93 / 93 executed (100%)
- Pass rate: 91% (85 passed, 8 failed)

Generating WEEKLY-PROGRESS-REPORT.md:
  Reading template: tests/docs/templates/WEEKLY-PROGRESS-REPORT.md

  Executive Summary:
  - Tests Executed: 93 / 342 (27% complete)
  - Pass Rate: 91%
  - Bugs Found: 8 total (P0: 0, P1: 2, P2: 4, P3: 2)
  - Blockers: None
  - On Track: Yes

  [... fills in all sections ...]

  ✅ Report saved: tests/docs/reports/WEEK-1-PROGRESS-REPORT-2025-11-08.md

Comparing against BASELINE-METRICS.md:
  Baseline (2025-11-08):
  - Unit tests: 261 passing (100%)
  - P0 bugs: 0
  - Database: 18 skills, 32 tables

  Current (2025-11-08):
  - Total tests: 261 unit + 93 QA CLI = 354 tests
  - Overall pass rate: 96% (340 passed, 14 failed)
  - P0 bugs: 0 (maintaining baseline) ✅
  - Quality gates: On track for ≥80% target ✅

---

Auto-Resume Capability

If the LLM conversation gets interrupted or you need to pause, just paste the master prompt again.

The LLM will: 1. ✅ Read TEST-EXECUTION-TRACKING.csv 2. ✅ Find the last completed test case 3. ✅ Resume from the next test case 4. ✅ Continue exactly where you left off

Example:

Current State Detected:
- Last completed test: TC-CLI-042
- Current week: Week 1 (CLI Testing)
- Current day: Wednesday
- Remaining today: TC-CLI-043 through TC-CLI-045 (3 tests)

Resuming from TC-CLI-043...

---

Quality Gates Monitoring

The LLM automatically checks quality gates and alerts you if off track:

Pass Rate Check (Daily)

Quality Gate: Pass Rate ≥80%
Current: 91% ✅ PASSING
Status: On track

P0 Bugs Check (Daily)

Quality Gate: Zero P0 Bugs
Current: 0 P0 bugs ✅ PASSING
Status: Maintained

Code Coverage Check (Weekly)

Quality Gate: ≥80% Coverage
Current: 76% ⚠️ BELOW TARGET
Status: Need 4% improvement
Action: Add unit tests for uncovered modules

Release Readiness (End of Week 5)

RELEASE READINESS ASSESSMENT

Quality Gates Status:
✅ Test Execution: 100% (342/342 tests executed)
✅ Pass Rate: 87% (≥80% required) - PASS
✅ P0 Bugs: 0 (zero required) - PASS
⚠️ P1 Bugs: 7 (≤5 required) - FAIL (2 over limit)
✅ Code Coverage: 82% (≥80% required) - PASS
✅ Security: 90% OWASP coverage - PASS

Overall: ⚠️ AT RISK
Blocker: 2 excess P1 bugs must be fixed before release
ETA: 2 weeks (1 sprint)

RECOMMENDATION: Do NOT release until P1 bug count ≤5

---

Emergency Scenarios

P0 Bug Discovered

If the LLM discovers a P0 bug, it will:

🚨 CRITICAL: P0 Bug Discovered - STOPPING ALL TESTING

Bug ID: BUG-005
Test Case: TC-CLI-023
Severity: P0 (Blocker - Requires 24-hour fix)
Issue: Path traversal vulnerability allows reading /etc/passwd

Immediate Actions Taken:
✅ Bug report filed in BUG-TRACKING-TEMPLATE.csv (all fields complete)
✅ Test case marked as "Blocked" in TEST-EXECUTION-TRACKING.csv
✅ All further testing STOPPED

ESCALATION REQUIRED:
- Notify: QA Lead, Engineering Lead, Product Manager
- Required Fix Time: 24 hours
- Impact: Security vulnerability, release blocker

Escalation email drafted (see below).

Awaiting fix before resuming testing.

Environment Broken

If the environment breaks, the LLM will:

🔧 ENVIRONMENT ISSUE DETECTED

Issue: Supabase containers not responding
Diagnostic Results:
  ✅ Docker running: Yes
  ❌ Supabase containers: 3 unhealthy
  ❌ Database connection: Failed

Troubleshooting Steps Taken:
  1. Checked docker ps → 3 containers unhealthy
  2. Checked logs → Error: port 54321 already in use
  3. Attempted restart → Failed (port conflict)

RECOMMENDATION:
  1. Kill conflicting process: lsof -ti:54321 | xargs kill -9
  2. Restart Supabase: npx supabase start
  3. Verify: docker ps (all containers healthy)
  4. Resume testing from TC-CLI-042

Marking tests as "Blocked" until environment fixed.

---

Files Automatically Updated

When you run the master prompt, these files get updated automatically:

You don't need to manually update anything! The LLM handles all tracking automatically.

---

Success Criteria

After 5 weeks of using this master prompt, you should have:

✅ All 342 test cases executed (100% completion) ✅ TEST-EXECUTION-TRACKING.csv fully populated (all 342 rows with results) ✅ Bugs documented in BUG-TRACKING-TEMPLATE.csv (complete with P0-P4 classification) ✅ 5 weekly progress reports (one per week in tests/docs/reports/) ✅ Quality gates met (≥80% pass rate, zero P0 bugs, ≥80% coverage) ✅ Release readiness assessment (go/no-go decision based on objective criteria)

---

Daily Workflow Example

Day 1 (First Time)

You: [Paste master prompt]

LLM:
  Current State: No testing done yet
  Action: Starting Day 1 onboarding
  [Executes 5-hour onboarding checklist]
  Result: ✅ Day 1 complete, ready for Week 1 Monday

Day 2 (Monday, Week 1)

You: [Paste master prompt]

LLM:
  Current State: Day 1 complete, Week 1 not started
  Action: Execute TC-CLI-001 through TC-CLI-015 (15 tests)
  [Executes 15 CLI tests, updates CSV after each]
  Result: 14 passed, 1 failed, 1 bug filed

Day 3 (Tuesday, Week 1)

You: [Paste master prompt]

LLM:
  Current State: Last completed TC-CLI-015
  Action: Execute TC-CLI-016 through TC-CLI-030 (15 tests)
  [Continues testing from where left off]
  Result: 15 passed, 0 failed

Day 6 (Friday, Week 1)

You: [Paste master prompt]

LLM:
  Current State: Last completed TC-CLI-077
  Action: Execute TC-CLI-078 through TC-CLI-093 (16 tests) + Weekly report
  [Finishes Week 1 tests, generates weekly report]
  Result: Week 1 complete (93/93 tests), 91% pass rate, 8 bugs filed

Day 8 (Monday, Week 2)

You: [Paste master prompt]

LLM:
  Current State: Week 1 complete, Week 2 not started
  Action: Execute TC-WEB-001 through TC-WEB-015 (15 tests)
  [Switches to Web testing automatically]
  Result: 13 passed, 2 failed, 2 bugs filed

This continues for 5 weeks until all 342 test cases are executed!

---

Customizations (Optional)

Skip Day 1 Onboarding

Add this line to the prompt:

ASSUMPTION: Day 1 onboarding is already complete. Skip to test execution.

Execute Specific Tests

Add this line to the prompt:

TODAY ONLY: Execute test cases TC-CLI-020 through TC-CLI-035 (ignore normal schedule).

Focus on Bug Investigation

Add this line to the prompt:

PRIORITY: Investigate and reproduce Bug ID BUG-003 before continuing test execution.

Generate Weekly Report Only

Replace the master prompt with this shorter version:

You are a senior QA engineer. Generate the weekly progress report for Week [N].

Read:
- tests/docs/templates/WEEKLY-PROGRESS-REPORT.md (template)
- tests/docs/templates/TEST-EXECUTION-TRACKING.csv (test results)
- tests/docs/templates/BUG-TRACKING-TEMPLATE.csv (bug data)
- tests/docs/BASELINE-METRICS.md (baseline comparison)

Fill in ALL sections with actual data. Save report as:
tests/docs/reports/WEEK-[N]-PROGRESS-REPORT-2025-11-[DD].md

Start now.

---

Pro Tip: Bookmark this page and copy-paste the master prompt every morning. The LLM will handle the rest! 🚀