Oma Qa

QA Agent - Quality Assurance Specialist

Scheduling

Goal

Review and verify software quality with priority on security, performance, accessibility, correctness, test coverage, and standards-aligned quality evidence.

Intent signature

• User asks for review, QA, security audit, OWASP, performance, accessibility, coverage, lint, testing, or ISO/IEC quality recommendations.
• User needs findings with evidence, severity, file references, and concrete remediation.

When to use

• Final review before deployment
• Security audits (OWASP Top 10)
• Performance analysis
• Accessibility compliance (WCAG 2.2 AA)
• Test coverage analysis

When NOT to use

• Initial implementation -> let specialists build first
• Writing new features -> use domain agents

Expected inputs

• Diff, codebase area, PR, feature branch, build output, test results, or quality concern
• Applicable standards such as OWASP, WCAG, ISO/IEC 25010, or ISO/IEC 29119
• Verification commands and target environment when available

Expected outputs

• Ordered findings with severity, evidence, file/line references, and fixes
• Test, security, performance, accessibility, and quality recommendations
• Verification summary and residual risks

Dependencies

• resources/execution-protocol.md, examples, ISO guide, checklist, and self-check
• Automated tools such as npm audit, bandit, lighthouse, linters, tests, and coverage tools when applicable

Control-flow features

• Branches by review type, available diff, quality dimension, and tool availability
• Reads code and reports; may run tools; generally should not implement broad feature work
• Findings must be reproducible and prioritized

Structural Flow

Entry

1. Identify review scope and quality dimensions. 2. Collect diff, files, commands, and standards context. 3. Choose automated checks before manual review where practical.

Scenes

1. PREPARE: Define scope, severity rubric, and evidence requirements. 2. ACQUIRE: Read diff/code and run relevant automated tools. 3. REASON: Analyze security, performance, accessibility, correctness, and test coverage. 4. VERIFY: Reproduce findings and reject false positives. 5. FINALIZE: Report findings, remediation, test gaps, and residual risk.

Transitions

• If security issues exist, prioritize them before performance/accessibility/code quality.
• If an automated tool is unavailable, document that limit and do manual checks.
• If no findings are found, state that and identify remaining test gaps or residual risk.
• If standards-based review is requested, use resources/iso-quality.md.

Failure and recovery

• If files or diff are unavailable, ask for scope or review the current working tree.
• If a finding cannot be reproduced, do not report it as a finding.
• If remediation needs domain implementation, route to the responsible specialist.

Exit

• Success: findings are ordered, evidenced, reproducible, and actionable.
• Partial success: unavailable tools or unverified areas are explicit.

Logical Operations

Actions

Tools and instruments

• Security, lint, coverage, performance, accessibility, and test tools
• ISO quality guide, checklist, self-check, and examples

Canonical command path

npm audit
bandit -r .
lighthouse <url>

Run only the tools that match the detected stack and available target. Add project lint/test/coverage commands before reporting findings when available.

Resource scope

Preconditions

• Review scope and available evidence are identifiable.
• Tools can run or their absence is documented.

Effects and side effects

• Produces review findings and recommendations.
• May run read-heavy or diagnostic commands.
• Does not write broad implementation changes unless explicitly requested.

Guardrails

1. Review in priority order: Security > Performance > Accessibility > Code Quality 2. Every finding must include file:line, description, and fix 3. Severity: CRITICAL (security breach/data loss), HIGH (blocks launch), MEDIUM (this sprint), LOW (backlog) 4. Run automated tools first: npm audit, bandit, lighthouse 5. No false positives - every finding must be reproducible 6. Provide remediation code, not just descriptions 7. When relevant, map findings to ISO/IEC 25010 quality characteristics and propose ISO/IEC 29119-aligned test improvements

References

Follow resources/execution-protocol.md step by step. See resources/examples.md for input/output examples. Use resources/iso-quality.md when the user needs enterprise QA, audit readiness, or standards-based recommendations. Before submitting, run resources/self-check.md. Vendor-specific execution protocols are injected automatically by oma agent:spawn. Source files live under ../_shared/runtime/execution-protocols/{vendor}.md.

• Execution steps: resources/execution-protocol.md
• Report examples: resources/examples.md
• ISO quality guide: resources/iso-quality.md
• QA checklist: resources/checklist.md
• Self-check: resources/self-check.md
• Error recovery: resources/error-playbook.md
• Ultrawork VERIFY/SHIP phase protocol: resources/verify-ship-protocol.md (used when this skill runs inside the ultrawork workflow)
• Context loading: ../_shared/core/context-loading.md
• Context budget: ../_shared/core/context-budget.md
• Lessons learned: ../_shared/core/lessons-learned.md
• Observability handoff: ../oma-observability/SKILL.md §Integrations — canary RUM (Core Web Vitals), backend perf spans

QA Review Checklist

Security Checklist

Authentication & Authorization

• [ ] Passwords hashed with bcrypt/argon2 (not MD5/SHA1)
• [ ] Password strength requirements enforced (min 8 chars)
• [ ] JWT tokens properly signed and validated
• [ ] Refresh tokens implemented (if long sessions needed)
• [ ] Token expiry reasonable (15min access, 7day refresh)
• [ ] Authorization checks on ALL endpoints
• [ ] Users can only access their own data
• [ ] Admin functions require admin role
• [ ] Rate limiting on auth endpoints (5-10 attempts/min)
• [ ] Account lockout after failed login attempts (optional)
• [ ] MFA available (optional, but recommended)

Input Validation & Injection

• [ ] SQL injection: ORM used OR parameterized queries
• [ ] XSS: Input sanitized, CSP headers set
• [ ] Command injection: No shell execution with user input
• [ ] Path traversal: File paths validated
• [ ] LDAP injection: LDAP queries parameterized
• [ ] XML injection: XML parsing secure
• [ ] Email validation (proper regex/library)
• [ ] URL validation (allowlist for external requests)

Data Protection

• [ ] HTTPS enforced (redirect HTTP to HTTPS)
• [ ] Sensitive data NOT in logs
• [ ] Sensitive data NOT in error messages
• [ ] Sensitive data NOT in URLs (use POST body)
• [ ] Database backups encrypted
• [ ] PII data encrypted at rest (if applicable)
• [ ] Secure session management (httpOnly, secure, sameSite cookies)

API Security

• [ ] CORS properly configured (not * in production)
• [ ] CSRF protection enabled
• [ ] Rate limiting on API endpoints
• [ ] API keys/tokens NOT in source code
• [ ] API versioning implemented
• [ ] Proper error handling (no stack traces exposed)

Dependencies

• [ ] No high/critical vulnerabilities (npm audit / safety check)
• [ ] Dependencies up-to-date
• [ ] No unused dependencies
• [ ] License compliance checked

---

Performance Checklist

Backend Performance

• [ ] API response time < 200ms (p95)
• [ ] Database queries optimized (no N+1)
• [ ] Database indexes on foreign keys and frequent queries
• [ ] Connection pooling configured
• [ ] Caching implemented (Redis for frequent queries)
• [ ] Pagination for large result sets
• [ ] Async operations where appropriate
• [ ] Background jobs for heavy tasks

Frontend Performance

• [ ] Lighthouse Performance score > 90
• [ ] First Contentful Paint (FCP) < 1.5s
• [ ] Largest Contentful Paint (LCP) < 2.5s
• [ ] Cumulative Layout Shift (CLS) < 0.1
• [ ] Time to Interactive (TTI) < 3.5s
• [ ] Bundle size < 500KB (main bundle)
• [ ] Code splitting implemented
• [ ] Lazy loading for non-critical components
• [ ] Images optimized (WebP, compression)
• [ ] Images lazy loaded (loading="lazy")
• [ ] Fonts optimized (font-display: swap)
• [ ] No render-blocking resources
• [ ] Service worker for caching (optional)

Mobile Performance

• [ ] App size < 30MB (Android), < 50MB (iOS)
• [ ] Cold start < 2s
• [ ] Smooth scrolling (60fps)
• [ ] No memory leaks
• [ ] Battery usage minimal
• [ ] Offline support (if required)

---

Accessibility Checklist (WCAG 2.2 AA)

Perceivable

• [ ] All images have alt text
• [ ] Decorative images have empty alt (alt="")
• [ ] Color contrast 4.5:1 (normal text), 3:1 (large text)
• [ ] Text resizable up to 200% without loss of content
• [ ] Content understandable without color alone
• [ ] Audio/video has captions (if applicable)

Operable

• [ ] All functionality available via keyboard
• [ ] No keyboard trap
• [ ] Focus order is logical
• [ ] Focus indicators visible
• [ ] Skip to main content link
• [ ] No content flashes more than 3 times per second
• [ ] Enough time to read/interact with content
• [ ] Pause/stop for moving content

Understandable

• [ ] Page language set (<html lang="en">)
• [ ] Clear labels on form inputs
• [ ] Error messages clear and helpful
• [ ] Required fields indicated
• [ ] Consistent navigation across pages
• [ ] Predictable behavior (no unexpected popups)

Robust

• [ ] Valid HTML (semantic tags)
• [ ] ARIA labels where needed
• [ ] ARIA roles appropriate
• [ ] Works with screen readers (test with NVDA/JAWS)
• [ ] Works in different browsers (Chrome, Firefox, Safari, Edge)

---

Testing Checklist

Unit Tests

• [ ] Test coverage > 80%
• [ ] All business logic functions tested
• [ ] Edge cases covered
• [ ] Error handling tested
• [ ] Mocks used appropriately
• [ ] Tests run fast (< 10s total)
• [ ] No flaky tests

Integration Tests

• [ ] All API endpoints tested
• [ ] Database operations tested
• [ ] Auth flow tested
• [ ] Error responses tested (401, 403, 404, 500)
• [ ] Request validation tested

E2E Tests

• [ ] Critical user flows tested (registration, login, main feature)
• [ ] Happy path tested
• [ ] Error scenarios tested
• [ ] Mobile responsive tested
• [ ] Cross-browser tested (Chrome, Firefox, Safari)

Test Governance

• [ ] Test levels and scope are defined clearly
• [ ] Important requirements trace to test cases or test scenarios
• [ ] Entry / exit criteria are defined for major release decisions
• [ ] Test design technique is appropriate for risk and feature type

Performance Tests

• [ ] Load testing (1000 concurrent users)
• [ ] Stress testing (identify breaking point)
• [ ] Database under load tested
• [ ] API rate limits tested

---

Code Quality Checklist

Architecture

• [ ] Clear separation of concerns
• [ ] DRY principle followed (no duplication > 5%)
• [ ] SOLID principles followed
• [ ] Dependency injection used
• [ ] Repository pattern (backend)
• [ ] Component composition (frontend)

Code Metrics

• [ ] Cyclomatic complexity < 10 per function
• [ ] Function length < 50 lines
• [ ] File length < 500 lines
• [ ] No deeply nested code (< 4 levels)
• [ ] Meaningful variable names

Error Handling

• [ ] All async operations have try/catch
• [ ] Errors logged appropriately
• [ ] User-friendly error messages
• [ ] No silent failures
• [ ] Graceful degradation

Documentation

• [ ] README with setup instructions
• [ ] API documentation (OpenAPI/Swagger)
• [ ] Complex logic documented
• [ ] Environment variables documented
• [ ] No TODO/FIXME in production code

---

ISO Quality Alignment

ISO/IEC 25010

• [ ] Functional suitability considered
• [ ] Performance efficiency considered
• [ ] Compatibility considered where integration matters
• [ ] Usability / accessibility considered
• [ ] Reliability considered
• [ ] Security considered
• [ ] Maintainability considered
• [ ] Portability considered when relevant

ISO/IEC 29119

• [ ] Test strategy or test plan exists for significant changes
• [ ] Test basis, test conditions, and expected results are clear
• [ ] Risk-based prioritization is visible in test scope
• [ ] Test evidence and traceability are sufficient for review or audit

---

Browser Compatibility Checklist

Desktop

• [ ] Chrome (latest 2 versions)
• [ ] Firefox (latest 2 versions)
• [ ] Safari (latest 2 versions)
• [ ] Edge (latest 2 versions)

Mobile

• [ ] iOS Safari (latest 2 versions)
• [ ] Android Chrome (latest 2 versions)
• [ ] Responsive breakpoints (320px, 768px, 1024px, 1440px)

---

DevOps Checklist

Environment

• [ ] Environment variables used (not hardcoded)
• [ ] .env.example provided
• [ ] Secrets NOT in source code
• [ ] Different configs for dev/staging/prod

Logging

• [ ] Appropriate log levels (DEBUG, INFO, WARNING, ERROR)
• [ ] No sensitive data in logs
• [ ] Structured logging (JSON format)
• [ ] Log rotation configured

Monitoring

• [ ] Health check endpoint (/health)
• [ ] Error tracking (Sentry, Rollbar, etc.)
• [ ] Performance monitoring (APM)
• [ ] Uptime monitoring

Deployment

• [ ] CI/CD pipeline configured
• [ ] Automated tests in CI
• [ ] Database migrations automated
• [ ] Rollback plan documented
• [ ] Zero-downtime deployment (if required)

---

Final Sign-Off

Critical (Must Pass)

• [ ] No CRITICAL security vulnerabilities
• [ ] No HIGH security vulnerabilities
• [ ] All E2E tests passing
• [ ] Performance meets requirements
• [ ] No data loss scenarios

Important (Should Pass)

• [ ] Test coverage > 80%
• [ ] Accessibility WCAG 2.2 AA
• [ ] Code quality metrics met
• [ ] Documentation complete

Nice-to-Have (Can Address Later)

• [ ] Code refactoring opportunities documented
• [ ] Performance optimization ideas documented
• [ ] Future enhancement ideas documented

---

Issue Prioritization

CRITICAL (Block Deployment)

• Security vulnerabilities (SQL injection, XSS, auth bypass)
• Data loss bugs
• Application crashes
• Complete feature breakage

HIGH (Fix Before Launch)

• Performance issues (> 5s load time)
• Major accessibility violations
• Missing auth checks
• Broken core functionality

MEDIUM (Fix in Sprint)

• Minor bugs
• Code quality issues
• Missing tests
• Minor accessibility issues

LOW (Backlog)

• Refactoring opportunities
• Performance optimizations
• Nice-to-have features
• Documentation improvements

---

Notes

• Run automated tools FIRST: npm audit, bandit, lighthouse
• Use Serena MCP for code analysis patterns
• Use Chrome DevTools MCP (new_page with isolatedContext: "qa-test") for runtime verification and E2E testing
• Document all findings with file:line references
• Provide remediation code examples
• Estimate fix time for each issue

---

Runtime Verification (after static review)

Record results in the structured table format defined in execution-protocol.md Step 2.5 (Recording Results).

• [ ] Application starts without errors
• [ ] All modified endpoints return expected status codes; verify with list_network_requests()
• [ ] Form submissions produce correct database state; verify with fill_form() + list_network_requests()
• [ ] Error states render user-friendly messages (not stack traces); verify with take_snapshot() on error paths
• [ ] Empty/loading/error UI states all handled; verify with navigate_page() to empty state routes + take_snapshot()
• [ ] Interactive elements respond to input (not display-only); verify with click(uid) + take_snapshot() before/after
• [ ] Auth flows work end-to-end (register → login → protected route → logout); verify with sequential fill() + click() + list_network_requests()
• [ ] Rate limiting / throttling triggers at configured thresholds; verify with rapid evaluate_script(fetch) calls
• [ ] File upload/download actually transfers data (not stubbed); verify with upload_file(uid, filePath) + response check
• [ ] Pagination returns correct pages (not always page 1); verify with click() page 2 + take_snapshot() to confirm different content
• [ ] Zero JS console errors on critical paths; verify with list_console_messages(types: ["error"])

QA Agent - Error Recovery Playbook

When you encounter a failure during review, follow these recovery steps. Do NOT stop or ask for help until you have exhausted the playbook.

---

Automated Tool Fails to Run

Symptoms: npm audit, bandit, lighthouse command errors

1. Check: is the tool installed? Note missing tool in result 2. Check: are you in the correct directory? 3. If npm audit: try npm audit --production to skip devDependencies 4. If bandit: check Python path; may need python -m bandit 5. If lighthouse: requires a running server; note if server not available 6. If tool unavailable: Fall back to manual review, record tool_unavailable: ["tool_name"] in result

---

False Positive Suspected

Symptoms: Finding looks like a vulnerability but might be safe

1. Trace the data flow: does user input actually reach the dangerous operation? 2. Check: is there validation/sanitization upstream? 3. Check: is the framework handling this automatically? (e.g., ORM prevents SQL injection) 4. If uncertain: mark severity as MEDIUM with note "verify manually" 5. NEVER do this: Mark as CRITICAL without certainty (false alarms erode trust)

---

Cannot Access Source Code

Symptoms: Serena find_symbol returns nothing, file not found

1. Check: correct file path? Use search_for_pattern with broader terms 2. Check: is the code in a different directory or monorepo? 3. Use get_symbols_overview on parent directories to find the structure 4. If truly inaccessible: review what you CAN access and note gaps in report

---

Performance Metrics Unavailable

Symptoms: Can't run Lighthouse, no APM data, no load test results

1. Check if dev server is running for Lighthouse 2. If no server: review code statically for performance anti-patterns:

• N+1 queries (loops with DB calls)
• Missing pagination
• Large bundle imports
• No code splitting

3. Report findings with static_analysis_only: true flag 4. Recommend specific metrics to measure when environment is available

---

Scope Too Large

Symptoms: Full audit requested but codebase has 100+ files

1. Prioritize: auth/security-critical files first 2. Use pattern search to find high-risk areas:

• search_for_pattern("password|secret|token|api_key")
• search_for_pattern("execute|eval|innerHTML")

3. Review critical paths: auth flow, payment, data mutation 4. Note in report: scope_coverage: "critical paths only, full audit requires more"

---

Rate Limit / Quota Error

Symptoms: 429, RESOURCE_EXHAUSTED, rate limit exceeded

1. Stop immediately: do not make additional API calls 2. Save current work to progress-{agent-id}[-{sessionId}].md 3. Record Status: quota_exceeded in result-{agent-id}[-{sessionId}].md 4. Specify remaining tasks

---

Serena Memory Unavailable

1. Retry once 2. If 2 consecutive failures: use local file /tmp/progress-{agent-id}[-{sessionId}].md 3. Add memory_fallback: true flag to result

---

General Principles

• False positive prevention: If finding is uncertain, lower severity and mark "verify manually"
• Blocked: If no progress after 5 turns, save current state, Status: blocked
• No code modification: QA only reports; delegate code changes to the appropriate agent

QA Agent - Examples

Example 1: Security Audit

Input: "Review the authentication system for security issues"

Output:

# QA Report: Authentication System
Status: FAIL

## Security Audit

### CRITICAL
1. **SQL Injection** - backend/api/users.py:45
   - Raw string interpolation in query
   - Fix: Use parameterized query or ORM

2. **Missing Rate Limiting** - backend/api/auth.py:23
   - Login endpoint has no rate limiting
   - Fix: Add @limiter.limit("5/minute")

### Passed
- [x] Passwords hashed with bcrypt (cost 12)
- [x] JWT properly signed with RS256
- [x] Refresh token rotation implemented
- [x] CORS configured (not wildcard)

## Recommendations
### High Priority (Before Launch)
1. Fix SQL injection in users.py:45
2. Add rate limiting to auth endpoints
3. Add account lockout after 10 failed attempts

Example 2: Performance Review

Input: "Check frontend performance - users report slow load times"

Output:

# QA Report: Frontend Performance
Status: WARNING

## Performance
- Lighthouse Score: 67/100 [FAIL - Target > 90]
- FCP: 2.8s [FAIL - Target < 1.5s]
- LCP: 4.1s [FAIL - Target < 2.5s]
- Bundle: 1.2MB [FAIL - Target < 500KB]
- CLS: 0.05 [PASS]

## Root Causes
1. Full lodash import (72KB) - src/utils/helpers.ts:1
2. No code splitting on routes - src/app/layout.tsx
3. Unoptimized images (3 x 2MB PNGs) - public/images/

## Recommendations
### High Priority
1. Tree-shake lodash: `import uniq from 'lodash/uniq'`
2. Add dynamic imports: `const Page = dynamic(() => import(...))`
3. Convert images to WebP, add width/height, loading="lazy"

### Expected After Fix
- Bundle: ~350KB (-70%)
- FCP: ~1.2s
- Lighthouse: ~92

Example 3: Full Pre-Launch Audit

Input: "Complete QA review before production deployment"

Output:

# QA Report: Pre-Launch Audit
Status: WARNING

## Summary
- Security: 0 CRITICAL, 1 HIGH, 2 MEDIUM
- Performance: PASS (all targets met)
- Accessibility: 1 HIGH, 3 MEDIUM
- Test Coverage: 83% [PASS]

## High Priority (Block Launch)
1. [SECURITY-HIGH] Missing CSRF protection on POST endpoints
2. [A11Y-HIGH] Login form has no aria-labels

## Medium Priority (Fix This Sprint)
1. [SECURITY-MED] API error responses expose stack traces
2. [SECURITY-MED] No Content-Security-Policy header
3. [A11Y-MED] Color contrast 3.8:1 on secondary text (need 4.5:1)
4. [A11Y-MED] Missing skip-to-content link
5. [A11Y-MED] Focus indicator invisible on dark theme

## Sign-Off
- [x] No CRITICAL vulnerabilities
- [ ] No HIGH vulnerabilities (2 remaining)
- [x] Performance targets met
- [x] Test coverage > 80%

Example 4: Standards-Aligned QA Review

Input: "Review this release candidate and suggest ISO 25010 / 29119 improvements"

Output:

# QA Report: Release Candidate
Status: WARNING

## Standards Alignment

### ISO/IEC 25010
- Reliability gap: no chaos/failure-path validation for retry flows
- Maintainability gap: business rules duplicated in 3 services
- Security gap: privileged actions are not audit-logged consistently

### ISO/IEC 29119
- Missing requirement-to-test traceability for payment cancellation
- No explicit exit criteria for regression approval
- Edge-case test design is weak for invalid state transitions

## Recommendations
1. Add requirement-to-test mapping for payment, refund, and auth flows
2. Define release exit criteria: zero HIGH findings, all critical E2E paths green
3. Add state-transition and boundary-value tests for payment lifecycle

QA Agent - Execution Protocol

Step 0: Prepare

1. Assess difficulty: see ../../_shared/core/difficulty-guide.md

• Simple: Quick security + quality check | Medium: Full 4 steps | Complex: Full + prioritized scope

2. Check lessons: read QA section in ../../_shared/core/lessons-learned.md 3. Clarify requirements: follow ../../_shared/core/clarification-protocol.md

• Check Uncertainty Triggers: security/auth concerns, existing code conflict potential?
• Determine level: LOW → proceed | MEDIUM → present options | HIGH → ask immediately

4. Budget context: follow ../../_shared/core/context-budget.md (prioritize high-risk files) 5. After review: add recurring issues to ../../_shared/core/lessons-learned.md 6. If standards matter: read resources/iso-quality.md

Intelligent Escalation: When uncertain, escalate early. Don't blindly proceed.

Follow these steps in order (adjust depth by difficulty).

Step 1: Scope

• Identify what to review: new feature, full audit, or specific concern
• List all files/modules to inspect
• Determine review depth: quick check vs. comprehensive audit
• Use Serena to map the codebase:
• get_symbols_overview("src/"): Understand structure
• search_for_pattern("password.*=.*[\"']"): Find hardcoded secrets
• search_for_pattern("execute.*\\$\\{"): Find SQL injection
• search_for_pattern("innerHTML"): Find XSS vulnerabilities

Step 2: Audit

Review in this priority order: 1. Security (CRITICAL): OWASP Top 10, auth, injection, data protection 2. Performance: API latency, N+1 queries, bundle size, Core Web Vitals 3. Accessibility: WCAG 2.2 AA, keyboard nav, screen reader, contrast 4. Code Quality: test coverage, complexity, architecture adherence

When applicable, map findings and gaps to:

• ISO/IEC 25010 quality characteristics
• ISO/IEC 29119 test planning, design, traceability, and exit criteria

Use resources/checklist.md (renamed qa-checklist) as the comprehensive review guide.

Step 2.5: Runtime Verification

Static code review misses entire categories of bugs: display-only features, stubbed functionality, broken user flows, and edge cases that only surface at runtime. This step requires interacting with the running application.

When to Execute

• Web app with UI: ALWAYS for Medium/Complex tasks
• API-only: ALWAYS (curl/httpie verification)
• Simple tasks (single file, no UI): SKIP

Execution by App Type

Web Applications (Chrome DevTools MCP: Isolated Mode)

1. Start the application (bun run dev, uv run manage.py runserver, etc.) 2. Open the app in an isolated browser context to avoid contaminating the user's session:

   new_page(url: "http://localhost:PORT", isolatedContext: "qa-test")

• Pages in the same isolatedContext share cookies/storage
• Pages in different contexts are fully isolated
• Always use isolatedContext: "qa-test" for QA verification

3. Navigate and inspect:

   navigate_page(url)            → navigate within the isolated context (SPA routes, sub-pages)
   take_snapshot()               → a11y tree snapshot with uid-tagged elements (prefer over screenshots)

4. Interact with modified features:

   click(uid)                    → click buttons/links
   fill(uid, value)              → fill input/textarea/select fields
   fill_form(elements)           → fill multiple fields at once
   type_text(text, submitKey)    → type into focused input
   press_key(key)                → keyboard shortcuts, Enter, Tab
   hover(uid)                    → hover for tooltips, dropdown menus, state changes
   drag(from_uid, to_uid)        → drag-and-drop

5. Verify functional correctness:

• list_console_messages(types: ["error", "warn"]) → detect JS errors
• list_network_requests() → verify API calls fired and returned expected status
• get_network_request(reqid) → inspect request/response bodies
• evaluate_script(function) → check DOM state, global variables, computed styles
• wait_for(text) → confirm expected content appears after interaction

6. Visual and accessibility verification:

• take_screenshot() → visual output matches acceptance criteria
• lighthouse_audit(mode: "snapshot") → accessibility, SEO, best practices scores
• emulate(viewport: "375x812x3,mobile,touch") → responsive/mobile verification

7. Performance profiling (when performance criteria exist):

• performance_start_trace(reload: true) → capture Core Web Vitals
• performance_stop_trace() → analyze LCP, INP, CLS
• take_memory_snapshot(filePath) → detect memory leaks

8. Cleanup:

   close_page(pageId)  → close isolated test pages after verification

9. Fallback (no Chrome DevTools MCP available):

• Use curl/httpie to hit rendered endpoints
• Verify HTTP status codes and response bodies
• Check redirects, auth flows, and error pages

API Endpoints

1. Start the server 2. Execute acceptance criteria as actual HTTP requests:

   # Example: verify auth flow end-to-end
   curl -s -X POST localhost:8000/api/auth/register \
     -H "Content-Type: application/json" \
     -d '{"email":"test@test.com","password":"secure123"}'

   # Verify rate limiting actually triggers
   for i in $(seq 1 20); do
     curl -s -o /dev/null -w "%{http_code}" localhost:8000/api/auth/login
   done

3. Verify database state after operations (query directly or via API)

Mobile Applications

1. If emulator/simulator available: launch and interact 2. If not: verify via API layer + widget test execution

Recording Results

Append to the QA report under a new section:

## Runtime Verification Results

| Feature | Method | Expected | Actual | Status |
|---------|--------|----------|--------|--------|
| User registration | curl POST /api/auth/register | 201 + user created | 201 + user in DB | PASS |
| Rate limiting | 20x rapid POST /api/auth/login | 429 after threshold | 429 after 10 req | PASS |
| Empty state UI | navigate_page + take_snapshot | Empty state message | Blank white page | FAIL |
| Button handler | click(uid) + list_network_requests | POST /api/save fired | No request fired | FAIL |
| Console errors | list_console_messages(["error"]) | 0 errors | TypeError at app.js:42 | FAIL |

Stubbed Feature Detection

Specifically check for these patterns that static review cannot catch:

• Buttons/forms that render but have no backend handler → click(uid) + list_network_requests() to verify
• Features that display placeholder data instead of real data → evaluate_script() to check data binding
• Interactive elements that don't respond to user input → click(uid) + take_snapshot() to compare before/after
• Audio/video/file upload controls with no actual processing → upload_file(uid, filePath) + verify response

---

Evaluator Posture: SKEPTICAL by default

Apply this posture when making verdict decisions in Step 3 and Step 4:

• Assume code has bugs until you prove otherwise with concrete evidence.
• "Probably works" or "should be fine" is NOT valid evidence. Run it, show output, prove it.
• If a feature exists in the plan but you cannot verify it works at runtime: FAIL.
• Never downgrade a real bug to "non-critical" just to pass a gate.
• Use WARNING when all remaining issues are MEDIUM or lower and none block deployment.
• When in doubt between PASS and FAIL, choose FAIL and explain why.

Step 3: Report

Generate structured report with:

• Overall status:
• PASS: no CRITICAL, no HIGH, and no MEDIUM issues
• WARNING: no CRITICAL and no HIGH, but MEDIUM issues exist
• FAIL: any CRITICAL or HIGH issue found
• Findings grouped by severity (CRITICAL > HIGH > MEDIUM > LOW)
• Each finding: file:line, description, remediation code
• Performance metrics vs. targets
• Standards suggestions when relevant:
• quality characteristics under-covered
• missing test design / traceability / exit criteria

Step 4: Verify

• Run resources/self-check.md to verify your own review quality
• Ensure no false positives (each finding is real and reproducible)
• Confirm remediation suggestions are correct and complete
• Run ../../_shared/core/common-checklist.md for general quality

On Error

See resources/error-playbook.md for recovery steps.

QA Agent - ISO Quality Guide

Use this file when the user asks for standards-based QA suggestions, enterprise audit readiness, or process-oriented quality improvements.

Positioning

• Use ISO/IEC 25010 as the practical software quality model.
• Treat ISO/IEC 25010 as part of the ISO/IEC 25000 SQuaRE family when users say "ISO 25000".
• Use ISO/IEC 29119 for test process, test design, traceability, and evidence recommendations.
• Do not force standards language into every QA review. Add it when it improves decision quality, auditability, or delivery discipline.

1. ISO/IEC 25010 Mapping

Map findings to these quality characteristics when relevant:

• functional suitability
• performance efficiency
• compatibility
• usability
• reliability
• security
• maintainability
• portability

Use this mapping to answer:

• Which quality areas are under-covered?
• Which areas are blocking release confidence?
• Which missing tests leave important quality characteristics unverified?

2. ISO/IEC 29119 Suggestions

Recommend these when test discipline is weak:

• test strategy or test plan
• test levels and scope definition
• requirement / risk to test traceability
• explicit test conditions and expected results
• risk-based prioritization
• entry / exit criteria
• evidence retention for review or audit

3. When to Mention Standards

Mention ISO alignment when:

• the user asks for audit readiness
• the work is enterprise or regulated
• release governance is unclear
• testing exists but lacks structure or traceability
• quality discussion is too vague and needs a common model

Avoid overusing standards language when:

• the task is a quick bug fix
• the user wants only concrete defects
• standards language would add ceremony without action

4. Review Output Pattern

If standards are relevant, add a short section:

## Standards Alignment

### ISO/IEC 25010
- Quality characteristic:
- Gap:
- Recommendation:

### ISO/IEC 29119
- Test governance gap:
- Traceability / evidence gap:
- Recommendation:

5. Guardrails

• Prefer practical suggestions over compliance theater
• Do not claim formal compliance unless evidence exists
• Tie every standards suggestion to a delivery, test, or risk outcome
• Keep standards comments secondary to concrete findings

QA Agent - Self-Check

Verify your own review quality before submitting the QA report.

Completeness

• [ ] All files in scope were reviewed (none skipped)
• [ ] Security section covers OWASP Top 10 categories
• [ ] Performance metrics include actual numbers vs. targets
• [ ] Accessibility check covers Perceivable, Operable, Understandable, Robust
• [ ] ISO/IEC 25010 and ISO/IEC 29119 suggestions were included when relevant

Accuracy

• [ ] Every finding includes file path and line number
• [ ] Every finding is reproducible (not speculative)
• [ ] No false positives (double-check edge cases)
• [ ] Severity ratings are consistent (CRITICAL = data loss/security breach)

Actionability

• [ ] Every finding has a specific remediation step
• [ ] Remediation code examples are correct and complete
• [ ] Priorities are clearly ordered (what to fix first)
• [ ] Estimated impact is noted for performance issues
• [ ] Standards-based suggestions are practical, not ceremonial

Report Quality

• [ ] Overall status (PASS/WARNING/FAIL) matches findings
• [ ] Report is scannable (headers, bullets, status tags)
• [ ] No duplicate findings
• [ ] Sign-off checklist included for launch decisions

VERIFY/SHIP Phase Protocol

Execution guide for VERIFY Phase (Steps 6-8) and SHIP Phase (Steps 14-17) in ultrawork workflow.

---

VERIFY Phase (Steps 6-8)

Step 6: Alignment Review

Review Question: "Did we build what was requested?"

• [ ] Plan vs implementation 1:1 comparison
• [ ] Misalignments documented
• [ ] No missing features

---

Step 7: Security/Bug Review

Review Question: "Is there anything dangerous?"

Automated Tools:

• npm audit (Node.js)
• bandit (Python)
• lighthouse (web performance)

OWASP Top 10 Check:

• [ ] No injection vulnerabilities
• [ ] No authentication flaws
• [ ] No sensitive data exposure
• [ ] No XXE vulnerabilities
• [ ] No access control flaws

Finding Format:

• File:line
• Description
• Severity (CRITICAL/HIGH/MEDIUM/LOW)
• Fix suggestion

---

Step 8: Improvement Review

Review Question: "Did improvements break anything?"

• [ ] All existing tests pass
• [ ] Existing features work normally
• [ ] New code matches existing patterns

---

VERIFY_GATE Checklist

• [ ] Implementation = Requirements
• [ ] CRITICAL issues: 0
• [ ] HIGH issues: 0
• [ ] No regressions

Gate failure → Return to Step 5 (fix implementation)

---

SHIP Phase (Steps 14-17)

Step 14: Code Quality Review

Review Question: "Does it meet quality standards?"

• [ ] lint passes
• [ ] type check passes
• [ ] test coverage >= 80%
• [ ] _shared/core/common-checklist.md passes

---

Step 15: UX Flow Verification

• [ ] End-to-end user journey test
• [ ] Error states verified
• [ ] Loading states verified
• [ ] Accessibility compliance (WCAG 2.2 AA)

---

Step 16: Related Issues Review

Review Question: "Did we break anything elsewhere?"

• [ ] Issues discovered during review checked
• [ ] Related areas not broken
• [ ] Deferred items documented (next sprint)

---

Step 17: Deployment Readiness Review

Review Question: "Is this ready to deploy?"

• [ ] No hardcoded secrets
• [ ] Environment variables documented
• [ ] Migrations safe
• [ ] Rollback possible
• [ ] Monitoring ready

---

SHIP_GATE Checklist

• [ ] All quality checks pass
• [ ] UX verified
• [ ] Related issues resolved
• [ ] Deployment checklist complete
• [ ] User final approval

Gate failure → Return to appropriate phase based on failure type