Test Automation Strategy

Name: Test Automation Strategy
Author: proffesor-for-testing

proffesor-for-testing/agentic-qe

226 installs
433 repo stars
Updated August 4, 2026
proffesor-for-testing/agentic-qe

Define automation scope, pyramid layers, tooling, and n8n coverage priorities for agentic-qe before committing to large flaky suites.

About

test-automation-strategy from proffesor-for-testing/agentic-qe guides early scoping of automation—pyramid design, tooling selection, n8n coverage priorities, and CI constraints—before building large test estates.

Test pyramid and risk-based scope
Tooling choices for agentic QE
n8n coverage prioritization
CI cost and flake planning
Feeds pr-review and fundamentals

Test Automation Strategy by the numbers

226 all-time installs (skills.sh)
+6 installs in the week ending Aug 4, 2026 (Skillselion tracking)
Ranked #774 of 2,153 Testing & QA skills by installs in the Skillselion catalog
Data as of Aug 5, 2026 (Skillselion catalog sync)

npx skills add https://github.com/proffesor-for-testing/agentic-qe --skill test-automation-strategy

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/proffesor-for-testing/agentic-qe/test-automation-strategy.svg)](https://skillselion.com/skills/proffesor-for-testing/agentic-qe/test-automation-strategy)

Installs	226
repo stars	★ 433
Last updated	August 4, 2026
Repository	proffesor-for-testing/agentic-qe ↗

What it does

Define automation scope, pyramid layers, tooling, and n8n coverage priorities for agentic-qe before committing to large flaky suites.

Files

SKILL.mdMarkdownGitHub ↗

Test Automation Strategy

<default_to_action> When designing or improving test automation: 1. DETECT anti-patterns: Ice cream cone? Slow suite? Flaky tests? 2. USE patterns: Page Object Model, Builder pattern, Factory pattern 3. INTEGRATE in CI/CD: Every commit runs tests, fail fast 4. MANAGE flaky tests: Quarantine, fix, or delete - never ignore

Quick Anti-Pattern Detection:

Ice cream cone (many E2E, few unit) → Invert to pyramid
Slow tests (> 10 min suite) → Parallelize, mock external deps
Flaky tests → Fix timing, isolate data, or quarantine
Brittle selectors → Use data-testid, semantic locators

</default_to_action>

Quick Reference Card

When to Use

Building new automation framework
Improving existing test efficiency
Reducing flaky test burden
Optimizing CI/CD pipeline speed

Anti-Patterns to Detect

Problem	Symptom	Fix
Ice cream cone	80% E2E, 10% unit	Invert pyramid
Slow suite	30+ min CI	Parallelize, prune
Flaky tests	Random failures	Quarantine, fix timing
Coupled tests	Order-dependent	Isolate data
Brittle selectors	Break on CSS change	Use data-testid

---

CI/CD Integration

name: Test Pipeline
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run test:unit -- --coverage
        timeout-minutes: 5
      - uses: codecov/codecov-action@v3

  integration-tests:
    needs: unit-tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
    steps:
      - run: npm run test:integration
        timeout-minutes: 10

  e2e-tests:
    needs: integration-tests
    runs-on: ubuntu-latest
    steps:
      - run: npx playwright test
        timeout-minutes: 15

---

Flaky Test Management

// Quarantine flaky tests
describe.skip('Quarantined - INC-123', () => {
  test('flaky test awaiting fix', () => { /* ... */ });
});

// Agent-assisted stabilization
await Task("Fix Flaky Tests", {
  tests: quarantinedTests,
  analysis: ['timing-issues', 'data-isolation', 'race-conditions'],
  strategies: ['add-waits', 'isolate-fixtures', 'mock-externals']
}, "qe-flaky-test-hunter");

---

Agent-Assisted Automation

// Generate tests following pyramid
await Task("Generate Test Suite", {
  sourceCode: 'src/',
  pyramid: { unit: 70, integration: 20, e2e: 10 },
  patterns: ['page-object', 'builder', 'factory'],
  framework: 'jest'
}, "qe-test-generator");

// Optimize test execution
await Task("Optimize Suite", {
  algorithm: 'johnson-lindenstrauss',
  targetReduction: 0.3,
  maintainCoverage: 0.95
}, "qe-regression-risk-analyzer");

// Analyze flaky patterns
await Task("Flaky Analysis", {
  testHistory: 'last-30-days',
  detectPatterns: ['timing', 'data', 'environment'],
  recommend: 'stabilization-strategy'
}, "qe-flaky-test-hunter");

---

Agent Coordination Hints

Memory Namespace

aqe/automation/
├── test-pyramid/*        - Coverage by layer
├── page-objects/*        - Shared page objects
├── flaky-registry/*      - Quarantined tests
└── execution-metrics/*   - Suite performance data

Fleet Coordination

const automationFleet = await FleetManager.coordinate({
  strategy: 'test-automation',
  agents: [
    'qe-test-generator',         // Generate pyramid-compliant tests
    'qe-test-executor',          // Parallel execution
    'qe-coverage-analyzer',      // Coverage gaps
    'qe-flaky-test-hunter',      // Flaky detection
    'qe-regression-risk-analyzer' // Smart selection
  ],
  topology: 'hierarchical'
});

---

Related Skills

tdd-london-chicago - TDD for unit tests
api-testing-patterns - Integration patterns
cicd-pipeline-qe-orchestrator - Pipeline integration
shift-left-testing - Early automation

---

Remember

With Agents: Agents generate pyramid-compliant tests, detect flaky patterns, optimize execution time, and maintain test infrastructure. Use agents to scale automation quality.

Gotchas

Agent generates 80% E2E tests and 20% unit tests (inverted pyramid) — explicitly enforce 70/20/10 ratio
Page Object Model tests become brittle when selectors change — prefer data-testid attributes over CSS selectors
Flaky tests quarantined but never fixed is technical debt — set a 2-week SLA to fix or delete
Agent treats test code as second-class — test code needs the same review standards as production code
Parallel test execution requires test isolation — shared state between tests causes non-deterministic failures

skill: test-automation-strategy
version: 1.0.0
description: >
  Evaluation suite for test-automation-strategy skill.
  Tests generation of comprehensive test automation strategies with proper
  prioritization, risk assessment, and tool recommendations.

models_to_test:
  - claude-sonnet-4-6    # Primary (high accuracy expected)
  - claude-haiku-4-5     # Fast model (minimum quality floor)

mcp_integration:
  enabled: true
  namespace: skill-validation
  query_patterns: true
  track_outcomes: true
  store_patterns: true
  share_learning: true
  update_quality_gate: true
  target_agents:
    - qe-learning-coordinator

learning:
  store_success_patterns: true
  store_failure_patterns: true
  pattern_ttl_days: 90
  min_confidence_to_store: 0.7

result_format:
  json_output: true
  include_timing: true
  include_token_usage: true

setup:
  required_tools:
    - jq

test_cases:
  - id: tc001_basic_strategy_generation
    description: "Skill generates basic test automation strategy"
    category: basic
    priority: critical

    input:
      projectDescription: "E-commerce platform with React frontend and Node.js backend"
      context:
        projectSize: "medium"
        riskLevel: "high"

    expected_output:
      must_contain:
        - "strategy"
        - "automation"
        - "test"
      must_not_contain:
        - "error"
        - "unable"

    validation:
      schema_check: true
      keyword_match_threshold: 0.8

  - id: tc002_strategy_priorities
    description: "Strategy identifies test priorities correctly"
    category: core
    priority: critical

    input:
      projectDescription: "API service with critical payment processing"
      context:
        criticalComponents: ["payment", "auth"]

    expected_output:
      must_contain:
        - "critical"
        - "priority"

    validation:
      schema_check: true

  - id: tc003_tool_recommendations
    description: "Appropriate testing tools are recommended"
    category: core
    priority: high

    input:
      projectDescription: "Full-stack web application"
      context:
        frontendFramework: "React"
        backendFramework: "Node.js"

    expected_output:
      must_contain:
        - "tool"
        - "recommend"

    validation:
      schema_check: true

  - id: tc004_test_coverage_analysis
    description: "Skill analyzes required test coverage"
    category: core
    priority: high

    input:
      projectDescription: "Banking application"
      context:
        regulatoryRequirements: "high"

    expected_output:
      must_contain:
        - "coverage"

    validation:
      schema_check: true
      allow_partial: true

  - id: tc005_risk_based_strategy
    description: "Strategy incorporates risk-based testing approach"
    category: core
    priority: high

    input:
      projectDescription: "Healthcare data platform"
      context:
        dataClassification: "sensitive"
        riskLevel: "critical"

    expected_output:
      must_contain:
        - "risk"

    validation:
      schema_check: true
      allow_partial: true

success_criteria:
  pass_rate: 0.8
  critical_pass_rate: 1.0
  avg_reasoning_quality: 0.7
  max_execution_time_ms: 300000
  cross_model_variance: 0.15

metadata:
  author: "qe-learning-coordinator"
  created: "2026-02-02"
  last_updated: "2026-02-02"
  coverage_target: "Strategy generation, prioritization, tool recommendations, risk analysis"

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://agentic-qe.dev/schemas/test-automation-strategy-output.json",
  "title": "AQE Test Automation Strategy Skill Output Schema",
  "description": "Schema for test-automation-strategy skill output validation. Extends the base skill-output template with test pyramid compliance, F.I.R.S.T. principles, and CI/CD integration.",
  "type": "object",
  "required": ["skillName", "version", "timestamp", "status", "trustTier", "output"],
  "properties": {
    "skillName": {
      "type": "string",
      "const": "test-automation-strategy",
      "description": "Must be 'test-automation-strategy'"
    },
    "version": {
      "type": "string",
      "pattern": "^\\d+\\.\\d+\\.\\d+(-[a-zA-Z0-9]+)?$"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time"
    },
    "status": {
      "type": "string",
      "enum": ["success", "partial", "failed", "skipped"]
    },
    "trustTier": {
      "type": "integer",
      "const": 3
    },
    "output": {
      "type": "object",
      "required": ["summary", "pyramidAnalysis", "firstPrinciples", "findings", "recommendations"],
      "properties": {
        "summary": {
          "type": "string",
          "minLength": 50,
          "maxLength": 2000,
          "description": "Human-readable summary of automation strategy assessment"
        },
        "score": {
          "$ref": "#/$defs/automationScore"
        },
        "pyramidAnalysis": {
          "$ref": "#/$defs/pyramidAnalysis",
          "description": "Test pyramid compliance analysis"
        },
        "firstPrinciples": {
          "$ref": "#/$defs/firstPrinciples",
          "description": "F.I.R.S.T. principles assessment"
        },
        "patterns": {
          "$ref": "#/$defs/automationPatterns",
          "description": "Automation patterns analysis"
        },
        "antiPatterns": {
          "$ref": "#/$defs/antiPatterns",
          "description": "Detected anti-patterns"
        },
        "cicdIntegration": {
          "$ref": "#/$defs/cicdIntegration",
          "description": "CI/CD pipeline integration"
        },
        "flakyTests": {
          "$ref": "#/$defs/flakyTestAnalysis",
          "description": "Flaky test analysis"
        },
        "findings": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/automationFinding"
          },
          "maxItems": 100
        },
        "recommendations": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/automationRecommendation"
          },
          "maxItems": 50
        },
        "metrics": {
          "$ref": "#/$defs/automationMetrics"
        },
        "artifacts": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/artifact"
          },
          "maxItems": 50
        }
      }
    },
    "metadata": {
      "$ref": "#/$defs/metadata"
    },
    "validation": {
      "$ref": "#/$defs/validationResult"
    },
    "learning": {
      "$ref": "#/$defs/learningData"
    }
  },
  "$defs": {
    "automationScore": {
      "type": "object",
      "required": ["value", "max"],
      "properties": {
        "value": {
          "type": "number",
          "minimum": 0,
          "maximum": 100
        },
        "max": {
          "type": "number",
          "const": 100
        },
        "grade": {
          "type": "string",
          "pattern": "^[A-F][+-]?$"
        },
        "maturityLevel": {
          "type": "string",
          "enum": ["initial", "managed", "defined", "quantitatively-managed", "optimizing"],
          "description": "Automation maturity level"
        }
      }
    },
    "pyramidAnalysis": {
      "type": "object",
      "required": ["layers", "shape"],
      "properties": {
        "layers": {
          "type": "object",
          "properties": {
            "unit": {
              "type": "object",
              "properties": {
                "count": { "type": "integer" },
                "percentage": { "type": "number" },
                "targetPercentage": { "type": "number", "default": 70 },
                "averageExecutionMs": { "type": "number" },
                "isolation": { "type": "string", "enum": ["complete", "partial", "none"] }
              }
            },
            "integration": {
              "type": "object",
              "properties": {
                "count": { "type": "integer" },
                "percentage": { "type": "number" },
                "targetPercentage": { "type": "number", "default": 20 },
                "averageExecutionMs": { "type": "number" },
                "isolation": { "type": "string" }
              }
            },
            "e2e": {
              "type": "object",
              "properties": {
                "count": { "type": "integer" },
                "percentage": { "type": "number" },
                "targetPercentage": { "type": "number", "default": 10 },
                "averageExecutionMs": { "type": "number" },
                "isolation": { "type": "string" }
              }
            }
          }
        },
        "shape": {
          "type": "string",
          "enum": ["healthy-pyramid", "ice-cream-cone", "hourglass", "cupcake", "diamond"],
          "description": "Current test pyramid shape"
        },
        "pyramidHealthScore": {
          "type": "number",
          "minimum": 0,
          "maximum": 100,
          "description": "How healthy is the pyramid"
        },
        "recommendations": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    },
    "firstPrinciples": {
      "type": "object",
      "required": ["fast", "isolated", "repeatable", "selfValidating", "timely"],
      "properties": {
        "fast": {
          "type": "object",
          "properties": {
            "score": { "type": "number", "minimum": 0, "maximum": 100 },
            "totalSuiteTime": { "type": "integer", "description": "Total suite execution time in seconds" },
            "targetTime": { "type": "integer", "description": "Target time in seconds" },
            "slowTests": { "type": "integer", "description": "Number of tests exceeding 1s" },
            "issues": { "type": "array", "items": { "type": "string" } }
          }
        },
        "isolated": {
          "type": "object",
          "properties": {
            "score": { "type": "number", "minimum": 0, "maximum": 100 },
            "sharedState": { "type": "boolean" },
            "orderDependencies": { "type": "boolean" },
            "parallelizable": { "type": "boolean" },
            "issues": { "type": "array", "items": { "type": "string" } }
          }
        },
        "repeatable": {
          "type": "object",
          "properties": {
            "score": { "type": "number", "minimum": 0, "maximum": 100 },
            "flakyTestCount": { "type": "integer" },
            "randomData": { "type": "boolean" },
            "timeDependencies": { "type": "boolean" },
            "environmentDependencies": { "type": "boolean" },
            "issues": { "type": "array", "items": { "type": "string" } }
          }
        },
        "selfValidating": {
          "type": "object",
          "properties": {
            "score": { "type": "number", "minimum": 0, "maximum": 100 },
            "manualVerification": { "type": "boolean" },
            "clearAssertions": { "type": "boolean" },
            "meaningfulFailures": { "type": "boolean" },
            "issues": { "type": "array", "items": { "type": "string" } }
          }
        },
        "timely": {
          "type": "object",
          "properties": {
            "score": { "type": "number", "minimum": 0, "maximum": 100 },
            "testFirstRatio": { "type": "number" },
            "testLag": { "type": "string", "description": "How long after code are tests written" },
            "issues": { "type": "array", "items": { "type": "string" } }
          }
        },
        "overallScore": {
          "type": "number",
          "minimum": 0,
          "maximum": 100
        }
      }
    },
    "automationPatterns": {
      "type": "object",
      "properties": {
        "pageObjectModel": {
          "type": "object",
          "properties": {
            "used": { "type": "boolean" },
            "coverage": { "type": "number" },
            "quality": { "type": "string", "enum": ["poor", "fair", "good", "excellent"] }
          }
        },
        "builderPattern": { "type": "object" },
        "factoryPattern": { "type": "object" },
        "fixtureManagement": {
          "type": "object",
          "properties": {
            "strategy": { "type": "string" },
            "isolation": { "type": "boolean" },
            "cleanup": { "type": "boolean" }
          }
        },
        "dataTestIds": {
          "type": "object",
          "properties": {
            "used": { "type": "boolean" },
            "coverage": { "type": "number" }
          }
        }
      }
    },
    "antiPatterns": {
      "type": "object",
      "properties": {
        "detected": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": {
                "type": "string",
                "enum": ["ice-cream-cone", "slow-suite", "flaky-tests", "coupled-tests", "brittle-selectors", "test-duplication", "hardcoded-data", "sleep-statements", "no-assertions"]
              },
              "severity": { "type": "string", "enum": ["critical", "high", "medium", "low"] },
              "occurrences": { "type": "integer" },
              "examples": { "type": "array", "items": { "type": "string" } },
              "remediation": { "type": "string" }
            }
          }
        },
        "totalCount": { "type": "integer" },
        "criticalCount": { "type": "integer" }
      }
    },
    "cicdIntegration": {
      "type": "object",
      "properties": {
        "integrated": { "type": "boolean" },
        "runsOnEveryCommit": { "type": "boolean" },
        "parallelExecution": { "type": "boolean" },
        "pipelineStages": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "tests": { "type": "string", "enum": ["unit", "integration", "e2e", "all"] },
              "duration": { "type": "integer" },
              "blocking": { "type": "boolean" }
            }
          }
        },
        "totalPipelineDuration": { "type": "integer" },
        "fastFeedback": { "type": "boolean", "description": "< 10 minutes" }
      }
    },
    "flakyTestAnalysis": {
      "type": "object",
      "properties": {
        "flakyCount": { "type": "integer" },
        "flakyRate": { "type": "number", "description": "Percentage of flaky tests" },
        "quarantined": { "type": "integer" },
        "rootCauses": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "cause": {
                "type": "string",
                "enum": ["timing", "data-isolation", "race-condition", "environment", "network", "unknown"]
              },
              "count": { "type": "integer" }
            }
          }
        },
        "recommendations": { "type": "array", "items": { "type": "string" } }
      }
    },
    "automationFinding": {
      "type": "object",
      "required": ["id", "title", "type", "severity"],
      "properties": {
        "id": {
          "type": "string",
          "pattern": "^TAS-\\d{3,6}$"
        },
        "title": {
          "type": "string",
          "minLength": 10,
          "maxLength": 200
        },
        "description": { "type": "string" },
        "type": {
          "type": "string",
          "enum": ["pyramid-issue", "first-violation", "antipattern", "flaky-test", "ci-gap", "pattern-gap", "opportunity"]
        },
        "severity": {
          "type": "string",
          "enum": ["critical", "high", "medium", "low", "info"]
        },
        "firstPrinciple": {
          "type": "string",
          "enum": ["fast", "isolated", "repeatable", "self-validating", "timely"]
        }
      }
    },
    "automationRecommendation": {
      "type": "object",
      "required": ["id", "title", "priority"],
      "properties": {
        "id": {
          "type": "string",
          "pattern": "^REC-\\d{3,6}$"
        },
        "title": { "type": "string" },
        "description": { "type": "string" },
        "priority": {
          "type": "string",
          "enum": ["critical", "high", "medium", "low"]
        },
        "impact": {
          "type": "array",
          "items": {
            "type": "string",
            "enum": ["speed", "reliability", "maintainability", "coverage", "cost"]
          }
        },
        "effort": {
          "type": "string",
          "enum": ["trivial", "low", "medium", "high", "major"]
        }
      }
    },
    "automationMetrics": {
      "type": "object",
      "properties": {
        "totalTests": { "type": "integer" },
        "passRate": { "type": "number", "minimum": 0, "maximum": 100 },
        "flakyRate": { "type": "number", "minimum": 0, "maximum": 100 },
        "coverage": { "type": "number", "minimum": 0, "maximum": 100 },
        "averageExecutionTime": { "type": "integer" },
        "pyramidScore": { "type": "number", "minimum": 0, "maximum": 100 },
        "firstScore": { "type": "number", "minimum": 0, "maximum": 100 }
      }
    },
    "artifact": {
      "type": "object",
      "required": ["type", "path"],
      "properties": {
        "type": { "type": "string" },
        "path": { "type": "string" },
        "format": { "type": "string" },
        "description": { "type": "string" }
      }
    },
    "metadata": {
      "type": "object",
      "properties": {
        "executionTimeMs": { "type": "integer" },
        "agentId": { "type": "string" },
        "modelUsed": { "type": "string" },
        "environment": { "type": "string" }
      }
    },
    "validationResult": {
      "type": "object",
      "properties": {
        "schemaValid": { "type": "boolean" },
        "contentValid": { "type": "boolean" },
        "confidence": { "type": "number" }
      }
    },
    "learningData": {
      "type": "object",
      "properties": {
        "patternsDetected": { "type": "array", "items": { "type": "string" } },
        "reward": { "type": "number" }
      }
    }
  }
}

{
  "skillName": "test-automation-strategy",
  "skillVersion": "1.0.0",
  "requiredTools": [
    "jq"
  ],
  "optionalTools": [
    "ajv",
    "jsonschema",
    "python3"
  ],
  "schemaPath": "schemas/output.json",
  "requiredFields": [
    "skillName",
    "status",
    "output",
    "output.summary"
  ],
  "requiredNonEmptyFields": [
    "output.summary"
  ],
  "mustContainTerms": [
    "strategy",
    "automation",
    "test"
  ],
  "mustNotContainTerms": [
    "TODO",
    "placeholder"
  ],
  "enumValidations": {
    ".status": [
      "success",
      "partial",
      "failed",
      "skipped"
    ]
  }
}