Api Tests

/configure:api-tests

Check and configure API contract testing infrastructure for validating API contracts, schemas, and consumer-provider agreements.

When to Use This Skill

Context

• Lock files: !find . -maxdepth 1 \( -name 'bun.lockb' -o -name 'bun.lock' -o -name 'package-lock.json' -o -name 'yarn.lock' \)
• Project files: !find . -maxdepth 1 \( -name 'tsconfig.json' -o -name 'pyproject.toml' -o -name 'package.json' \)
• Pact installed: !find . -maxdepth 1 \( -name package.json -o -name 'requirements*.txt' -o -name pyproject.toml \) -exec grep -l "pact-foundation/pact\|pact-python" {} +
• OpenAPI spec: !find . -maxdepth 2 \( -name 'openapi.yaml' -o -name 'openapi.yml' -o -name 'openapi.json' -o -name 'swagger.json' -o -name 'swagger.yaml' \)
• Schema validator: !grep -l '"ajv"\|"zod"' package.json
• OpenAPI validation lib: !grep -l "swagger-parser\|@apidevtools" package.json
• Pact contracts dir: !find . -maxdepth 1 -type d -name \'pacts\'
• Contract tests: !find . -maxdepth 3 -type d -name 'contract'
• API test files: !find . -maxdepth 4 \( -name '*.pact.*' -o -name '*.openapi.*' -o -name '*.contract.*' \)
• CI workflows: !find .github/workflows -maxdepth 1 \( -name '*api*' -o -name '*contract*' -o -name '*pact*' \)

Parameters

Parse $ARGUMENTS for these flags:

API Testing Types:

Execution

Execute this API testing compliance check:

Step 1: Detect project infrastructure

Scan the project for existing API testing indicators:

If --type is specified, only check the relevant component.

Step 2: Analyze current state

Check completeness of each API testing component:

Contract Testing (Pact): 1. Check if @pact-foundation/pact (JS/TS) or pact-python (Python) is installed 2. Look for consumer tests in tests/contract/consumer/ 3. Look for provider verification in tests/contract/provider/ 4. Check for Pact Broker configuration in CI files 5. Check for can-i-deploy CI gate

OpenAPI Validation: 1. Verify OpenAPI specification file exists and is valid 2. Check for request validation middleware 3. Check for response validation in tests 4. Look for breaking change detection (oasdiff)

Schema Testing: 1. Check for JSON Schema or Zod definitions 2. Verify validator is installed (ajv or zod) 3. Look for response validation test helpers

Step 3: Generate compliance report

Print a formatted compliance report:

API Testing Compliance Report
==============================
Project: [name]
API Type: [REST | GraphQL | gRPC]

Contract Testing (Pact):
  Package:                   [INSTALLED | MISSING]
  Consumer tests:            [FOUND | NONE]
  Provider tests:            [FOUND | NONE]
  Pact Broker:               [CONFIGURED | OPTIONAL]

OpenAPI Validation:
  OpenAPI spec:              [EXISTS | MISSING]
  Spec version:              [CURRENT | OUTDATED]
  Request validation:        [CONFIGURED | MISSING]
  Response validation:       [CONFIGURED | MISSING]
  Breaking change CI:        [CONFIGURED | OPTIONAL]

Schema Testing:
  JSON Schemas:              [EXISTS | N/A]
  Schema validator:          [INSTALLED | MISSING]
  Response validation:       [CONFIGURED | MISSING]

Overall: [X issues found]
Recommendations: [list specific actions]

If --check-only is set, stop here.

Step 4: Apply configuration

If --fix is set or user confirms, apply fixes for missing components. Use the project language detected in Context to select TypeScript or Python templates.

For each missing component, create the appropriate files using templates from REFERENCE.md:

1. Pact Contract Testing: Install dependencies, create consumer test template, create provider verification template 2. OpenAPI Validation: Install validation libraries, create OpenAPI validator helper, create compliance tests 3. Schema Testing (Zod): Install Zod, create schema definitions, create schema validation tests 4. OpenAPI Breaking Change Detection: Install oasdiff, add CI step

Step 5: Configure CI/CD integration

Create or update .github/workflows/api-tests.yml with jobs for: 1. Consumer contract tests 2. Provider verification (with database service if needed) 3. OpenAPI spec validation and breaking change detection 4. Pact artifact publishing (main branch only)

Add test scripts to package.json. Use CI workflow template from REFERENCE.md.

Step 6: Update standards tracking

Update .project-standards.yaml with API testing configuration:

components:
  api_tests: "2025.1"
  api_tests_contract: "[pact|none]"
  api_tests_openapi: true
  api_tests_schema: "[zod|ajv|none]"
  api_tests_breaking_change_ci: true

Step 7: Generate final report

Print a summary of all changes applied including:

• Configuration files created
• Dependencies installed
• Test commands available (with exact bun run / npm run commands)
• CI/CD jobs configured
• Recommended next steps for verification

Agentic Optimizations

Examples

# Check compliance and offer fixes
/configure:api-tests

# Check only, no modifications
/configure:api-tests --check-only

# Auto-fix all issues
/configure:api-tests --fix

# Configure Pact only
/configure:api-tests --fix --type pact

# Configure OpenAPI validation only
/configure:api-tests --fix --type openapi

Error Handling

See Also

• /configure:tests - Unit testing configuration
• /configure:integration-tests - Integration testing
• /configure:all - Run all compliance checks
• Pact documentation
• OpenAPI specification
• Zod documentation

API Tests Reference

Template code for API contract testing configuration. Used by the /configure:api-tests skill during Step 4 (Apply configuration) and Step 5 (CI/CD integration).

Pact Contract Testing (JavaScript/TypeScript)

Install Dependencies

bun add --dev @pact-foundation/pact @pact-foundation/pact-core

Consumer Test Template

Create tests/contract/consumer/userService.pact.ts:

import { PactV4, MatchersV3 } from '@pact-foundation/pact';
import { resolve } from 'path';

const { like, eachLike, regex, datetime } = MatchersV3;

const provider = new PactV4({
  consumer: 'frontend-app',
  provider: 'user-service',
  dir: resolve(__dirname, '../../../pacts'),
  logLevel: 'warn',
});

describe('User Service Contract', () => {
  describe('GET /api/users/:id', () => {
    it('returns a user when user exists', async () => {
      await provider
        .addInteraction()
        .given('a user with ID 1 exists')
        .uponReceiving('a request to get user 1')
        .withRequest({
          method: 'GET',
          path: '/api/users/1',
          headers: {
            Accept: 'application/json',
          },
        })
        .willRespondWith({
          status: 200,
          headers: {
            'Content-Type': 'application/json',
          },
          body: {
            id: like(1),
            name: like('John Doe'),
            email: regex(/^[\w.-]+@[\w.-]+\.\w+$/, 'john@example.com'),
            createdAt: datetime("yyyy-MM-dd'T'HH:mm:ss.SSSXXX"),
          },
        })
        .executeTest(async (mockServer) => {
          const response = await fetch(`${mockServer.url}/api/users/1`, {
            headers: { Accept: 'application/json' },
          });

          expect(response.status).toBe(200);

          const user = await response.json();
          expect(user).toHaveProperty('id');
          expect(user).toHaveProperty('name');
          expect(user).toHaveProperty('email');
        });
    });

    it('returns 404 when user does not exist', async () => {
      await provider
        .addInteraction()
        .given('no user with ID 999 exists')
        .uponReceiving('a request to get non-existent user')
        .withRequest({
          method: 'GET',
          path: '/api/users/999',
          headers: {
            Accept: 'application/json',
          },
        })
        .willRespondWith({
          status: 404,
          headers: {
            'Content-Type': 'application/json',
          },
          body: {
            error: like('User not found'),
            code: like('USER_NOT_FOUND'),
          },
        })
        .executeTest(async (mockServer) => {
          const response = await fetch(`${mockServer.url}/api/users/999`, {
            headers: { Accept: 'application/json' },
          });

          expect(response.status).toBe(404);
        });
    });
  });

  describe('POST /api/users', () => {
    it('creates a new user', async () => {
      await provider
        .addInteraction()
        .uponReceiving('a request to create a user')
        .withRequest({
          method: 'POST',
          path: '/api/users',
          headers: {
            'Content-Type': 'application/json',
            Accept: 'application/json',
          },
          body: {
            name: like('Jane Doe'),
            email: like('jane@example.com'),
          },
        })
        .willRespondWith({
          status: 201,
          headers: {
            'Content-Type': 'application/json',
          },
          body: {
            id: like(1),
            name: like('Jane Doe'),
            email: like('jane@example.com'),
            createdAt: datetime("yyyy-MM-dd'T'HH:mm:ss.SSSXXX"),
          },
        })
        .executeTest(async (mockServer) => {
          const response = await fetch(`${mockServer.url}/api/users`, {
            method: 'POST',
            headers: {
              'Content-Type': 'application/json',
              Accept: 'application/json',
            },
            body: JSON.stringify({
              name: 'Jane Doe',
              email: 'jane@example.com',
            }),
          });

          expect(response.status).toBe(201);
        });
    });
  });
});

Provider Verification Template

Create tests/contract/provider/userService.provider.ts:

import { Verifier } from '@pact-foundation/pact';
import { resolve } from 'path';
import { app } from '../../../src/app'; // Your Express/Fastify app
import { setupTestDatabase, seedProviderStates } from '../helpers/database';

describe('User Service Provider Verification', () => {
  let server: any;

  beforeAll(async () => {
    await setupTestDatabase();
    server = app.listen(3001);
  });

  afterAll(async () => {
    server.close();
  });

  it('validates the expectations of the consumer', async () => {
    const verifier = new Verifier({
      providerBaseUrl: 'http://localhost:3001',
      pactUrls: [resolve(__dirname, '../../../pacts/frontend-app-user-service.json')],
      // Or use Pact Broker:
      // pactBrokerUrl: process.env.PACT_BROKER_URL,
      // providerVersion: process.env.GIT_SHA,
      // publishVerificationResult: process.env.CI === 'true',

      stateHandlers: {
        'a user with ID 1 exists': async () => {
          await seedProviderStates({
            users: [{ id: 1, name: 'John Doe', email: 'john@example.com' }],
          });
        },
        'no user with ID 999 exists': async () => {
          // Ensure user 999 doesn't exist (default state after cleanup)
        },
      },
    });

    await verifier.verifyProvider();
  });
});

Pact Contract Testing (Python)

Install Dependencies

uv add --group dev pact-python

Consumer Test Template

Create tests/contract/consumer/test_user_service.py:

import pytest
from pact import Consumer, Provider, Like, EachLike, Term
import requests

pact = Consumer('frontend-app').has_pact_with(
    Provider('user-service'),
    pact_dir='./pacts',
    log_dir='./logs',
)

@pytest.fixture(scope='module')
def pact_setup():
    pact.start_service()
    yield pact
    pact.stop_service()

def test_get_user(pact_setup):
    """Test getting a user by ID."""
    expected = {
        'id': Like(1),
        'name': Like('John Doe'),
        'email': Term(r'^[\w.-]+@[\w.-]+\.\w+$', 'john@example.com'),
    }

    (pact_setup
        .given('a user with ID 1 exists')
        .upon_receiving('a request to get user 1')
        .with_request('GET', '/api/users/1')
        .will_respond_with(200, body=expected))

    with pact_setup:
        result = requests.get(f'{pact_setup.uri}/api/users/1')

    assert result.status_code == 200
    assert 'id' in result.json()
    assert 'name' in result.json()

def test_get_nonexistent_user(pact_setup):
    """Test 404 response for non-existent user."""
    (pact_setup
        .given('no user with ID 999 exists')
        .upon_receiving('a request to get non-existent user')
        .with_request('GET', '/api/users/999')
        .will_respond_with(404, body={
            'error': Like('User not found'),
            'code': Like('USER_NOT_FOUND'),
        }))

    with pact_setup:
        result = requests.get(f'{pact_setup.uri}/api/users/999')

    assert result.status_code == 404

OpenAPI Validation (JavaScript/TypeScript)

Install Dependencies

bun add --dev @apidevtools/swagger-parser ajv ajv-formats
bun add --dev openapi-typescript  # For TypeScript types from OpenAPI

OpenAPI Validator Helper

Create tests/api/openapi-validator.ts:

import Ajv from 'ajv';
import addFormats from 'ajv-formats';
import SwaggerParser from '@apidevtools/swagger-parser';
import { OpenAPIV3 } from 'openapi-types';

export class OpenAPIValidator {
  private ajv: Ajv;
  private spec: OpenAPIV3.Document | null = null;
  private schemas: Map<string, object> = new Map();

  constructor() {
    this.ajv = new Ajv({ allErrors: true, strict: false });
    addFormats(this.ajv);
  }

  async loadSpec(specPath: string): Promise<void> {
    this.spec = await SwaggerParser.validate(specPath) as OpenAPIV3.Document;

    // Register all schemas from components
    if (this.spec.components?.schemas) {
      for (const [name, schema] of Object.entries(this.spec.components.schemas)) {
        this.schemas.set(name, schema);
        this.ajv.addSchema(schema, `#/components/schemas/${name}`);
      }
    }
  }

  validateResponse(
    path: string,
    method: string,
    statusCode: number,
    body: unknown
  ): { valid: boolean; errors: string[] } {
    if (!this.spec) {
      throw new Error('OpenAPI spec not loaded. Call loadSpec() first.');
    }

    const pathItem = this.spec.paths?.[path];
    if (!pathItem) {
      return { valid: false, errors: [`Path ${path} not found in spec`] };
    }

    const operation = pathItem[method.toLowerCase() as keyof OpenAPIV3.PathItemObject] as OpenAPIV3.OperationObject;
    if (!operation) {
      return { valid: false, errors: [`Method ${method} not found for path ${path}`] };
    }

    const response = operation.responses?.[statusCode] || operation.responses?.['default'];
    if (!response) {
      return { valid: false, errors: [`Status ${statusCode} not defined for ${method} ${path}`] };
    }

    const responseObj = response as OpenAPIV3.ResponseObject;
    const content = responseObj.content?.['application/json'];
    if (!content?.schema) {
      // No schema defined, consider valid
      return { valid: true, errors: [] };
    }

    const validate = this.ajv.compile(content.schema);
    const valid = validate(body);

    return {
      valid: !!valid,
      errors: validate.errors?.map(e => `${e.instancePath} ${e.message}`) || [],
    };
  }

  validateRequest(
    path: string,
    method: string,
    body: unknown
  ): { valid: boolean; errors: string[] } {
    if (!this.spec) {
      throw new Error('OpenAPI spec not loaded. Call loadSpec() first.');
    }

    const pathItem = this.spec.paths?.[path];
    if (!pathItem) {
      return { valid: false, errors: [`Path ${path} not found in spec`] };
    }

    const operation = pathItem[method.toLowerCase() as keyof OpenAPIV3.PathItemObject] as OpenAPIV3.OperationObject;
    if (!operation?.requestBody) {
      return { valid: true, errors: [] };
    }

    const requestBody = operation.requestBody as OpenAPIV3.RequestBodyObject;
    const content = requestBody.content?.['application/json'];
    if (!content?.schema) {
      return { valid: true, errors: [] };
    }

    const validate = this.ajv.compile(content.schema);
    const valid = validate(body);

    return {
      valid: !!valid,
      errors: validate.errors?.map(e => `${e.instancePath} ${e.message}`) || [],
    };
  }
}

// Helper for tests
export async function createValidator(specPath: string = './openapi.yaml'): Promise<OpenAPIValidator> {
  const validator = new OpenAPIValidator();
  await validator.loadSpec(specPath);
  return validator;
}

OpenAPI Compliance Test Template

Create tests/api/users.openapi.test.ts:

import { describe, it, expect, beforeAll } from 'vitest';
import request from 'supertest';
import { app } from '../../src/app';
import { createValidator, OpenAPIValidator } from './openapi-validator';

describe('Users API - OpenAPI Compliance', () => {
  let validator: OpenAPIValidator;

  beforeAll(async () => {
    validator = await createValidator('./openapi.yaml');
  });

  describe('GET /api/users', () => {
    it('response matches OpenAPI schema', async () => {
      const response = await request(app)
        .get('/api/users')
        .expect(200);

      const result = validator.validateResponse('/api/users', 'GET', 200, response.body);

      expect(result.valid).toBe(true);
      if (!result.valid) {
        console.error('Validation errors:', result.errors);
      }
    });
  });

  describe('POST /api/users', () => {
    it('request matches OpenAPI schema', async () => {
      const requestBody = {
        name: 'Test User',
        email: 'test@example.com',
      };

      const requestValidation = validator.validateRequest('/api/users', 'POST', requestBody);
      expect(requestValidation.valid).toBe(true);

      const response = await request(app)
        .post('/api/users')
        .send(requestBody)
        .expect(201);

      const responseValidation = validator.validateResponse('/api/users', 'POST', 201, response.body);
      expect(responseValidation.valid).toBe(true);
    });

    it('rejects invalid request body', async () => {
      const invalidBody = {
        name: 123, // Should be string
        // Missing required email
      };

      const validation = validator.validateRequest('/api/users', 'POST', invalidBody);
      expect(validation.valid).toBe(false);
      expect(validation.errors.length).toBeGreaterThan(0);
    });
  });

  describe('GET /api/users/:id', () => {
    it('404 response matches OpenAPI schema', async () => {
      const response = await request(app)
        .get('/api/users/99999')
        .expect(404);

      const result = validator.validateResponse('/api/users/{id}', 'GET', 404, response.body);
      expect(result.valid).toBe(true);
    });
  });
});

OpenAPI Breaking Change Detection

Install oasdiff

# Via npm/bun
bun add --dev @oasdiff/oasdiff

# Or via homebrew
brew install oasdiff

CI Breaking Change Check

- name: Check for breaking API changes
  run: |
    # Fetch main branch spec
    git fetch origin main
    git show origin/main:openapi.yaml > openapi-main.yaml

    # Check for breaking changes
    oasdiff breaking openapi-main.yaml openapi.yaml --fail-on ERR

    # Generate changelog
    oasdiff changelog openapi-main.yaml openapi.yaml

Schema Testing with Zod

Install Dependencies

bun add zod
bun add --dev @anatine/zod-openapi  # Optional: generate OpenAPI from Zod

Schema Definitions Template

Create src/schemas/user.ts:

import { z } from 'zod';

export const UserSchema = z.object({
  id: z.number().int().positive(),
  name: z.string().min(1).max(100),
  email: z.string().email(),
  createdAt: z.string().datetime(),
  updatedAt: z.string().datetime().optional(),
});

export const CreateUserSchema = UserSchema.omit({
  id: true,
  createdAt: true,
  updatedAt: true,
});

export const UpdateUserSchema = CreateUserSchema.partial();

export const UserListSchema = z.array(UserSchema);

export type User = z.infer<typeof UserSchema>;
export type CreateUser = z.infer<typeof CreateUserSchema>;
export type UpdateUser = z.infer<typeof UpdateUserSchema>;

Schema Validation Test Template

Create tests/api/schema.test.ts:

import { describe, it, expect } from 'vitest';
import request from 'supertest';
import { app } from '../../src/app';
import { UserSchema, UserListSchema } from '../../src/schemas/user';

describe('API Schema Validation', () => {
  describe('GET /api/users', () => {
    it('response matches User list schema', async () => {
      const response = await request(app)
        .get('/api/users')
        .expect(200);

      const result = UserListSchema.safeParse(response.body);

      expect(result.success).toBe(true);
      if (!result.success) {
        console.error('Schema errors:', result.error.format());
      }
    });
  });

  describe('GET /api/users/:id', () => {
    it('response matches User schema', async () => {
      // Assuming user 1 exists
      const response = await request(app)
        .get('/api/users/1')
        .expect(200);

      const result = UserSchema.safeParse(response.body);

      expect(result.success).toBe(true);
    });
  });
});

CI/CD Workflow Template

Create .github/workflows/api-tests.yml:

name: API Contract Tests

on:
  push:
    branches: [main]
  pull_request:
    paths:
      - 'openapi.yaml'
      - 'src/api/**'
      - 'tests/contract/**'

jobs:
  consumer-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: oven-sh/setup-bun@v2

      - name: Install dependencies
        run: bun install --frozen-lockfile

      - name: Run consumer contract tests
        run: bun run test:contract:consumer

      - name: Upload pacts
        uses: actions/upload-artifact@v4
        with:
          name: pacts
          path: pacts/

  provider-tests:
    runs-on: ubuntu-latest
    needs: consumer-tests
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test_db
        ports:
          - 5432:5432

    steps:
      - uses: actions/checkout@v4

      - uses: oven-sh/setup-bun@v2

      - name: Install dependencies
        run: bun install --frozen-lockfile

      - name: Download pacts
        uses: actions/download-artifact@v4
        with:
          name: pacts
          path: pacts/

      - name: Run provider verification
        run: bun run test:contract:provider
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/test_db

  openapi-validation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Validate OpenAPI spec
        run: |
          bunx @apidevtools/swagger-cli validate openapi.yaml

      - name: Check for breaking changes
        if: github.event_name == 'pull_request'
        run: |
          git fetch origin main
          git show origin/main:openapi.yaml > openapi-main.yaml || echo "No existing spec"

          if [ -f openapi-main.yaml ]; then
            bunx oasdiff breaking openapi-main.yaml openapi.yaml --fail-on ERR
          fi

  # Optional: Publish to Pact Broker
  publish-pacts:
    runs-on: ubuntu-latest
    needs: [consumer-tests, provider-tests]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4

      - name: Download pacts
        uses: actions/download-artifact@v4
        with:
          name: pacts
          path: pacts/

      - name: Publish to Pact Broker
        run: |
          curl -X PUT \
            -H "Content-Type: application/json" \
            -d @pacts/frontend-app-user-service.json \
            "${{ secrets.PACT_BROKER_URL }}/pacts/provider/user-service/consumer/frontend-app/version/${{ github.sha }}"
        env:
          PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}

Package.json Scripts

Add these scripts to package.json:

{
  "scripts": {
    "test:contract": "bun run test:contract:consumer && bun run test:contract:provider",
    "test:contract:consumer": "vitest run tests/contract/consumer/",
    "test:contract:provider": "vitest run tests/contract/provider/",
    "test:openapi": "vitest run tests/api/*.openapi.test.ts",
    "test:schema": "vitest run tests/api/schema.test.ts",
    "openapi:validate": "bunx @apidevtools/swagger-cli validate openapi.yaml",
    "openapi:bundle": "bunx @apidevtools/swagger-cli bundle openapi.yaml -o dist/openapi.json",
    "openapi:types": "bunx openapi-typescript openapi.yaml -o src/types/api.d.ts"
  }
}

Final Report Template

API Testing Configuration Complete
===================================

Contract Testing: Pact
Schema Validation: Zod
OpenAPI: 3.1

Configuration Applied:
  - @pact-foundation/pact installed
  - Consumer contract tests created
  - Provider verification configured
  - OpenAPI validator created
  - Zod schemas configured

Test Structure:
  - tests/contract/consumer/ - Consumer tests
  - tests/contract/provider/ - Provider verification
  - tests/api/*.openapi.test.ts - OpenAPI validation
  - pacts/ - Generated contracts

Scripts Added:
  - bun run test:contract (all contract tests)
  - bun run test:contract:consumer (consumer only)
  - bun run test:contract:provider (provider only)
  - bun run test:openapi (OpenAPI validation)
  - bun run openapi:validate (spec validation)

CI/CD:
  - Consumer tests job
  - Provider verification job
  - OpenAPI breaking change detection
  - Pact artifact upload

Next Steps:
  1. Run consumer tests:
     bun run test:contract:consumer

  2. Verify provider:
     bun run test:contract:provider

  3. Validate OpenAPI spec:
     bun run openapi:validate

  4. Check API compliance:
     bun run test:openapi

Documentation:
  - Pact: https://docs.pact.io
  - OpenAPI: https://swagger.io/specification
  - Zod: https://zod.dev