Playwright Devops

# CI Health Report

Generate a CI health report for the last commit on the `main` branch of `microsoft/playwright`.
This is an overall tree health report — not a commit regression analysis. The goal is to show
the full picture of what's failing, grouping by root cause. If any failures appear to be
regressions introduced by this specific commit, call that out, but most failures will be
pre-existing flakes, infrastructure issues, or platform-specific problems.

## Phase 1 — Fetch logs

Run the fetch script to download all failed job logs into a local folder:

```
bash .claude/skills/playwright-devops/fetch-commit-logs.sh [<sha>]
```

- If no SHA is provided, it fetches the last commit on `main`.
- Creates `~/tmp/commit-<short-sha>/` with:
  - `summary.json` — commit info, failed workflows, and failed job metadata
  - `<workflow-name>/<job-name>.log` — failed log output for each failed job

**Note:** The script fetches failed jobs from both failed AND in-progress workflows.
Workflows may still be running while some of their jobs have already failed — these
must be included in the report. If any workflows are still in progress, note this in
the report summary.

## Phase 2 — Analyze

1. **Read `summary.json`** to get the commit message and the list of failed workflows/jobs.

2. **Read each `.log` file** and extract failing test names and error messages.

3. **Compile the report leading with a summary**, then detailed tables:

   ```markdown
   # CI Health Report — <short-sha>

   Commit: `<commit message>`

   ## Summary

   Brief overview: N workflows, N failed jobs, N total test failures.
   Note if any workflows are still in progress.

   ### Possible regressions (may be related to this commit)
   - **N failures** in `test/file.spec.ts` across <browsers/platforms> — <brief description>

   ### Pre-existing / flaky
   - **N failures** in `test/file.spec.ts` — <brief description> (timeouts, infrastructure, platform-specific)

   ### Infrastructure issues
   - <description of non-test failures>

   ---

   ## Detailed Failures

   ### Workflow: <name> (run <id>)

   #### <job name> (job <id>) -- N failures
   | Test | Error |
   |------|-------|
   | `path/to/test.spec.ts:line` -- test title | error message |
   ```

   The summary should appear first so readers immediately see what matters. Group related failures
   (e.g. same test failing across browsers) into single summary bullet points rather than listing each individually.

4. **Save the report** to `ci-failures-<short-sha>.md` in the repo root.


#!/bin/bash
set -euo pipefail

# Usage: fetch-commit-logs.sh [<sha>]
# Fetches CI failure logs for a commit into ~/tmp/commit-<short-sha>/

REPO="microsoft/playwright"
REF="${1:-main}"

# Resolve commit
COMMIT_JSON=$(gh api "repos/$REPO/commits/$REF" --jq '{sha: .sha, message: .commit.message}')
SHA=$(echo "$COMMIT_JSON" | jq -r '.sha')
SHORT_SHA="${SHA:0:7}"
MESSAGE=$(echo "$COMMIT_JSON" | jq -r '.message' | head -1)

OUTDIR="$HOME/tmp/commit-$SHORT_SHA"
mkdir -p "$OUTDIR"

echo "Commit: $SHORT_SHA — $MESSAGE"
echo "Output: $OUTDIR"

# Get all workflow runs for this commit
RUNS_JSON=$(gh api "repos/$REPO/actions/runs?head_sha=$SHA&per_page=50" \
  --jq '[.workflow_runs[] | {id, name, conclusion, workflow_id}]')

# Filter to failed workflows
FAILED_RUNS=$(echo "$RUNS_JSON" | jq -c '[.[] | select(.conclusion == "failure" or .conclusion == null)]')
FAILED_COUNT=$(echo "$FAILED_RUNS" | jq 'length')

if [ "$FAILED_COUNT" -eq 0 ]; then
  echo "No failed workflows."
  echo '{"sha":"'"$SHA"'","short_sha":"'"$SHORT_SHA"'","message":"'"$MESSAGE"'","workflows":[]}' | jq . > "$OUTDIR/summary.json"
  echo "$OUTDIR"
  exit 0
fi

echo "Found $FAILED_COUNT failed workflow(s). Fetching failed jobs..."

# Build summary and collect jobs to fetch
SUMMARY='{"sha":"'"$SHA"'","short_sha":"'"$SHORT_SHA"'","message":'"$(echo "$MESSAGE" | jq -Rs .)"',"workflows":[]}'

sanitize() {
  echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9._-]/-/g' | sed 's/