Learn From Code Review

Learn from Code Review

Analyze code review comments from GitHub pull requests and distill them into reusable skills or repository guidelines that improve future code quality.

Overview

Code review feedback contains valuable institutional knowledge that often gets buried across hundreds of PRs. This skill extracts meaningful patterns from review comments and transforms them into:

1. Repository-specific skills - Placed in .openhands/skills/ for domain-specific patterns 2. AGENTS.md guidelines - Overall repository conventions and best practices

Prerequisites

• GITHUB_TOKEN environment variable must be set
• GitHub CLI (gh) should be available

Workflow

Step 1: Identify Target Repository

Determine the repository to analyze:

# Get current repo info
gh repo view --json nameWithOwner -q '.nameWithOwner'

If not in a repository, ask the user which repository to analyze.

Step 2: Fetch Review Comments

Retrieve PR review comments from the repository:

# Fetch merged PRs from the last 30 days (adjustable)
gh pr list --repo {owner}/{repo} \
  --state merged \
  --limit 50 \
  --json number,title,mergedAt

# For each PR, fetch review comments
gh api repos/{owner}/{repo}/pulls/{pr_number}/comments \
  --jq '.[] | {body: .body, path: .path, user: .user.login, created_at: .created_at}'

# Also fetch review-level comments (not tied to specific lines)
gh api repos/{owner}/{repo}/pulls/{pr_number}/reviews \
  --jq '.[] | select(.body != "") | {body: .body, user: .user.login, state: .state}'

Step 3: Filter and Categorize Comments

Apply noise filtering to keep only meaningful feedback:

Exclude:

• Bot comments (dependabot, copilot, github-actions, etc.)
• Low-signal responses ("LGTM", "+1", "looks good", "thanks", "nice")
• Comments shorter than 30 characters
• Auto-generated comments (CI status, coverage reports)

Categorize remaining comments by:

• Security concerns
• Performance patterns
• Code style/conventions
• Architecture/design patterns
• Error handling
• Testing requirements
• Documentation standards

Step 4: Distill Patterns

For each category with sufficient examples (3+ similar comments), identify:

1. The recurring issue - What mistake or oversight keeps appearing 2. The desired pattern - What reviewers consistently ask for 3. Example context - Concrete before/after code snippets when available

Step 5: Generate Output

If clear, actionable patterns emerge, generate focused skill files. If no clear patterns emerge, report this to the user—it's fine to produce no output when the codebase already has strong conventions or when review comments don't cluster into recurring themes.

When creating skills, place them in .openhands/skills/{domain-name}/SKILL.md:

---
name: database-queries
description: Database query patterns and best practices for this repository.
---

# Database Query Guidelines

### Always Use Parameterized Queries
[Pattern description with examples]

### Connection Pool Management
[Pattern description with examples]

Prefer skills over AGENTS.md updates, since AGENTS.md typically already contains general coding guidelines.

Step 6: Create Draft PR (if applicable)

Use the create_pr tool to open a draft PR with the proposed changes. The PR description should include:

• Number of PRs analyzed
• Number of comments processed
• Categories of patterns found
• List of proposed changes (new skills and/or AGENTS.md updates)

Example Output

Sample Skill: API Error Handling

---
name: api-error-handling
description: API error handling patterns for this repository.
---

# API Error Handling

## Always Return Structured Errors

❌ Avoid:

return {"error": str(e)}

✅ Prefer:

return { "error": { "code": "VALIDATION_ERROR", "message": "Invalid input", "details": {"field": "email", "reason": "Invalid format"} } }

## Log Before Returning Errors

logger.error(f"API error in {endpoint}: {e}", exc_info=True) return error_response(e)

Defaults

This workflow analyzes PRs from the past 30 days by default.

Best Practices

1. Run periodically - Schedule monthly or quarterly to capture evolving patterns 2. Review before merging - Generated content is a draft; human review is essential 3. Iterate - Refine patterns based on team feedback 4. Avoid duplication - Check existing AGENTS.md and skills before adding 5. Cite sources - Reference PR numbers when documenting patterns

Error Handling

Handle these common edge cases gracefully:

• Repository has few PRs: If fewer than 10 merged PRs exist in the timeframe, inform the user that there may not be enough data to identify patterns. Proceed with analysis but note the limited sample size.
• No patterns emerge: When comments don't cluster into recurring themes (common for well-established codebases), report this to the user and suggest either expanding the time range or that the codebase may already have strong conventions.
• Token lacks repository access: If the GitHub API returns 403/404, explain that the token may not have access to the repository and suggest checking token permissions.
• `gh` CLI unavailable: Fall back to direct GitHub API calls using curl with $GITHUB_TOKEN, or inform the user that gh needs to be installed.

Limitations

• Only analyzes accessible repositories (requires appropriate permissions)
• Cannot capture verbal feedback from pair programming or meetings
• Patterns may reflect individual reviewer preferences vs. team consensus
• Historical comments may reference outdated code patterns

Additional Resources

For posting structured code reviews, see the github-pr-review skill. For creating new skills, see the skill-creator skill.

Read and follow the complete instructions in the SKILL.md file located in this skill's directory.

$ARGUMENTS

Learn from Code Review

Distill code review feedback from GitHub pull requests into reusable skills and repository guidelines.

What It Does

This skill analyzes PR review comments from a repository and extracts recurring patterns of feedback. These patterns are then transformed into:

• Repository-specific skills (.openhands/skills/) for domain-specific patterns
• AGENTS.md updates for general coding conventions and best practices

When to Use

• "Learn from our code reviews"
• "Distill PR feedback into guidelines"
• "What patterns do reviewers keep pointing out?"
• "Generate coding standards from review history"
• Running /learn-from-reviews

Requirements

• GITHUB_TOKEN environment variable
• GitHub CLI (gh) available

Example

User: Learn from our code reviews over the past month

Agent: I'll analyze recent PR review comments and distill them into actionable guidelines.

[Analyzes 25 merged PRs with 150 review comments]
[Filters out bot comments and low-signal responses]
[Identifies 4 recurring patterns]

Found the following patterns from code review feedback:

1. Error Handling (12 comments)
   - Always include context when logging errors
   - Use structured error responses in APIs

2. Testing (8 comments)
   - Add edge case tests for validation logic
   - Mock external services consistently

3. Database Queries (6 comments)
   - Use parameterized queries exclusively
   - Add appropriate indexes for new queries

I'll create a draft PR with:
- New skill: `.openhands/skills/error-handling/SKILL.md`
- Updates to `AGENTS.md` with testing and database guidelines

Output

The skill generates a draft PR containing proposed changes based on the analysis. Human review is expected before merging.

Related Skills

• github-pr-review - Post structured code reviews
• code-review - Perform code reviews
• skill-creator - Create new skills manually
• agent-memory - Persist repository knowledge