Workflow Monitor

Table of Contents

• Philosophy
• Quick Start
• Detection Patterns
• Workflow
• Issue Template
• Configuration
• Guardrails
• Integration Points
• Output Format

Workflow Monitor

Monitor workflow executions for errors and inefficiencies, automatically creating issues on the detected git platform (GitHub/GitLab) for improvements. Check session context for git_platform: and use Skill(leyline:git-platform) for CLI command mapping.

Philosophy

Workflows should improve over time. When execution issues occur, capturing them systematically enables continuous improvement. This skill hooks into workflow execution to detect problems and propose fixes.

Quick Start

Manual Invocation

# After a failed workflow
/workflow-monitor --analyze-last

# Monitor a specific workflow execution
/workflow-monitor --session <session-id>

# Analyze efficiency of recent workflows
/workflow-monitor --efficiency-report

Automatic Monitoring (via hooks)

When enabled, workflow-monitor observes execution and flags:

• Command failures (exit codes > 0)
• Timeout events
• Repeated retry patterns
• Context exhaustion
• Inefficient tool usage

Detection Patterns

Error Detection

Efficiency Detection

Workflow

Phase 1: Capture (workflow-monitor:capture-complete)

1. Log execution events - Commands, outputs, timing 2. Tag anomalies - Failures, timeouts, inefficiencies 3. Store evidence - For reproducibility

Phase 2: Analyze (workflow-monitor:analysis-complete)

1. Classify issues - Error type, severity, scope 2. Identify root cause - What triggered the issue 3. Suggest fix - What would prevent recurrence

Phase 3: Report (workflow-monitor:report-generated)

1. Generate issue body - Structured format 2. Assign labels - workflow, bug, enhancement 3. Link evidence - Command outputs, session info

Phase 4: Create Issue (workflow-monitor:issue-created)

1. Check for duplicates - Search existing issues 2. Create if unique - Via gh CLI 3. Link to session - For traceability

Issue Template

## Background

Detected during workflow execution on [DATE].

**Source:** [workflow name] session [session-id]

## Problem

[Description of the error or inefficiency]

**Evidence:**

[Command that failed or was inefficient] [Output excerpt]

## Suggested Fix

[What should change to prevent this]

## Acceptance Criteria

- [ ] [Specific fix criterion]
- [ ] Tests added for new behavior
- [ ] Documentation updated

---
*Created automatically by workflow-monitor*

Configuration

# .workflow-monitor.yaml
enabled: true
auto_create_issues: false  # Require approval before creating
severity_threshold: "medium"  # Only report medium+ severity
efficiency_threshold: 0.7  # Flag workflows below 70% efficiency

detection:
  command_failures: true
  timeouts: true
  retry_loops: true
  context_exhaustion: true
  tool_misuse: true

efficiency:
  verbose_output_limit: 500
  max_file_reads: 2
  parallel_detection: true

Guardrails

1. No duplicate issues - Check existing issues before creating 2. Approval required - Unless auto_create_issues: true 3. Evidence required - Every issue must have reproducible evidence 4. Rate limiting - Max 5 issues per session

Required TodoWrite Items

1. workflow-monitor:capture-complete 2. workflow-monitor:analysis-complete 3. workflow-monitor:report-generated 4. workflow-monitor:issue-created (if issue created)

Integration Points

• `imbue:proof-of-work`: Captures execution evidence
• `sanctum:fix-workflow`: Implements suggested fixes
• Hooks: Can be triggered by session hooks for automatic monitoring

Output Format

Efficiency Report

## Workflow Efficiency Report

**Session:** [session-id]
**Duration:** 12m 34s
**Efficiency Score:** 0.72 (72%)

### Issues Detected

| Type | Count | Impact |
|------|-------|--------|
| Verbose output | 3 | Medium |
| Redundant reads | 2 | Low |
| Sequential tasks | 1 | Medium |

### Recommendations

1. Use `--quiet` flags for npm/pip commands
2. Cache file contents instead of re-reading
3. Parallelize independent file operations

### Create Issues?

- [ ] Issue 1: Verbose output from npm install
- [ ] Issue 2: Redundant file reads in validation

Related Skills

• imbue:proof-of-work: Evidence capture methodology
• sanctum:fix-workflow: Workflow improvement command
• imbue:proof-of-work: Validation methodology

---

Status: Skeleton implementation. Requires:

• Hook integration for automatic monitoring
• Efficiency scoring algorithm
• Duplicate detection logic

Detection Patterns

Patterns for detecting workflow errors and inefficiencies.

Error Patterns

Command Failure

# Detection: Exit code > 0
command_output=$(some_command 2>&1)
exit_code=$?
if [ $exit_code -ne 0 ]; then
  echo "ERROR: Command failed with exit code $exit_code"
fi

Signals:

• Non-zero exit code
• stderr output containing "error", "failed", "exception"
• Traceback patterns in output

Timeout Events

# Detection: Command exceeds timeout
timeout 120 some_long_command
if [ $? -eq 124 ]; then
  echo "TIMEOUT: Command exceeded 120s limit"
fi

Signals:

• Exit code 124 (timeout)
• "timed out" in output
• Session timeout warnings

Retry Loops

Detection: Same command executed more than 3 times in a session.

def detect_retry_loop(commands: list[str]) -> bool:
    """Detect if same command is retried excessively."""
    from collections import Counter
    counts = Counter(commands)
    return any(count > 3 for count in counts.values())

Signals:

• Repeated identical commands
• Similar commands with minor variations
• "retrying" patterns in output

Context Exhaustion

Detection: Context usage exceeds threshold.

Signals:

• "/context" shows >90% usage
• Truncation warnings
• "context limit" messages

Efficiency Patterns

Verbose Output

Detection: Command produces excessive output.

# Check output line count
output_lines=$(some_command | wc -l)
if [ $output_lines -gt 500 ]; then
  echo "WARNING: Verbose output ($output_lines lines)"
  echo "Suggestion: Use --quiet or redirect to file"
fi

Common offenders:

• npm install without --silent
• pip install without --quiet
• git log without -n limit
• find without | head

Redundant File Reads

Detection: Same file read multiple times.

def detect_redundant_reads(read_events: list[dict]) -> list[str]:
    """Find files read more than twice."""
    from collections import Counter
    file_counts = Counter(e["file_path"] for e in read_events)
    return [f for f, count in file_counts.items() if count > 2]

Suggestions:

• Cache file contents in variables
• Use Read tool with offset/limit for large files
• Batch related reads together

Sequential vs Parallel

Detection: Independent operations run sequentially.

def detect_parallelizable(operations: list[dict]) -> bool:
    """Check if operations could be parallelized."""
    # Operations are independent if:
    # - No data dependencies between them
    # - Different target files/resources
    # - No ordering requirements
    pass

Examples:

• Multiple independent gh API calls
• Reading unrelated files
• Running independent tests

Over-Fetching

Detection: Large file read when only portion needed.

Signals:

• Full file read followed by small extraction
• Large files read without offset/limit
• Regex search on entire file content

Severity Classification

Evidence Collection

For each detected pattern, collect:

1. Command/action - What was executed 2. Output - Full or relevant excerpt 3. Timing - When it occurred, duration 4. Context - What was happening before/after 5. Severity - Based on classification above

Format as evidence log entry:

{
  "id": "E1",
  "type": "command_failure",
  "severity": "high",
  "command": "npm test",
  "exit_code": 1,
  "output_excerpt": "FAIL src/test.js\n  Test failed: expected...",
  "timestamp": "2025-01-14T10:30:00Z",
  "context": "Running validation phase"
}

Efficiency Metrics

Metrics and scoring algorithms for workflow efficiency analysis.

Efficiency Score

Overall efficiency is scored 0.0 to 1.0:

efficiency_score = 1.0 - (penalty_sum / max_penalty)

Where penalties are accumulated for detected inefficiencies.

Penalty Categories

Output Verbosity (max 0.3 penalty)

Common offenders:

npm_install:
  default: 500+ lines
  with_silent: ~10 lines
  recommendation: "npm install --silent"

pip_install:
  default: 200+ lines
  with_quiet: ~5 lines
  recommendation: "pip install --quiet"

git_log:
  default: unlimited
  with_limit: controlled
  recommendation: "git log --oneline -10"

Redundant Operations (max 0.25 penalty)

Detection:

def calculate_redundancy_penalty(operations: list[dict]) -> float:
    """Calculate penalty for redundant operations."""
    from collections import Counter

    # Group by operation signature (command + key args)
    signatures = [op.get("signature", op.get("command")) for op in operations]
    counts = Counter(signatures)

    max_repetitions = max(counts.values()) if counts else 0

    if max_repetitions <= 2:
        return 0.0
    elif max_repetitions == 3:
        return 0.05
    elif max_repetitions <= 5:
        return 0.15
    else:
        return 0.25

Parallelization Misses (max 0.2 penalty)

Detection criteria:

• Operations with no data dependencies
• Different target resources
• No ordering requirements
• Could be batched in single tool call

Over-Fetching (max 0.15 penalty)

Detection:

• Large file read (>1000 lines) followed by small extraction
• Full file read when offset/limit would suffice
• Entire directory listing when glob pattern would work

Context Waste (max 0.1 penalty)

Efficiency Score Interpretation

Metric Collection

Required Data Points

@dataclass
class WorkflowMetrics:
    """Metrics collected during workflow execution."""

    # Output metrics
    total_output_lines: int = 0
    verbose_commands: list[str] = field(default_factory=list)

    # Redundancy metrics
    operation_counts: dict[str, int] = field(default_factory=dict)
    file_read_counts: dict[str, int] = field(default_factory=dict)

    # Parallelization metrics
    sequential_independent_ops: int = 0
    parallel_opportunities_missed: int = 0

    # Over-fetching metrics
    large_reads_without_limit: int = 0
    full_reads_with_extraction: int = 0

    # Context metrics
    context_usage_percent: float = 0.0
    context_warnings: int = 0

Calculation Example

def calculate_efficiency_score(metrics: WorkflowMetrics) -> float:
    """Calculate overall efficiency score."""
    penalties = 0.0

    # Output verbosity penalty
    if metrics.total_output_lines > 1000:
        penalties += 0.30
    elif metrics.total_output_lines > 500:
        penalties += 0.15
    elif metrics.total_output_lines > 100:
        penalties += 0.05

    # Redundancy penalty
    max_repeats = max(metrics.operation_counts.values(), default=0)
    if max_repeats > 5:
        penalties += 0.25
    elif max_repeats > 3:
        penalties += 0.15
    elif max_repeats == 3:
        penalties += 0.05

    # Parallelization penalty
    if metrics.parallel_opportunities_missed > 5:
        penalties += 0.20
    elif metrics.parallel_opportunities_missed > 2:
        penalties += 0.10
    elif metrics.parallel_opportunities_missed > 0:
        penalties += 0.05

    # Over-fetching penalty
    over_fetches = metrics.large_reads_without_limit + metrics.full_reads_with_extraction
    if over_fetches > 2:
        penalties += 0.15
    elif over_fetches > 0:
        penalties += 0.05

    # Context waste penalty
    if metrics.context_usage_percent > 90:
        penalties += 0.10
    elif metrics.context_usage_percent > 80:
        penalties += 0.06
    elif metrics.context_usage_percent > 60:
        penalties += 0.03

    # Cap at 1.0 total penalty
    return max(0.0, 1.0 - min(penalties, 1.0))

Reporting

Efficiency Report Format

## Workflow Efficiency Report

**Session:** {{SESSION_ID}}
**Duration:** {{DURATION}}
**Overall Score:** {{SCORE}} ({{RATING}})

### Penalty Breakdown

| Category | Penalty | Details |
|----------|---------|---------|
| Output verbosity | 0.15 | 3 verbose commands (750 lines total) |
| Redundant ops | 0.05 | Same file read 3 times |
| Parallelization | 0.00 | No missed opportunities |
| Over-fetching | 0.05 | 1 large file read |
| Context waste | 0.03 | 65% context used |
| **Total** | **0.28** | |

### Recommendations

1. Use `npm install --silent` (saves ~500 lines)
2. Cache config.json after first read
3. Consider using offset/limit for large-file.md

Thresholds

Default thresholds (configurable in .workflow-monitor.yaml):

efficiency:
  score_threshold: 0.7  # Report if below
  output_limit: 500     # Lines before penalty
  max_file_reads: 2     # Per file before penalty
  context_warning: 80   # Percent before penalty

Issue Templates

Templates for creating GitHub issues from workflow monitoring findings.

Template Selection

Select template based on issue type:

Error Report Template

## Background

Detected during workflow execution on {{DATE}}.

**Source:** {{WORKFLOW_NAME}} session {{SESSION_ID}}
**Severity:** {{SEVERITY}}

## Problem

{{ERROR_DESCRIPTION}}

**Command:**

{{COMMAND}}

**Output:**

{{OUTPUT_EXCERPT}}

**Exit code:** {{EXIT_CODE}}

## Context

- Previous operation: {{PREVIOUS_OP}}
- Working directory: {{WORKING_DIR}}
- Environment factors: {{ENV_NOTES}}

## Suggested Fix

{{SUGGESTED_FIX}}

## Acceptance Criteria

- [ ] Error no longer occurs in similar conditions
- [ ] Tests added to catch regression
- [ ] Documentation updated if behavior changed

---
*Created by workflow-monitor*

Performance Issue Template

## Background

Performance issue detected on {{DATE}}.

**Source:** {{WORKFLOW_NAME}} session {{SESSION_ID}}
**Type:** {{ISSUE_TYPE}} (timeout / slow response / resource exhaustion)

## Problem

{{PERFORMANCE_DESCRIPTION}}

**Evidence:**
- Duration: {{DURATION}}
- Expected: {{EXPECTED_DURATION}}
- Resource usage: {{RESOURCE_NOTES}}

## Impact

{{IMPACT_DESCRIPTION}}

## Suggested Fix

{{SUGGESTED_FIX}}

## Acceptance Criteria

- [ ] Operation completes within expected time
- [ ] No resource exhaustion under normal conditions
- [ ] Performance test added

---
*Created by workflow-monitor*

Flaky Workflow Template

## Background

Flaky behavior detected on {{DATE}}.

**Source:** {{WORKFLOW_NAME}} session {{SESSION_ID}}
**Retry count:** {{RETRY_COUNT}}

## Problem

{{FLAKY_DESCRIPTION}}

**Retry pattern:**

{{RETRY_HISTORY}}

## Root Cause Analysis

{{ROOT_CAUSE_ANALYSIS}}

## Suggested Fix

{{SUGGESTED_FIX}}

## Acceptance Criteria

- [ ] Workflow succeeds reliably (>95% success rate)
- [ ] Flaky test quarantined or fixed
- [ ] Retry logic added if appropriate

---
*Created by workflow-monitor*

Enhancement Template

## Background

Efficiency improvement opportunity detected on {{DATE}}.

**Source:** {{WORKFLOW_NAME}} session {{SESSION_ID}}
**Efficiency score:** {{EFFICIENCY_SCORE}}

## Opportunity

{{ENHANCEMENT_DESCRIPTION}}

**Current behavior:**
{{CURRENT_BEHAVIOR}}

**Suggested improvement:**
{{SUGGESTED_IMPROVEMENT}}

## Impact

- Time saved: {{TIME_ESTIMATE}}
- Context saved: {{CONTEXT_ESTIMATE}}
- Complexity reduction: {{COMPLEXITY_NOTES}}

## Acceptance Criteria

- [ ] Improved behavior implemented
- [ ] No regression in functionality
- [ ] Documentation updated

---
*Created by workflow-monitor*

Template Variables

Usage

def render_template(template: str, variables: dict) -> str:
    """Replace template variables with values."""
    result = template
    for key, value in variables.items():
        placeholder = f"{{{{{key}}}}}"
        result = result.replace(placeholder, str(value))
    return result

Duplicate Detection

Before creating an issue, check for duplicates:

# Search for similar issues
gh issue list --state all --search "{{COMMAND}} OR {{ERROR_EXCERPT}}" --json title,number,state

# Check if recent issue exists
gh issue list --state open --label workflow --json title,createdAt | \
  jq '[.[] | select(.createdAt > "'"$(date -d '7 days ago' -Iseconds)"'")]'

Rate Limiting

• Maximum 5 issues per session
• Minimum 10 minutes between issue creation
• Require user confirmation unless auto_create_issues: true