Data System Ops Lead

Data System Operations Lead

Overview

Run data system operations and reliability engineering. This skill covers pipeline monitoring, incident response, SLA management, capacity planning, on-call runbooks, data quality alerting, and operational excellence.

Features

• Pipeline monitoring with alerting thresholds and dashboard design
• Incident response: severity classification, escalation paths, post-incident reviews
• SLA management with performance tracking and breach prevention
• Capacity planning: resource forecasting, scaling triggers, cost optimization
• On-call runbooks with step-by-step troubleshooting procedures
• Data quality alerting with anomaly detection and validation rules

Usage

1. Identify the user's data ops need (monitoring, incident response, SLA, capacity, or runbooks) 2. Follow the corresponding workflow below 3. Produce structured outputs: monitoring dashboards, incident reports, SLA scorecards, or runbooks

Examples

• User: "Set up pipeline monitoring"

Agent: Runs Monitoring workflow, defines alert thresholds, creates dashboard with latency/throughput/error rate metrics

• User: "Handle a pipeline failure"

Agent: Runs Incident Response workflow, classifies severity, executes runbook, produces post-incident report with root cause

• User: "Create an on-call runbook"

Agent: Runs Runbook Creation workflow, documents troubleshooting steps, escalation paths, and verification checks

When to Use

• Running daily data platform health checks, pipeline triage, and SLA enforcement
• Leading incident response, post-incident reviews, and on-call/shift handoffs
• Managing vendor escalations, cost reviews, and operational process design
• Coaching ops engineers on runbooks, capacity, and alert hygiene

When NOT to Use

• Strategic data mesh/lakehouse architecture or governance policy → use data-architect
• Program roadmaps, quarterly planning, or governance committee operations → use data-manager
• Writing dimensional models or MERGE-based incremental ETL → use data-warehouse-engineer
• General cloud/Kubernetes infrastructure outside the data platform → use infrastructure-engineer
• Org-wide on-call program, SEV definitions, postmortem tooling → use incident-management-engineer

Core Workflows

1. Platform Operations Oversight

Daily operational cadence:

Health check checklist:

• [ ] All critical pipelines completed successfully
• [ ] Data freshness within SLA thresholds
• [ ] No critical or high alerts active >30 min
• [ ] Storage utilization <85%
• [ ] Query performance within baseline
• [ ] Backup jobs completed

2. Incident & Problem Management

Incident severity matrix:

Incident lifecycle: 1. Detect → 2. Triage → 3. Mitigate → 4. Resolve → 5. Review

Post-incident review (within 48 hours for P1-P2):

• Timeline of events
• Root cause (5 Whys)
• Impact assessment
• Action items with owners and dates
• Process improvements

3. Team & Shift Leadership

On-call rotation design:

• Primary + secondary (overlapping coverage)
• Weekly rotations (not daily — too disruptive)
• Include weekend coverage in planning
• Escalation path: Engineer → Lead → Manager → Director

Shift handoff template:

## Shift Handoff — [Date] [Shift]

### Active Incidents
| ID | Severity | Status | Owner | Notes |
|---|---|---|---|---|
| INC-001 | P2 | Mitigated | @alice | Awaiting permanent fix |

### Alerts Requiring Attention
- [ ] Storage forecast will hit 90% in 3 days

### Changes Deployed
- [ ] Pipeline X updated to v2.1 (stable)

### Planned Work Next Shift
- [ ] Apply security patches to warehouse

### Issues for Lead Attention
- Recurring alert on pipeline Y — may need threshold tuning

4. Vendor & Cost Management

Monthly cost review:

• Actual vs budgeted spend
• Cost per TB processed, per pipeline run
• Identify optimization opportunities
• Vendor contract renewal timeline

Vendor escalation path: 1. Technical support (standard ticket) 2. Account manager (business impact) 3. Executive escalation (contract-level)

Operational Processes

Change Management

Change Types

Change Request Template

## Change Request

**Title:** [Brief description]
**Type:** Standard / Normal / Emergency / Maintenance
**Risk:** Low / Medium / High
**Rollback plan:** [Steps to undo]

**Description:**
What is changing and why?

**Impact:**
- Systems: [affected systems]
- Downtime: [expected duration or none]
- Users: [who is affected]

**Testing:**
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Staging validation complete
- [ ] Rollback tested

**Schedule:**
- Proposed window: [date/time]
- Duration: [expected]

**Approvals:**
- [ ] Technical review: [name]
- [ ] Stakeholder sign-off: [name]

Maintenance Windows

• Schedule regularly (e.g., monthly Tuesday 02:00 UTC)
• Announce 1 week ahead
• Have rollback plan ready
• Post-results summary within 4 hours

Deployment Procedures

Deployment Checklist

• [ ] Change request approved
• [ ] Staging tests passed
• [ ] Runbook updated for new behavior
• [ ] Monitoring dashboards verified
• [ ] On-call team notified
• [ ] Rollback artifact ready
• [ ] Communication sent to stakeholders
• [ ] Post-deployment validation completed

Blue-Green Deployment (Data Systems)

1. Deploy new version to green environment 2. Run parallel validation on green 3. Switch traffic/router to green 4. Monitor for 1 hour 5. Decommission blue (or keep for quick rollback)

Canary Deployment

1. Deploy to 5% of pipelines/workload 2. Monitor metrics vs baseline 3. Gradually increase to 25%, 50%, 100% 4. Roll back immediately on anomaly detection

Problem Management

Problem vs Incident

• Incident: Single occurrence of service disruption
• Problem: Underlying cause that may cause multiple incidents

Problem Lifecycle

1. Identification: Trend analysis, incident correlation 2. Recording: Document in problem ticket 3. Investigation: Root cause analysis 4. Workaround: Temporary fix to reduce impact 5. Resolution: Permanent fix 6. Closure: Verify fix, update documentation

Trend Analysis

Review weekly:

• Recurring alerts (same alert >3 times in 7 days)
• Recurring incidents (same component, same symptoms)
• MTTR trends (increasing = skills or process gap)
• False positive rate (high = threshold tuning needed)

Operational Reviews

Weekly Operational Review

## Ops Review — Week of [Date]

### Incidents
| ID | Severity | MTTR | Root Cause | Recurring? |
|---|---|---|---|---|
| INC-001 | P2 | 45 min | Disk full | Yes (3rd time) |

### Alerts
| Alert | Count | False Positive? | Action |
|---|---|---|---|
| Pipeline lag | 12 | 8 | Tune threshold |

### Changes
| Change | Status | Issues |
|---|---|---|
| DW upgrade | Successful | None |

### Capacity
| Resource | Utilization | Forecast |
|---|---|---|
| Storage | 82% | 90% in 14 days |

### Action Items
| Item | Owner | Due |
|---|---|---|
| Fix recurring disk issue | @alice | Friday |

Monthly Business Review

• SLA performance vs target (6-month trend)
• Cost analysis (actual vs budget, per-unit trends)
• Incident trends (severity distribution, MTTR)
• Team health (on-call burden, burnout indicators)
• Roadmap review (upcoming projects, capacity impact)

Process Improvement

Kaizen for Operations

• Weekly: One small improvement per engineer
• Monthly: Team retrospective with action items
• Quarterly: Major process review (SLA definitions, escalation paths)

Metrics for Improvement

• MTTR (Mean Time To Recovery): Target reduction 10% per quarter
• MTBF (Mean Time Between Failures): Target increase
• Change success rate: % of changes without rollback
• Automation coverage: % of manual tasks automated
• Runbook completeness: % of alerts with runbooks

Documentation Standards

Runbook Requirements

Every alert or recurring incident must have: 1. What triggers it 2. Immediate checks (commands, links) 3. Resolution steps (ordered, copy-paste where possible) 4. Escalation criteria and path 5. Post-resolution actions

Change Log

Document all changes in central log:

| Date | System | Change | Owner | Result |
|---|---|---|---|---|
| 2024-01-15 | Pipeline X | Added retry logic | @bob | Success |

Knowledge Base

• Searchable (not buried in chat)
• Updated within 24 hours of new incident resolution
• Peer-reviewed quarterly for accuracy

Platform Operations

Monitoring Stack

Metrics by Layer

Infrastructure:

Data warehouse:

Data pipelines:

Alerting Best Practices

• Alert on symptoms, not causes (when possible)
• Every alert must have a runbook link
• Page only for P1-P2; P3-P4 to ticket queue
• Threshold review monthly (reduce false positives)

SLA Framework

SLA Definitions

SLA Monitoring Dashboard

## SLA Dashboard — [Week]

### Availability
| System | Uptime | Incidents | Status |
|---|---|---|---|
| Data warehouse | 99.95% | 0 | On track |
| Pipeline platform | 99.2% | 1 (P3) | At risk |

### Freshness
| Dataset | SLA | Actual | Breaches |
|---|---|---|---|
| f_orders | <1h | 45 min | 0 |
| d_customer | <4h | 6h | 2 |

### Action Items
| Issue | Owner | Due | Status |
|---|---|---|---|
| Customer dim latency | @ops-team | Friday | In progress |

Incident Response Runbooks

P1 Response (Critical)

0-15 minutes: Detection & Triage 1. Acknowledge page/alert 2. Assess scope: which systems, users, data affected 3. Create war room (Slack/Zoom) with bridge number 4. Notify stakeholders (status page, executive if needed)

15-60 minutes: Mitigation 1. Identify root cause (don't fix yet, just understand) 2. Apply temporary fix (rollback, redirect, scale up) 3. Confirm mitigation with monitoring 4. Communicate status every 15 minutes

1-4 hours: Resolution 1. Deploy permanent fix or plan for later 2. Verify all systems stable 3. Close incident when stable for >1 hour 4. Schedule post-mortem within 48 hours

Runbook Template

# [System/Alert Name] Runbook

## Alert
[What triggers this alert]

## Symptoms
- [ ] Symptom 1
- [ ] Symptom 2

## Immediate Checks
1. Check dashboard: [link]
2. Check logs: [query/link]
3. Check recent changes: [deployment log]

## Resolution Steps
1. [Step 1 with command]
2. [Step 2 with command]
3. Verify: [how to confirm]

## Escalation
If unresolved in [X] minutes:
- Escalate to: [person/team]
- With: [what information to provide]

## Post-Resolution
- [ ] Update status page
- [ ] Document in incident log

Capacity Planning

Monthly Review

• Review utilization trends (CPU, storage, memory)
• Identify bottlenecks and forecast growth
• Plan for known upcoming workloads (launches, migrations)

Quarterly Planning

• Project growth based on business forecasts
• Evaluate reserved capacity / committed use discounts
• Right-size infrastructure (scale up or down)

Capacity Triggers

Backup & Disaster Recovery

Backup Strategy

DR Patterns

• Active-Passive: Lower cost, hours RTO
• Active-Active: Higher cost, minutes RTO
• Pilot Light: Core always on, scale up on failover

Data Quality Monitoring

Automated Checks

Quality Scorecard

| Dataset | Completeness | Uniqueness | Validity | Timeliness | Score |
|---|---|---|---|---|---|
| f_orders | 99% | 100% | 97% | 92% | 97 |
| d_customer | 98% | 99.9% | 96% | 100% | 98 |

Team Leadership

On-Call Rotation Design

Rotation Models

Schedule Design Principles

• Minimum 1 week per rotation (daily switches are too disruptive)
• Avoid back-to-back rotations (recovery time)
• Include holidays in planning (no surprise coverage)
• Publish schedule 1 month ahead
• Allow swap requests with 48-hour notice

Escalation Path

L1: On-call engineer (first responder)
  → L2: On-call lead (after 15 min unresolved)
    → L3: Manager (after 30 min, P1-P2 only)
      → L4: Director (business-critical, P1 only)

Shift Handoff

Handoff Meeting (10 min)

1. Active incidents and status 2. Alerts requiring attention 3. Changes deployed and their impact 4. Planned work for next shift 5. Issues needing lead attention

Handoff Document Template

## Shift Handoff — [Date] [Outgoing → Incoming]

### Incidents
| ID | Severity | Status | Notes |
|---|---|---|---|
| INC-001 | P2 | Monitoring | Awaiting vendor response |

### Alerts
- Storage forecast: will hit 90% in 3 days

### Changes
- Pipeline X v2.1 deployed, stable

### Planned Work
- Security patches (approved, ready to apply)

### Lead Attention
- Recurring alert on pipeline Y — threshold may need tuning

Burnout Prevention

Warning Signs

• Increased incident response time
• Declining quality of runbook updates
• Reluctance to take on-call rotations
• Increased sick days after on-call weeks
• Escalation of previously handled incidents

Mitigation Strategies

• Limit consecutive on-call weeks to 1
• Provide comp time after weekend/holiday coverage
• Rotate difficult shifts (nights, holidays) fairly
• Ensure uninterrupted sleep (no pages 00:00-06:00 except P1)
• Regular 1:1 check-ins focused on well-being

Team Development

Skills Matrix

Development Plan

• Cross-training: Pair junior with senior on incidents
• Shadowing: New team member shadows on-call for 2 weeks before solo
• Rotation: Rotate incident commander role to build skills
• Training budget: Conferences, certifications (AWS, data platforms)

Hiring & Staffing

Team Size Formula

Minimum team size = (24/7 coverage needs) / (shift hours) + (buffer for PTO/training)

Example:
- Need 24/7 coverage = 168 hours/week
- Shift = 40 hours/week
- Minimum = 168/40 = 4.2 → 5 engineers
- Buffer (PTO, training, sick) = +2
- Total recommended = 7 engineers

Role Progression

Communication

Team Cadence

Stakeholder Reporting

Weekly status to leadership:

## Data Ops Status — Week of [Date]

### Health: 🟢 On Track / 🟡 At Risk / 🔴 Degraded

### Incidents
| Severity | Count | Notes |
|---|---|---|
| P1 | 0 | — |
| P2 | 1 | Storage lag, resolved |

### SLA Performance
| Tier | Target | Actual | Status |
|---|---|---|---|
| Tier 1 | 99.9% | 99.95% | ✅ |

### Cost
| Actual | Budget | Variance |
|---|---|---|
| $12K | $11K | +9% |

### Coming Up
- Security patch window: Thursday 02:00 UTC
- Q2 capacity review: next week

Vendor & Tool Management

Vendor Evaluation

Evaluation Criteria

Vendor Scorecard

| Vendor | Tech | Reliability | Cost | Security | Health | Ease | Total |
|---|---|---|---|---|---|---|---|
| Vendor A | 4 | 3 | 2 | 4 | 3 | 4 | 3.2 |
| Vendor B | 3 | 4 | 4 | 3 | 4 | 3 | 3.4 |

Due Diligence Checklist

• [ ] Financial statements (if critical vendor)
• [ ] Security questionnaire (SOC 2, ISO 27001)
• [ ] Reference customers (similar size/use case)
• [ ] Support SLA and escalation path
• [ ] Data residency and GDPR/CCPA compliance
• [ ] Exit strategy (data portability, format)

Cost Optimization

Cost Attribution

Tagging strategy:

• Environment: prod, staging, dev
• Team: analytics, engineering, science
• Project: feature-name, migration
• Data tier: tier1, tier2, tier3

Chargeback models: 1. Showback: Visibility only (recommended starting point) 2. Chargeback: Teams billed for usage 3. Hybrid: Base allocation + overage chargeback

Optimization Techniques

Monthly Cost Review

## Cost Review — [Month]

### Actual vs Budget
| Category | Budget | Actual | Variance |
|---|---|---|---|
| Compute | $5K | $5.5K | +10% |
| Storage | $3K | $2.8K | -7% |
| Egress | $1K | $1.2K | +20% |

### Per-Unit Costs
| Metric | This Month | Last Month | Trend |
|---|---|---|---|
| Cost per TB queried | $12 | $14 | ↓ |
| Cost per pipeline run | $0.05 | $0.06 | ↓ |

### Optimization Opportunities
1. [Opportunity] — Estimated savings: $X/month

### Contract Renewals (Next 6 Months)
| Vendor | Renewal Date | Current Spend | Action |
|---|---|---|---|
| Snowflake | March | $4K/mo | Evaluate alternatives |

Tool Selection

Monitoring & Alerting

Data Observability

Automation & Orchestration

Backup & DR

Contract Management

Key Terms to Negotiate

• Pricing: Per-unit vs flat rate, annual vs monthly
• SLA: Uptime guarantee, credit structure
• Support: Response time, escalation path, dedicated CSM
• Termination: Notice period, data return format
• Liability: Cap on damages, indemnification
• Auto-renewal: Opt-in vs opt-out

Renewal Timeline

• T-90 days: Evaluate current vendor performance
• T-60 days: Research alternatives, issue RFP if needed
• T-30 days: Negotiate terms, prepare fallback
• T-14 days: Final decision, notify stakeholders
• T-0: Execute or migrate

Automation Strategy

Automation Priorities

1. High volume + low complexity (first target)

• Example: Daily backup verification

2. Error-prone manual tasks

• Example: Schema drift detection

3. Time-sensitive responses

• Example: Auto-remediation for known issues

Automation Framework

Trigger → Validation → Action → Verification → Notification

Example: Auto-remediation

def auto_remediate_disk_full():
    if disk_usage > 90%:
        # Validate: check it's not a known growth pattern
        if growth_rate < threshold:
            # Action: clean old logs
            clean_old_logs(days=7)
            # Verification
            if disk_usage < 85%:
                notify("Auto-remediated disk full")
            else:
                page_oncall("Disk still full after cleanup")

Exit Strategy

Vendor Migration Planning

1. Inventory: All integrations, users, data formats 2. Alternative evaluation: At least 2 options 3. Migration plan: Timeline, data migration, testing 4. Parallel run: Run both systems for validation period 5. Cutover: Defined go/no-go criteria 6. Decommission: Secure data deletion, contract closure

Data Portability Checklist

• [ ] Export in open format (Parquet, CSV, JSON)
• [ ] Document schema and transformations
• [ ] Verify completeness of export
• [ ] Test import into alternative system
• [ ] Update all downstream consumers