Data Manager

Data Manager

Overview

Manage data programs, governance operations, and data reliability. This skill covers data roadmaps, stakeholder coordination, metadata stewardship, lifecycle management, monitoring, incident response, capacity planning, and SLA frameworks.

Features

• Data roadmap planning with stakeholder alignment and delivery cadence
• Governance operations: stewardship, access reviews, lifecycle enforcement
• Data ops monitoring with incident response and escalation paths
• Team KPI/SLA scorecards and operational metrics
• Cross-functional coordination across engineers, analysts, scientists, and legal

Usage

1. Identify the user's data management need (roadmap, governance, ops, or coordination) 2. Follow the corresponding workflow below 3. Produce structured outputs: roadmaps, governance policies, incident reports, or KPI dashboards

Examples

• User: "Create a data team roadmap"

Agent: Runs Program Management workflow, produces quarterly roadmap with initiatives, dependencies, and stakeholder sign-offs

• User: "Set up data governance"

Agent: Runs Governance Operations workflow, defines stewardship roles, access review cadence, and lifecycle policies

• User: "Handle a data incident"

Agent: Runs Data Ops workflow, triages severity, executes runbook, produces post-incident report with action items

When to Use

• Own the data roadmap, stakeholder reviews, and data product delivery cadence
• Run governance operations (stewardship, access reviews, lifecycle enforcement)
• Establish data ops monitoring, incident response, and team KPI/SLA scorecards
• Coordinate engineers, analysts, scientists, and legal on cross-functional data work

When NOT to Use

• Deep platform architecture ADRs or ontology design → use data-architect or ontology-engineer
• Hands-on warehouse SQL optimization or SCD modeling → use data-warehouse-engineer
• ML experimentation, model evaluation, or MLOps deployment → use data-scientist
• Cloud VPC, Kubernetes, or IaC provisioning → use infrastructure-engineer
• Company-wide multi-team technical programs (non-data) → use technical-program-manager

Core Workflows

1. Data Program & Product Management

Responsibilities:

• Own the data roadmap aligned to business outcomes
• Translate stakeholder needs into data product requirements
• Coordinate cross-functional data work (engineers, analysts, scientists, legal)

Operational cadence:

Data product delivery checklist: 1. Define the business question and success criteria 2. Identify data sources and validate availability/quality 3. Design the data model (see data-architect skill) 4. Build with observability (logging, lineage, tests) 5. Validate with stakeholders before GA 6. Document and train consumers 7. Monitor usage and iterate

2. Governance Operations Execution

Core activities:

Escalation paths:

• Data incident → On-call engineer → Team lead → Director
• Quality breach → Data steward → Governance committee → CDO
• Access violation → Security team → Legal (if PII exposure)

3. Data Operations & Reliability

Monitoring stack:

Incident response phases: 1. Detect: Alert fires or user reports issue 2. Triage: Assess severity (P1-P4), assign owner 3. Mitigate: Stop bleeding (rollback, redirect traffic) 4. Resolve: Root cause fix deployed 5. Review: Post-mortem within 48 hours for P1-P2

4. Metrics & SLA Framework

Data team KPIs:

SLA tiers:

Data Operations & Reliability

Monitoring & Alerting

Key Metrics by System

Data warehouse:

Data pipelines:

Data quality:

Incident Response

Severity Definitions

Incident Response Runbook

Step 1: Detect & Triage (0-15 min)

• Acknowledge alert in PagerDuty/Opsgenie
• Assess severity using definitions above
• Create incident channel (Slack) or bridge
• Notify stakeholders if P1-P2

Step 2: Mitigate (15 min - 2 hours)

• Identify scope: which pipelines, tables, consumers affected
• Apply temporary fix: rollback, restart, manual backfill
• Do NOT root cause during mitigation
• Update status page if customer-facing

Step 3: Resolve (2-24 hours)

• Deploy permanent fix
• Verify fix with tests and monitoring
• Confirm data quality post-fix
• Close incident when stable for >1 hour

Step 4: Review (24-48 hours)

• Schedule post-mortem for P1-P2
• Document timeline, root cause, remediation
• Identify action items with owners and dates
• Share broadly; blameless culture

Backup & Recovery

Backup Strategy

Backup verification:

• Monthly restore test on non-production
• Validate data integrity post-restore
• Document restore procedures
• Test cross-region recovery annually

Disaster Recovery Patterns

Active-Passive:

• Primary region active, secondary on standby
• Failover: Manual or automated
• Cost: Lower (secondary scaled down)
• RTO: Hours

Active-Active:

• Both regions serving traffic
• Automatic failover
• Cost: Higher (2x compute)
• RTO: Minutes

Capacity Planning

Planning Cycle

Monthly:

• Review utilization trends (CPU, storage, memory)
• Identify bottlenecks
• Plan for known upcoming workloads

Quarterly:

• Project growth based on business forecasts
• Evaluate reserved capacity / committed use discounts
• Right-size infrastructure (scale up or down)

Annually:

• Major architecture review
• Evaluate new technologies
• Budget planning for next fiscal year

Capacity Triggers

Change Management

Change Types

Change Request Template

## Change Request

**Title:** __________
**Type:** Standard / Normal / Emergency
**Risk:** Low / Medium / High
**Rollback plan:** __________

**Description:**
What is changing and why?

**Impact:**
- Systems affected: __________
- Downtime expected: __________
- Consumers notified: __________

**Testing:**
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Staging validation complete
- [ ] Rollback tested

**Approvals:**
- [ ] Technical review
- [ ] Stakeholder sign-off
- [ ] Scheduled for: __________

On-Call Operations

On-Call Rotation

• Primary + secondary (overlapping)
• Weekly rotations preferred over daily
• Handoff: Document open issues, known risks
• Escalation: After 15 min unresolved → secondary → manager

On-Call Checklist (Shift Start)

• [ ] Review open incidents from previous shift
• [ ] Check alert backlog (clear false positives)
• [ ] Review scheduled changes for shift
• [ ] Confirm runbook access and tooling
• [ ] Check capacity dashboard for anomalies

Governance Operations

Data Stewardship Models

Centralized Stewardship

• One team owns all metadata and quality
• Best for: Small orgs, strong governance need
• Risk: Bottleneck, limited domain expertise

Federated Stewardship

• Domain owners are stewards for their data
• Central team sets standards and tooling
• Best for: Data mesh, large enterprises
• Risk: Inconsistent quality across domains

Hybrid Stewardship

• Central stewards for shared/reference data
• Domain stewards for domain-specific data
• Best for: Most medium-to-large orgs

Metadata Management

Required Metadata Fields

Catalog Enrichment Workflow

1. Auto-ingest technical metadata (schemas, types, stats) 2. Assign owners (auto-detect from query patterns or manual) 3. Tag with business glossary terms 4. Document descriptions and use cases 5. Validate completeness quarterly

Access Review Process

Quarterly Access Review

Step-by-step: 1. Export current access matrix (roles, users, tables) 2. Notify owners: "Review access by [date]" 3. Owners confirm or revoke per user 4. Security team audits high-risk access (PII, admin) 5. Apply changes and log in audit trail 6. Report completion to compliance

Access review template:

Data Lifecycle Management

Lifecycle Stages

Retention Policy Enforcement

Automation rules:

• Partitioned tables: Drop partitions older than retention
• Object storage: Lifecycle policies (S3/GCS)
• Logs: Aggregate then delete raw after 90 days
• PII: Anonymize after consent withdrawal or expiration

Legal hold override:

• Legal team can flag datasets for indefinite retention
• Hold metadata stored in catalog
• Notify operations before any deletion

Governance Committee

Charter Template

Purpose: Oversight of data policies, standards, and issue resolution

Members:

• Chair: CDO or VP Data
• Members: Data engineering lead, analytics lead, legal, security, business domain reps
• Observer: Compliance officer

Meeting cadence: Monthly Quorum: 50% of voting members

Agenda Template

1. Action items from last meeting (5 min)
2. Policy proposals (20 min)
   - New data classification scheme
   - Access policy update
3. Quality scorecard review (15 min)
   - Trending issues
   - Remediation status
4. Incident review (15 min)
   - Recent data breaches/quality failures
5. New business (5 min)

Policy Documentation

Required Policies

Policy template structure:

# [Policy Name]

## Purpose
Why this policy exists

## Scope
Who and what it applies to

## Policy Statement
The rule

## Roles & Responsibilities
Who does what

## Enforcement
Consequences of violation

## Exceptions
How to request exceptions

## Review
When and how policy is reviewed

Metrics & SLA Framework

Data Quality Scorecard

Dimensions & Weights

Overall quality score = weighted average of dimension scores

Scorecard Template

## Data Quality Scorecard — [Month]

### Overall Score: 94/100 (-2 from last month)

| Dataset | Completeness | Uniqueness | Validity | Timeliness | Consistency | Score |
|---|---|---|---|---|---|---|
| f_orders | 99% | 100% | 97% | 92% | 99% | 97 |
| d_customer | 98% | 99.9% | 96% | 100% | 98% | 98 |
| f_inventory | 95% | 100% | 99% | 85% | 99% | 94 |

### Issues
1. f_orders timeliness: Shopify API delay caused 8% late loads
   - Owner: @data-eng-shopify
   - Action: Implement retry with exponential backoff
   - Due: [date]

2. d_customer validity: 4% invalid phone numbers
   - Owner: @data-steward-customer
   - Action: Update regex validation rule
   - Due: [date]

SLA Framework

SLA Definitions

Availability: % of time system is operational

• Formula: (Total time - Downtime) / Total time × 100
• Target: 99.9% (8.76 hours downtime/year)

Freshness: Time from source event to availability for query

• Formula: MAX(event_time) - NOW() for each dataset
• Tier 1: <1 hour, Tier 2: <4 hours, Tier 3: <24 hours

Latency: Query response time

• P50 (median) and P95 targets
• Dashboard queries: P95 <5 seconds
• Ad-hoc queries: P95 <30 seconds

Accuracy: % of correct data (vs source of truth)

• Measured by sampling and reconciliation
• Target: 99.5% for Tier 1, 99% for Tier 2

SLA Monitoring Dashboard

## SLA Dashboard — [Week of]

### Availability
| System | Uptime | Incidents | Status |
|---|---|---|---|
| Snowflake | 99.95% | 0 | 🟢 On track |
| Airflow | 99.2% | 1 (P3) | 🟡 At risk |

### Freshness
| Dataset | SLA | Actual | Breaches | Status |
|---|---|---|---|---|
| f_orders | <1h | 45 min | 0 | 🟢 On track |
| d_customer | <4h | 6h | 2 | 🔴 Breached |

### Latency
| Query Type | P50 | P95 | Target P95 | Status |
|---|---|---|---|---|
| Dashboard | 1.2s | 3.5s | <5s | 🟢 On track |
| Ad-hoc | 5s | 25s | <30s | 🟢 On track |

Team Productivity Metrics

Engineering Metrics (DORA-inspired)

Analyst Productivity

Operational Metrics

Cost Management Metrics

Cost Per Unit

Cost Attribution

Tagging strategy:

• Environment: prod, staging, dev
• Team: analytics, engineering, science
• Product: revenue, marketing, operations
• Project: feature-name, migration

Chargeback model options: 1. Showback: Visibility only (recommended starting point) 2. Chargeback: Teams billed for their usage 3. Hybrid: Base allocation + overage chargeback

Reporting Cadence

Data Program & Product Management

Roadmap Framework

Horizon Planning

Prioritization Frameworks

RICE (for product features):

Score = (Reach × Impact × Confidence) / Effort

WSJF (Weighted Shortest Job First):

WSJF = (Business value + Time criticality + Risk reduction) / Job size

MoSCoW (for stakeholder alignment):

• Must have: Non-negotiable for release
• Should have: Important but not blocking
• Could have: Nice to have if capacity allows
• Won't have: Explicitly deferred

Stakeholder Management

Stakeholder Matrix

Communication Templates

Weekly status email:

1. This week: What shipped (with links)
2. Next week: What's committed
3. Blockers: What needs leadership help
4. Metrics: Pipeline health, quality scores
5. Shoutouts: Recognize team wins

Quarterly business review:

1. Goals vs outcomes
2. Key deliverables (demos)
3. Metrics trend (6 months)
4. Challenges & learnings
5. Next quarter priorities
6. Ask for feedback/decisions

Data Product Management

Product Lifecycle

1. Discovery: Interview stakeholders, define the question 2. Definition: Write PRD with acceptance criteria 3. Design: Data model, pipeline architecture, UX (dashboards) 4. Build: Iterative delivery with demos 5. Launch: Staged rollout (alpha → beta → GA) 6. Operate: Monitor, support, iterate 7. Sunset: Deprecate when obsolete (see EOL communication)

PRD Template (Data Product)

# [Product Name] PRD

## Problem
What business question are we answering?

## Success Criteria
- Metric: __________
- Baseline: __________
- Target: __________
- Timeline: __________

## Data Requirements
- Sources: __________
- Granularity: __________
- History needed: __________
- Refresh frequency: __________

## Dependencies
- Upstream: __________
- Downstream: __________
- Blockers: __________

## Acceptance Criteria
- [ ] Data passes quality tests
- [ ] Dashboard loads in <5 seconds
- [ ] Documentation complete
- [ ] Stakeholder sign-off received

Team Coordination

RACI for Data Work

Meeting Guidelines

Effective data standup (15 min):

• What did I complete yesterday?
• What am I working on today?
• What's blocking me?
• No problem-solving in standup (schedule breakout)

Sprint demo (30 min):

• Demo working data, not slides
• Show the business impact
• Record for async stakeholders