Product Management Human Data Platform

Product Management — Human Data Platform

When to Use

• Define vision, roadmap, and prioritization for labeling, RLHF, or human-eval products
• Write PRDs for annotation UI, project setup, QA, workforce, or export/API features
• Design annotation tasks (taxonomy, instructions, rubrics, edge cases)
• Specify quality programs: gold tasks, consensus, adjudication, rejection reasons
• Scope customer workflows (ML teams): projects, batches, SLAs, delivery formats
• Improve contributor/annotator productivity, fairness, and trust/safety product surfaces
• Set metrics: throughput, quality, cost per label, time-to-delivery, contributor retention
• Partner on privacy and ethics requirements for human-submitted data (PII, consent, locale)

When NOT to Use

• Facilitate generic process maps and BRDs without product ownership → business-analyst
• Wireframes and visual design only → product-designer
• RAG/copilot enterprise architecture → applied-ai-architect-commercial-enterprise
• Build eval harnesses and judges in code → prompt-engineer-agent-prompts-evals
• SOC/ISO evidence automation → compliance-engineer
• Data warehouse modeling → data-warehouse-engineer
• Cross-team delivery RAID without product discovery → technical-program-manager

Related skills

Core Workflows

1. Vision, roadmap, and prioritization

Outcomes, segments, themes, RICE/ICE.

See `references/roadmap_prioritization.md`.

2. Annotation task and taxonomy design

Instructions, rubrics, schema, edge cases.

See `references/annotation_task_design.md`.

3. Quality systems

Gold sets, IAA, adjudication, rejection taxonomy.

See `references/quality_systems.md`.

4. Customer (ML team) delivery

Projects, pipelines, exports, SLAs.

See `references/customer_ml_workflows.md`.

5. Contributor and workforce product

Task UX, payments, trust, locale.

See `references/contributor_workforce_product.md`.

6. Privacy, ethics, and policy

PII, consent, retention, labor.

See `references/privacy_ethics_policy.md`.

Output standards

• PRDs state persona, problem, success metrics, non-goals, and launch tier
• Task specs include worked examples (gold, borderline, reject)
• Quality bar defined as measurable thresholds, not "high quality"
• Every feature maps to cost, quality, or speed lever
• Escalate legal/labor questions; do not ship policy in product copy alone

When to load references

• Roadmap → references/roadmap_prioritization.md
• Tasks → references/annotation_task_design.md
• Quality → references/quality_systems.md
• Customers → references/customer_ml_workflows.md
• Contributors → references/contributor_workforce_product.md
• Privacy → references/privacy_ethics_policy.md

Annotation Task Design

Task design deliverables

Instruction quality checklist

• [ ] Objective defined in one sentence
• [ ] Definitions for every label with positive and negative examples
• [ ] Order of operations (e.g. "check safety before sentiment")
• [ ] What to do when unsure (skip, flag, best guess policy)
• [ ] Media-specific rules (audio clipping, occlusion, PII blur)
• [ ] Change log when taxonomy updates

Taxonomy rules

Pair with ontology-engineer when enterprise customers need shared ontologies across programs.

Modality considerations

Pre-labeling and automation

Document expected edit rate so ops can staff reviewers.

Versioning

• Task spec version tied to exported dataset metadata
• Breaking taxonomy changes require relabel or migrate plan
• Customer notification for downstream training impact

Review with design and eng

• product-designer: task UI affordances, error prevention
• Eng: latency, autosave, conflict on concurrent edit
• Ops: estimated handle time per item for pricing/SLA

Contributor and Workforce Product

Contributor jobs to be done

• Understand task quickly
• Complete work accurately with fair pay
• Know why work was rejected and how to improve
• Trust platform on payments, safety, and privacy

Core product surfaces

Fairness and gaming resistance

Workforce ops product hooks

• Capacity forecasts by locale and skill
• Surge pricing or bonus campaigns (policy-reviewed)
• Contributor blocking and reinstatement workflows
• Language and timezone coverage maps

Localization

• Instructions and UI in contributor language
• Locale-specific payment and labor disclosures
• Cultural nuance in rubrics (not literal translation only)

Metrics

RLHF / preference-specific UX

• Side-by-side comparison clarity
• Tie and "both bad" handling
• Rationale capture optional vs required (trade speed vs quality)
• Leakage prevention (hide model identity if blind eval required)

Ethics

• Avoid deceptive task framing about purpose of data use where regulated
• Limit exposure to traumatic content; rotation and wellness features
• Clear opt-out for sensitive categories

Legal review for labor classification and regional workforce law—not PM alone.

Customer ML Team Workflows

End-to-end journey

1. Scope — modality, volume, deadline, quality bar 2. Configure — project, taxonomy, workforce tier, instructions 3. Pilot — small batch, calibrate IAA and handle time 4. Scale — production batches, monitoring, SLA tracking 5. Accept — QA sign-off, export, lineage metadata 6. Iterate — taxonomy v2, active learning loops

Project setup (product requirements)

Self-serve vs managed

Product should not blur modes without clear pricing.

API and integration

• Batch create/upload, status webhooks, export download
• Idempotent job IDs for pipeline orchestration
• Rate limits and pagination documented
• Sandbox project for integration testing

Dashboards (customer)

• Progress: completed / in review / rejected
• Quality: gold accuracy, IAA, rework trend
• Cost: spend vs estimate (if usage-based)
• Blockers: queue starvation, instruction tickets

Delivery acceptance

## Acceptance checklist
- [ ] Volume and format match SOW
- [ ] Quality metrics meet appendix thresholds
- [ ] Label version and task spec hash in metadata
- [ ] Known limitations documented (classes with low support)
- [ ] PII attestation if applicable

Expansion and retention

• Track time-to-first-export for new accounts
• Instrument repeat project rate by vertical
• Capture quality incident root cause for product backlog

Handoffs

• Technical integration issues → support/engineering paths in org
• Legal/DPA → commercial-counsel + compliance-engineer
• Custom ontology → ontology-engineer

Privacy, Ethics, and Policy

Human data risks

Privacy-by-design features

• Role-based access (customer, contributor, reviewer, admin)
• Region pinning for storage and processing
• Retention TTL per project with legal hold override
• Download logging and watermarking on exports
• Contributor view: minimum data needed for task (masking)

Customer responsibilities (product enforces)

Before production labeling:
- [ ] Customer confirms rights to use data for labeling
- [ ] Classification selected (public / confidential / regulated)
- [ ] Retention and deletion requirements captured
- [ ] Banned content categories acknowledged

Ethics review triggers

Escalate product/legal review when:

• Biometric, medical, or children's data
• Deception studies or undisclosed recording
• Political, religious, or violence-heavy content at scale
• Surveillance or law-enforcement sensitive use cases
• Export to jurisdictions with conflicting rules

Alignment with compliance engineering

Map features to controls with compliance-engineer:

• Access reviews for admin roles
• Encryption at rest/transit
• Audit logs for label changes and exports
• DPIA inputs for new modalities or regions

Transparency

• Contributor: what data is used for (within legal bounds)
• Customer: how quality and workforce tiers work
• Document known limitations in release notes

Incident response (product angle)

Pair with community-executive-escalations-program-manager if public reputational escalation.

AI training downstream

• Metadata: label provenance, spec version, workforce tier
• Avoid implying labels are "ground truth" in API docs—document error bars
• Support customer deletion requests propagating to derived exports where feasible

Anti-patterns

• Shipping geo features without residency analysis
• Using contributor-generated content to train platform models without disclosed consent
• Hiding quality problems from customer dashboards

Quality Systems

Quality model

Raw labels → QA gates → Accepted dataset → Customer export
                ↑
         Gold / consensus / adjudication

Mechanisms

Metrics (define targets per project type)

Report confidence intervals on small samples.

Gold set program

• Size: enough power for weekly monitoring (often 1–5% injection)
• Refresh: rotate gold to prevent memorization
• Stratify: cover rare classes and edge cases
• Never use customer production data as gold without rights

Rejection taxonomy

Standardize reject reasons for product analytics:

- Instruction misunderstanding
- Taxonomy error
- Incomplete annotation
- Tooling bug
- Policy violation (PII, unsafe)
- Borderline / needs adjudication

Tiered workforce

Promotion rules must be transparent to contributors.

Customer-facing quality SLAs

Document in contract appendix:

• Minimum accuracy on gold (if customer supplies gold)
• Consensus configuration
• Escalation when IAA drops below threshold
• Rework turnaround time

Product features roadmap signals

• Spike in rework → instruction or UI issue
• IAA collapse on new class → taxonomy or training gap
• Gold accuracy drift → contributor gaming or spec ambiguity

Pair with eval teams

Export formats should support prompt-engineer-agent-prompts-evals golden sets and regression harnesses without manual reformatting.

Roadmap and Prioritization

Platform value levers

Personas (typical)

Roadmap themes (examples)

• Core labeling — task types, media support, shortcuts
• Quality — consensus, gold injection, analytics
• Delivery — exports, API, versioning, lineage metadata
• Workforce — skills tests, tiers, incentives, appeals
• Enterprise — SSO, VPC, private workforce, DPA features
• GenAI-era — RLHF pairs, rubric ranking, red-team data collection

Prioritization framework (RICE-lite)

Force-rank one primary lever per epic to avoid "everything is P0."

PRD skeleton

## Problem
## Personas
## Success metrics (baseline → target)
## Scope (in / out)
## User stories
## Task/quality implications
## Dependencies (eng, legal, ops)
## Launch plan (alpha → GA)
## Risks

Discovery inputs

• Customer QBR themes and churn reasons
• Contributor NPS and task-abandon funnels
• Quality regressions (IAA drops, rework rate)
• Competitive gaps (modalities, pricing model)
• Cost per label trend by project type

Anti-patterns

• Roadmap driven only by largest customer's custom ask without platform abstraction
• Features without quality metric definition
• Ignoring contributor experience when optimizing customer dashboards only