Ml Research Engineer Safeguards

ML / Research Engineer, Safeguards

When to Use

• Define research questions on harm detection, jailbreak resistance, or policy categories
• Curate or audit safety datasets — labeling guidelines, bias checks, version control
• Train or fine-tune classifiers, rankers, or small LLM judges for moderation
• Design benchmarks and eval suites — golden sets, adversarial slices, regression harnesses
• Run ablations — architecture, threshold, data mix, ensemble vs single model
• Analyze metrics — precision/recall, calibration, false positive/negative slices
• Write research memos — methods, results, limitations, production recommendation
• Specify promotion bar for a new safeguard model version

When NOT to Use

• Deploy gateways, GPU serving, canary routing → ml-infrastructure-engineer-safeguards
• Execute structured red-team engagements on prod → ai-redteam
• Draft acceptable-use policy or risk tiers → ai-risk-governance
• Build customer-facing RAG/agents → ai-engineer
• General literature survey unrelated to safety → ai-researcher
• Token/context compression research → research-engineer-scientist-tokens
• Product A/B and business metrics → data-scientist
• PII detection benchmarks, memorization, logging minimization → privacy-research-engineer-safeguards

Related skills

Core Workflows

1. Research framing (safety)

Hypotheses, harm taxonomy, success metrics.

See `references/research_framing_safety.md`.

2. Benchmarks and datasets

Golden sets, labeling, versioning.

See `references/safety_benchmarks_datasets.md`.

3. Model development

Training, fine-tuning, ensembles.

See `references/classifier_model_development.md`.

4. Evaluation and metrics

Slices, calibration, error analysis.

See `references/evaluation_metrics_analysis.md`.

5. Ablations and experiments

Controls, reproducibility.

See `references/ablation_experiment_design.md`.

6. Handoff to production

Promotion criteria, monitoring hooks.

See `references/research_to_production_handoff.md`.

Outputs

• Research brief — question, baseline, hypothesis, metrics
• Dataset card — sources, label schema, known limitations
• Benchmark spec — cases, categories, pass/fail rubric
• Results table — metrics by slice with confidence intervals where possible
• Error analysis — representative FP/FN clusters
• Promotion recommendation — go/no-go vs current production classifier

Principles

• Measure what policy cares about — category-level recall on high-severity harms
• Report failures honestly — FPs hurt UX; FNs hurt safety
• Hold out adversarial refresh — do not train on the only test set
• Reproducible — seeds, data version, model hash, eval script
• Separate research from ops — research proves lift; infra ships it

Ablation and experiment design

Table of contents

1. Controls 2. Ablation matrix 3. Reproducibility 4. Stopping rules

Controls

One change at a time unless factorial design is intentional:

Fixed across ablations: eval sets, random seed policy, hardware class.

Ablation matrix

Example table for memo:

Mark Pareto improvements — better safety metric without latency regression.

Reproducibility

Checklist:

• [ ] Data version pinned
• [ ] Code commit hash
• [ ] Training config YAML archived
• [ ] Eval script version
• [ ] Model artifact checksum
• [ ] Results JSON in artifact store

Enable another researcher to re-run eval without retraining.

Stopping rules

Pre-register:

• Minimum lift on primary metric to pursue promotion
• Maximum acceptable FP regression on benign hard set
• Maximum latency increase for infra

Stop early if critical FN appears on golden set in mid-training checkpoint.

Classifier model development

Table of contents

1. Model families 2. Training workflow 3. LLM-as-judge 4. Ensembles 5. Constraints

Model families

Pick based on latency budget from ml-infrastructure-engineer-safeguards.

Training workflow

1. Split data — stratified by category; no user leakage across splits 2. Train with class weights if imbalance 3. Validate on held-out set + frozen golden benchmark 4. Calibrate thresholds on validation only 5. Export artifact — weights, tokenizer, config, eval report

Log experiment: data version, hyperparameters, git SHA, metrics.

LLM-as-judge

When using LLM judges:

• Fixed rubric prompt version
• Temperature 0 for scoring
• Parse structured output (JSON schema)
• Measure judge stability — repeat labels on subset
• Cost and latency estimate for prod path

Not a substitute for human eval on high-severity launches.

Ensembles

• Vote or score fusion across models
• Cascade: cheap model first, expensive on uncertain band
• Document uncertainty band — route to human review

Ablate ensemble vs single model on same eval (ablation_experiment_design.md).

Constraints

• No training on test-only adversarial sets
• PII handling in training pipelines
• Model size limits for edge deployment
• Document failure modes for model card (ai-risk-governance)

Evaluation and metrics analysis

Table of contents

1. Confusion matrix by category 2. Slice analysis 3. Calibration 4. Error analysis 5. Statistical caution

Confusion matrix by category

Report per category:

Highlight FN on critical categories first in executive summary.

Slice analysis

Mandatory slices where data allows:

• Language / locale
• Attack type (direct, encoded, indirect injection)
• Prompt vs response classification
• Tenant tier (consumer vs enterprise)
• Content length buckets

Flag regressions on any slice even if global metric improves.

Calibration

• Reliability diagram or ECE on validation
• Choose threshold per category to hit target FP or FN rate
• Document sensitivity — small τ change → large block rate swing

Provide recommended τ table for infra config.

Error analysis

Sample 20–50 each:

• False positives — benign blocked; cluster themes
• False negatives — harmful missed; attack pattern

Qualitative tags feed next data collection and ai-redteam focus areas.

Statistical caution

• Report confidence intervals on small sets
• Multiple comparisons — avoid cherry-picked slices
• Do not claim significance without adequate N
• Compare models on same eval version

For classical tests on proportions → coordinate with data-scientist if needed.

Research framing (safety)

Table of contents

1. Harm taxonomy 2. Hypothesis templates 3. Metrics selection 4. Baselines

Harm taxonomy

Align categories with policy (ai-risk-governance) but own the ML mapping:

Document overlap rules — when multiple labels apply, primary label for metrics.

Hypothesis templates

• "Fine-tuning on dataset X improves recall on category Y at fixed FP rate vs baseline B"
• "Ensemble of small classifier + LLM judge reduces FNs on jailbreak slice Z vs classifier alone"
• "Threshold τ calibrated on validation minimizes cost function C(FP, FN) for tier T"

Each hypothesis must be falsifiable with a pre-registered eval set.

Metrics selection

Define primary and guardrail metrics before running experiments.

Baselines

Always compare against:

1. Current production classifier (same eval set) 2. Simple baseline — keyword, regex, or vendor API 3. Previous research champion — last promoted model

Note data leakage if baseline was trained on overlapping data.

Research to production handoff

Table of contents

1. Promotion criteria 2. Deliverables package 3. Shadow and canary 4. Post-launch monitoring

Promotion criteria

Minimum bar (tune with ai-risk-governance and ai-lead-ops):

No promotion on val-set overfit alone.

Deliverables package

Hand to ml-infrastructure-engineer-safeguards:

• Model artifact URI and format (ONNX, Torch, API-only)
• Config: thresholds, category mapping, model ID
• Container/resource requirements
• Eval report JSON and summary memo
• Known limitations and do-not-deploy contexts

Shadow and canary

Research supports but does not operate prod:

1. Shadow mode — log scores, no user impact; compare distributions 2. Canary — infra owns traffic %; research monitors slice metrics 3. Rollback trigger — pre-agreed block-rate and error bounds

Attend launch war room; on-call for model behavior questions first 48h.

Post-launch monitoring

Define research-owned weekly checks first month:

• Slice metrics vs shadow baseline
• New FN/FP samples from production sampling
• Drift signals — score distribution shift
• Retrain trigger when attack surface shifts

Feed findings into next benchmark version and red-team backlog.

Safety benchmarks and datasets

Table of contents

1. Benchmark types 2. Dataset curation 3. Labeling 4. Versioning

Benchmark types

Keep golden set frozen between model iterations; refresh on schedule with governance review.

Dataset curation

Sources (document provenance and license):

• Internal production samples (redacted, sampled, consent)
• Synthetic generation with human review
• Public safety datasets (check license and bias)
• Red-team harvests (separate train/holdout splits)

De-duplication and near-duplicate detection before train/test split.

Labeling

• Guideline doc — definitions, examples, borderline cases
• Double-label subset; compute inter-annotator agreement (Cohen's κ)
• Escalation queue for disagreements
• Avoid single annotator on severity-critical labels

Track demographic and language representation — report slice gaps.

Versioning

Dataset card fields:

• dataset_id, version, date
• Size per category and language
• Label schema hash
• Known limitations and toxic content warnings for handlers
• Train/val/test split policy

Never bump version without changelog — downstream benchmarks depend on it.