Ml Infrastructure Engineer Safeguards

ML Infrastructure Engineer, Safeguards

When to Use

• Design inference gateway with safeguard stages (auth, rate limit, pre-filter, model, post-filter)
• Deploy model servers — GPU pools, replicas, autoscaling, health checks
• Operate moderation/classifier services (hosted or self-hosted) in production
• Configure policy runtime — thresholds, categories, block vs rewrite vs escalate
• Instrument safety metrics — block rate, false positive sampling, safety-path latency
• Roll out safeguard model versions — canary, rollback, config flags
• Plan capacity for safety + main model (queueing, shedding, degradation modes)
• Integrate human review queues and appeal flows at infrastructure boundary
• Debug production incidents — safety service down, filter bypass, p99 on guard path

When NOT to Use

• Draft AI acceptable-use policy or risk tiers → ai-risk-governance
• Implement product RAG, agents, or prompts → ai-engineer, prompt-engineer
• Run jailbreak/red-team engagements → ai-redteam
• Build generic developer portal or golden paths → platform-engineer
• SOC 2 control evidence mapping → compliance-engineer
• End-to-end commercial AI architecture ADRs → applied-ai-architect-commercial-enterprise
• AI ops cadence and vendor contracts → ai-lead-ops
• Classical ML experimentation and feature science → data-scientist

Related skills

Core Workflows

1. Inference and serving platform

Gateways, model servers, scaling.

See `references/inference_serving_platform.md`.

2. Safeguards request path

Pre/post filters, ordering, failure modes.

See `references/safeguards_request_path.md`.

3. Policy runtime enforcement

Thresholds, actions, human loop hooks.

See `references/policy_runtime_enforcement.md`.

4. Safety observability

Metrics, logs, sampling, privacy.

See `references/safety_observability.md`.

5. Model and config rollout

Versions, canaries, kill switches.

See `references/model_version_rollout.md`.

6. Capacity and reliability

GPU, degradation, DR.

See `references/capacity_reliability.md`.

Outputs

• Architecture diagram — request path with safeguard stages and dependencies
• Runbook — deploy, rollback, scale, incident playbooks for safety services
• Config spec — policy thresholds, model IDs, feature flags per environment
• SLO sheet — availability and p99 for gateway + each safety stage
• Capacity plan — GPU/CPU headroom, QPS limits, degradation matrix
• Rollout plan — canary criteria tied to block-rate and error budgets

Principles

• Fail closed for high-risk tiers — when safety path unavailable, default deny or safe response per policy
• Latency budgeted — safety is part of user-facing SLO, not best-effort
• Observable decisions — log decision codes, not raw harmful content in clear text
• Version everything — policy config, classifier weights, and gateway routing independently
• Test in prod-like path — staging must run full safeguard chain, not model-only

Capacity and reliability

Table of contents

1. Resource planning 2. Degradation modes 3. Multi-region 4. Dependencies

Resource planning

Estimate:

peak_qps × (tokens_in + tokens_out) × safety_multiplier

safety_multiplier accounts for extra model calls on pre/post path.

GPU:

• Model size → VRAM per replica
• Concurrent sessions vs batching
• Reserve headroom for traffic spikes and deploy surge

CPU classifiers: separate pool from GPU LLM to avoid starvation.

Align long-term GPU supply with data-center-compute-supply-efficiency when facility-bound.

Degradation modes

Ordered fallback (define per product tier):

1. Stricter rate limits 2. Smaller/cheaper model with same safeguard stack 3. Queue with timeout message 4. Read-only / cached responses where applicable 5. No unguarded LLM shortcut unless explicit break-glass with audit

Break-glass: dual control, time-limited flag, post-incident review.

Multi-region

• Policy and model versions consistent or explicitly documented per region
• Failover: DNS/anycast to healthy region; watch data residency
• Replicate classifier models; avoid cross-border PII for review queues

DR test: safety path included in game days, not model-only.

Dependencies

Maintain synthetic probes every minute through full path.

Performance tuning of hot paths → performance-engineer; own safeguard-specific SLOs here.

Inference and serving platform

Table of contents

1. Topology 2. Model servers 3. Gateway responsibilities 4. Scaling and health

Topology

Typical production stack:

Client → API gateway → Inference gateway → [safeguard stages] → Model server(s) → [post-safeguards] → Client

Separate control plane (config, routing tables) from data plane (request handling).

Document ownership per box: team, on-call, SLO.

Model servers

Requirements:

• Health: readiness when model loaded; liveness under load
• Graceful drain on deploy; in-flight request timeout
• Token/context limits enforced at gateway and server
• Secrets for weights/API keys via vault, not env in images

Hand off cluster bootstrap to cluster-deployment-engineer; own serving manifests and HPA rules here.

Gateway responsibilities

• Authentication, tenancy, rate limits (coordinate with product-infrastructure-security-engineer)
• Request ID propagation for tracing
• Routing by model ID, region, experiment flag
• Timeout and payload size limits
• Circuit breakers to downstream safety and model services

Keep gateway stateless; store session policy in external store if needed.

Scaling and health

• HPA on GPU utilization, queue depth, or custom metric (tokens/s)
• Cold start budget for scale-from-zero — often unacceptable for prod LLM
• Pre-warm replicas before traffic shift
• Load test full path including safeguards (performance-engineer)

Capacity signals: GPU memory headroom, batch queue length, OOM kills, throttle rate from vendor APIs.

Model and safeguard version rollout

Table of contents

1. Versioning model 2. Canary strategy 3. Rollback 4. Config-only changes

Versioning model

Independent versions:

• Main LLM weights or vendor model string
• Classifier/moderation models per stage
• Policy config (thresholds, categories)
• Gateway routing rules

Tag every response metadata: {policy_v, classifier_v, llm_v} for incident debug.

Canary strategy

1. Deploy new version to shadow (log decisions, do not enforce) OR 2. Traffic slice 1% → 5% → 25% → 100% with gates

Gate criteria examples:

• Block rate within ±X% of baseline on golden set
• No increase in safety-path 5xx
• p99 latency within budget
• Zero critical misses on mandatory ai-redteam suite for that release

Automate promotion; manual approve for high-risk tiers.

Rollback

• One-click revert routing to last known good triple (LLM + classifier + policy)
• Keep N-1 images/models warm for fast switch
• Document data compatibility — embedding index version if RAG adjacent

Post-rollback: incident timeline with version IDs.

Config-only changes

Threshold tweaks without model swap:

• Still require canary — small threshold moves change block rate sharply
• Diff config in PR; peer review with ai-risk-governance for regulated tiers
• Feature flags per tenant for enterprise exceptions

Never hot-patch prod without audit trail.

Policy runtime enforcement

Table of contents

1. Policy artifacts 2. Actions 3. Tiers and tenants 4. Human-in-the-loop

Policy artifacts

Versioned config (not hardcoded in app):

• Category thresholds and score → action mapping
• Model IDs per region and tier
• Refusal message templates (locale)
• Feature flags for experimental policies
• Allowlist/denylist references (synced from secure store)

Store in config service with audit log; promote dev → staging → prod.

Actions

Return machine-readable reason codes to clients (not internal policy names if sensitive).

Tiers and tenants

Enforce at gateway using claims + tenant metadata — single enforcement point.

Human-in-the-loop

Infrastructure hooks:

• Review queue — payload metadata + hashed content id
• SLA for reviewer turnaround; expiry → default action
• Audit trail: who approved, policy version, model version
• No PII in queue titles; encrypt at rest

Product workflow design may involve ai-risk-governance; own pipes and storage here.

Safeguards request path

Table of contents

1. Stage ordering 2. Pre-model filters 3. Post-model filters 4. Failure modes 5. Latency

Stage ordering

Default order (adapt per policy):

1. Auth / tenancy — who is calling 2. Rate limit / abuse — volumetric 3. Input normalization — encoding, length trim 4. Pre-classifiers — prompt injection, PII detect, policy categories 5. Main model — only if pre-stages pass or soft-flag path 6. Post-classifiers — toxic output, leakage, tool-output scan 7. Response shaping — redaction, refusal templates

Document short-circuit rules: which stage can block without calling model.

Pre-model filters

Run fast checks first (regex, small model) before large LLM call.

Post-model filters

• Harmful content, self-harm, illegal instructions (per policy)
• Data exfiltration patterns in output (secrets, system prompt leak)
• Tool call validation if agent path
• Structured output schema validation

On block: return safe completion with stable error code — not raw model output.

Failure modes

Align fail-open vs fail-closed with ai-risk-governance sign-off.

Latency

Allocate per-stage budget (e.g. pre 30ms, model 2s, post 50ms p99).

• Parallelize independent classifiers where safe
• Cache benign prompt hashes cautiously — risk of stale policy
• Async human review after synchronous path for non-blocking tiers

Track safety_path_ms separately from model_ms in traces.

Safety observability

Table of contents

1. Metrics 2. Logging 3. Sampling and review 4. Alerts

Metrics

Dashboards: exec summary + engineer drill-down by model/policy version.

Logging

• Log decision code, scores (bucketed), stage, latency — not full harmful text by default
• Hash or encrypt sensitive payloads in debug streams
• Retention aligned with privacy policy; separate security incident retention
• Correlate with trace_id across gateway → classifiers → model

Red-team repro artifacts → restricted access store, not general logs.

Sampling and review

• Random sample of allowed traffic for quality audit
• 100% capture of blocked metadata (not necessarily content)
• Feed samples to ai-redteam and policy owners on schedule
• Feedback loop: label → retrain threshold or model version

Alerts

Page when:

• Safety error rate > SLO
• Block rate spike/drop vs baseline (attack or broken classifier)
• Safety p99 regression breaking user SLO
• Fail-open path activated (if ever allowed)

Avoid alerting on raw block rate alone without baseline — seasonal traffic shifts.

Coordinate paging with devops and ai-lead-ops runbooks.