Ai Adversarial Robustness Engineer

AI Adversarial Robustness Engineer

When to Use

• Define threat models for evasion, poisoning, extraction, and inference attacks on ML/LLM systems
• Design robustness evaluation suites — ASR, perturbation budgets, slice metrics, regression harnesses
• Implement engineering defenses — adversarial training, input sanitization, detectors, ensembles
• Run lab/staging attack campaigns on model endpoints, APIs, or batch inference (authorized only)
• Audit training data and pipelines for poisoning, backdoors, and supply-chain tampering
• Specify production guardrails — input validation, output filtering, rate limits, anomaly monitors
• Compare certified vs empirical robustness claims and document limitations for stakeholders
• Investigate robustness regressions after model updates, fine-tunes, or data refreshes

When NOT to Use

• Broad LLM product red-team engagements, jailbreak policy, or ROE → ai-redteam
• AI governance, risk tiers, model cards, or compliance mapping → ai-risk-governance
• Safety classifier research, harm benchmarks, and moderation model training → ml-research-engineer-safeguards
• Safeguard gateways, GPU serving, canary routing, and inference SLOs → ml-infrastructure-engineer-safeguards
• PII, memorization, and privacy leakage research → privacy-research-engineer-safeguards
• Building production RAG/agents or LLM features → ai-engineer
• General literature survey without robustness scope → ai-researcher
• Network/web/AppSec penetration testing (non-model) → penetration-tester, web-pentester

Related skills

Core Workflows

1. Scope and threat model

1. Identify assets: weights, embeddings, training data, inference API, logs 2. Classify attacker goals and capabilities (white/gray/black box, budget, offline/online) 3. Map attacks to lifecycle stage (data, train, deploy, monitor) 4. Agree evaluation environment — no prod customer data without approval

See `references/adversarial_robustness_scope.md`.

2. Attack taxonomy and scenarios

Document evasion, poisoning, extraction, and inference threats with realistic preconditions.

See `references/threat_models_and_attack_taxonomy.md`.

3. Metrics and benchmarks

Select perturbation norms, ASR definitions, slices, and baselines; pre-register pass/fail gates.

See `references/evaluation_metrics_and_benchmarks.md`.

4. Defenses and mitigations

Choose layered controls: robust training, preprocessing, detectors, ensembles, and operational limits.

See `references/defenses_and_mitigations.md`.

5. Red-team campaigns on models

Plan authorized attacks in lab/staging; capture reproduction packages and severity.

See `references/red_team_campaigns_on_models.md`.

6. Production guardrails and monitoring

Translate findings into input/output policies, drift monitors, and incident playbooks.

See `references/production_guardrails_and_monitoring.md`.

Outputs

• Threat model — assets, adversaries, attack paths, assumptions
• Robustness eval spec — datasets, budgets, metrics, baselines, acceptance criteria
• Results report — ASR/slice tables, representative failures, confidence limits
• Defense plan — prioritized mitigations with residual risk
• Campaign log — authorized tests, payloads, reproduction steps (lab/staging)
• Guardrail spec — validation rules, monitors, rollback triggers

Principles

• Authorized testing only — written scope; never attack production without approval
• Empirical over claims — measure ASR and slices; treat certified bounds as supplementary
• Defense in depth — no single control; combine model, input, and operational layers
• Reproducibility — version data, model hash, attack code, and random seeds
• Honest limits — document threat-model mismatch and adaptive attackers

Adversarial robustness scope

Table of contents

1. Role boundary 2. In-scope systems 3. Lifecycle coverage 4. Engagement checklist

Role boundary

Own engineering robustness against adversarial ML threats:

In-scope systems

Out of scope: pure application OWASP testing without model surface (web-pentester).

Lifecycle coverage

Data ingest → Train/fine-tune → Eval → Deploy → Monitor
     │              │            │        │         │
  poisoning      backdoors    ASR/slices  I/O     drift/ASR
  supply chain   robust train  benchmarks  guards  alerts

Document which stages are in scope for each engagement.

Engagement checklist

• [ ] Written authorization and environment (lab/staging/prod exception documented)
• [ ] Model version, weights hash, and config frozen for eval
• [ ] Attack budget defined (L∞/L2 norm, query count, token budget)
• [ ] Success metrics pre-registered (ASR threshold, slice floors)
• [ ] Out-of-scope: customer PII, destructive load, social engineering (unless ai-redteam)
• [ ] Handoff owners for infra guardrails and governance sign-off identified

Defenses and mitigations

Table of contents

1. Defense layers 2. Training-time 3. Inference-time 4. Data and pipeline 5. Residual risk

Defense layers

Prefer defense in depth — document which layer catches which attack class:

Input sanitize → Detector → Robust model → Output policy → Monitor

No layer is sufficient alone against adaptive attackers.

Training-time

Track robust overfitting — robust val gap vs clean val gap.

Inference-time

Align runtime policies with ml-infrastructure-engineer-safeguards for gateway placement.

Data and pipeline

• Provenance: signed datasets, immutable version IDs, anomaly scans on new shards
• Label audit: spot-check high-loss points; trigger-based review for poison patterns
• Supply chain: verify pretrained weights checksums; scan for suspicious layers
• Retrain gates: block promotion if ASR regresses beyond threshold on golden attacks

Residual risk

Document explicitly:

• Adaptive attacks not evaluated
• Transfer from surrogate models untested
• Distribution shift between benchmark and production
• Cost/latency of defenses under peak load

Recommend monitoring hooks in production_guardrails_and_monitoring.md.

Evaluation metrics and benchmarks

Table of contents

1. Core metrics 2. Perturbation budgets 3. Benchmark selection 4. Reporting standards

Core metrics

Always report confidence intervals or multiple seeds for stochastic attacks.

Perturbation budgets

Fix budgets before running campaigns:

Document threat model mismatch if production inputs differ from benchmark norms.

Benchmark selection

Build internal golden sets from production failure clusters; version with dataset cards.

Regression harness: every model release runs fixed attack seed + budget against champion.

Reporting standards

Include in every robustness report:

1. Model ID, data version, attack library version 2. Budget table and attack list (FGSM/PGD/AutoAttack/…) 3. ASR and clean accuracy — overall and slices (class, locale, modality) 4. Representative failure examples (inputs, perturbations, outputs) 5. Comparison to baseline and previous release 6. Known gaps — adaptive attacks not run, transferability untested

Coordinate harm-category metrics with ml-research-engineer-safeguards when eval overlaps moderation.

Production guardrails and monitoring

Table of contents

1. Input guardrails 2. Output guardrails 3. Monitoring signals 4. Incident and rollback

Input guardrails

Implement at edge/gateway (ml-infrastructure-engineer-safeguards) before GPU batching.

Output guardrails

Separate robustness monitors from harm classifiers (ml-research-engineer-safeguards).

Monitoring signals

Track continuously (dashboards + alerts):

Set rollback triggers when canary ASR exceeds champion + δ for N hours.

Incident and rollback

1. Triage — evasion spike vs poison vs extraction vs infra 2. Contain — tighten rate limits, enable stricter sanitization, route to fallback model 3. Preserve — sample payloads (redacted), model version, config for offline replay 4. Remediate — hotfix guardrails vs emergency retrain (coordinate ai-engineer, ai-lead-ops) 5. Post-incident — update golden attack set; rerun full robustness eval before re-promote

Link governance notifications to ai-risk-governance for material robustness failures.

Red-team campaigns on models

Table of contents

1. Scope vs ai-redteam 2. Campaign planning 3. Execution patterns 4. Finding format

Scope vs ai-redteam

Run both when shipping LLM products: robustness eval on model + product red team on app.

Campaign planning

1. Authorize — scope, environment, models, stop conditions 2. Freeze — model hash, API version, defense config snapshot 3. Select attacks — aligned to threat model (evasion set, poison probes, extraction scripts) 4. Allocate budget — query caps, compute ceiling, calendar window 5. Define severity — link ASR deltas and exploit practicality to tiers

Never run destructive load or unapproved prod tests.

Execution patterns

Log: timestamp, seed, attack params, input hash, output, defense flags triggered.

Finding format

### [ID] Title
- **Class**: Evasion | Poisoning | Extraction | Inference
- **Severity**: Critical / High / Medium / Low
- **Environment**: lab | staging
- **Model**: name@version (hash)
- **Budget**: ε=..., queries=...
- **ASR / impact**: ...
- **Reproduction**: minimal script or curl sequence
- **Mitigation**: defense layer + owner
- **Residual risk**: adaptive follow-ups

Hand critical poisoning or supply-chain issues to devsecops and governance as needed.

Threat models and attack taxonomy

Table of contents

1. Adversary capabilities 2. Attack classes 3. LLM-specific notes 4. Threat-model template

Adversary capabilities

State assumptions explicitly — robustness claims are conditional on them.

Attack classes

Map each class to detection and mitigation owners in the defense plan.

LLM-specific notes

Distinguish model robustness from product safety policy (ai-redteam, ml-research-engineer-safeguards).

Threat-model template

## Assets
- Model weights / API / training set / embeddings

## Adversaries
- Capability matrix (knowledge, access, budget)

## Attack paths
1. [Stage] → [Attack] → [Impact]

## Assumptions
- What defender controls (preprocessing, monitoring, retrain cadence)

## Out of scope
- ...

## Metrics
- Primary: ASR @ ε, clean accuracy floor
- Slices: ...