Ai Redteam

AI Red Team

When to Use

• Red-teaming chatbots, agents, RAG systems, or copilots before launch
• Designing safety evaluation suites and adversarial test harnesses
• Reproducing reported prompt injection or jailbreak vulnerabilities
• Validating mitigations after incidents (retesting filters, hardening)
• Running multi-turn coercion, encoding, or indirect injection campaigns
• Assessing bias, harmful output, or data exfiltration risks in LLM applications
• Scoping rules of engagement and severity rubrics for AI security testing

When NOT to Use

• Writing corporate AI policy or risk governance frameworks → ai-risk-governance
• Building production LLM features or RAG pipelines → ai-engineer
• General network/AD/infra penetration testing → network-pentester
• Authorized web/API OWASP testing (non-LLM) → web-pentester
• Enterprise adversary simulation, MITRE ATT&CK campaigns, purple team → red-team-specialist
• Binary, firmware, or protocol reverse engineering → reverse-engineer
• CI/CD pipeline security → devsecops

Related skills

Core Workflows

1. Scope and rules of engagement

1. Define target: model, app surface, tools, data stores 2. Obtain written authorization and time window 3. Agree out-of-scope (e.g., no social engineering of employees unless approved) 4. Define success criteria: critical findings, reproduction steps, severity rubric 5. Plan safe test environment (no prod customer data)

See `references/engagement_scope.md` for ROE template and severity definitions.

2. Threat model for LLM applications

See `references/attack_catalog.md` for technique families and test prompts (use ethically).

3. Test execution

Phases:

1. Baseline — document intended refusals and allowed behaviors 2. Automated sweep — harness with curated attack set + fuzz mutations 3. Manual creativity — domain-specific abuse scenarios 4. Tool/RAG focus — indirect injection via retrieved documents 5. Regression — re-run after mitigations

Log: input, output, tool calls, latency, whether guardrail fired.

See `references/testing_harness.md` for harness design and datasets.

4. Reporting

Each finding includes:

• Title and severity (impact × likelihood)
• Steps to reproduce (minimal)
• Evidence (redacted transcripts)
• Affected component
• Recommended mitigation
• Retest criteria

See `references/reporting.md` for report template and remediation tracking.

5. Mitigation validation

See `references/mitigations.md` for defense depth and known weak controls.

When to load references

• ROE and scope → references/engagement_scope.md
• Attack types → references/attack_catalog.md
• Harness and automation → references/testing_harness.md
• Reports → references/reporting.md
• Defenses → references/mitigations.md

Attack catalog

Table of contents

1. Prompt injection 2. Jailbreak families 3. RAG attacks 4. Ethics

Prompt injection

• Direct: user overrides system in same turn
• Indirect: malicious content in retrieved doc or email parsed by agent
• Payload in metadata fields (titles, alt text)

Jailbreak families

• Role-play and fictional framing
• Encoding (base64, other languages)
• Multi-turn gradual compliance
• Refusal suppression ("ignore previous")

Test only on authorized systems; do not publish working exploits against third parties without permission.

RAG attacks

• Plant document in index with hidden instructions
• Poison chunk boundaries to split defenses

Ethics

Authorized testing only; minimize harm; redact PII in reports.

Engagement scope

Table of contents

1. ROE template 2. Severity rubric

ROE template

# AI red team — rules of engagement
- Target systems:
- Authorized testers:
- Window:
- Prohibited:
- Data handling:
- Emergency contact:

Severity rubric

Mitigations

Table of contents

1. Defense layers 2. Weak controls

Defense layers

Weak controls

• Security through obscurity in system prompt alone
• Single regex filter
• Assuming fine-tuning removes jailbreaks
• Trusting retrieved HTML without sanitization

Always validate with ai-redteam regression set after changes.

Reporting

Table of contents

1. Finding template 2. Remediation tracking

Finding template

### [SEV] Title
**Component:**
**Steps:**
1.
**Impact:**
**Evidence:** (redacted transcript)
**Recommendation:**
**Retest:**

Remediation tracking

Link findings to tickets; retest within 2 weeks of fix; close only on passing regression suite.

Testing harness

Table of contents

1. Harness design 2. Dataset sources

Harness design

for attack in dataset:
  response = app.send(attack)
  score = judge(policy_violation, leakage, tool_abuse)
  log(transcript, score)

• Pin model and prompt versions per run
• Parallelize with rate limits
• Compare to baseline after mitigations

Dataset sources

• Internal policy violation probes
• Public harm benchmarks (use responsibly)
• Custom cases from support tickets (sanitized)
• Mutations of failed cases from prior runs