Anti False Positive Decision Making

Anti-False-Positive Decision Making

When to Use

• Calibrate when to act vs monitor when false positives block users, delay revenue, or burn analyst capacity
• Design evidence bars and multi-signal corroboration before irreversible actions (block, freeze, terminate, auto-remediate)
• Tune security, fraud, or compliance alerts with explicit FP/FN trade-offs and base-rate context
• Define tiered response paths and human-in-the-loop gates for screening and monitoring workflows
• Choose metrics (precision, recall, FDR) and review cadence for alert quality—not model training alone
• Document decision rationale for auditors, regulators, or post-incident review when disposition matters
• Reduce alert fatigue and over-blocking without silently accepting unacceptable false negatives

When NOT to Use

• Train, deploy, or optimize production ML classifiers (feature engineering, hyperparameters, MLOps) → data-scientist, ml-ops-engineer
• Legal conclusions, sanctions determinations, or contract interpretation → commercial-counsel
• Author detection rules or queries only with no decision policy, escalation, or disposition framework → information-security-engineer, defensive-security-analyst, aml-compliance (as primary)
• Generic strategy stress-tests without operational decision design → executive-mentor
• Full AML program design, KYC policy, or STR narrative drafting → aml-compliance, str-report
• Internal audit workpapers and control effectiveness sampling → auditor
• Pre-flight architecture or build validation without alert/disposition lens → build-validator

Related skills

Core Workflows

1. Frame the decision and costs

1. Name the action (block, freeze, escalate, auto-close, notify, quarantine) 2. List cost of false positive (customer harm, ops load, revenue, reputation, legal exposure from over-action) 3. List cost of false negative (fraud loss, breach, regulatory miss, safety) 4. Estimate base rate or prevalence band for the population (see reference) 5. Set explicit risk appetite for this decision class (who approves exceptions)

See `references/anti_fp_decision_scope.md` for scope, boundaries, and handoffs.

2. Set evidence bar and corroboration

signal(s) → minimum evidence tier → optional corroboration → disposition → audit record

• Define single-signal vs multi-signal requirements before irreversible action
• Prefer independent evidence types (rule + graph + human review; not two copies of same feature)
• Document what would falsify the hypothesis before closing as benign

See `references/evidence_bars_and_corroboration.md`.

3. Map FP/FN trade-offs and thresholds

• Plot operating points: stricter threshold → fewer FPs, more FNs (usually)
• Use base rate to interpret raw alert rates; avoid judging rules on volume alone
• Segment by customer tier, geography, product, or asset criticality where policy differs

See `references/fp_fn_tradeoffs_and_base_rates.md`.

4. Design tiered response and HITL gates

See `references/tiered_response_and_hitl_gates.md`.

5. Tune alerts in security and compliance contexts

• Govern threshold changes: hypothesis, backtest, approval, post-change review
• Separate detection quality from workflow quality (routing, SLAs, training)
• Track precision at disposition and time-to-benign-close, not alert count alone

See `references/security_compliance_alert_tuning.md`.

6. Measure, calibrate, and document

• Define metrics owners and review cadence (weekly ops, quarterly risk)
• Record rationale template on every material disposition
• Run calibration reviews: sample closed-as-benign and closed-as-true-positive

See `references/metrics_calibration_and_documentation.md`.

When to load references

Anti–false-positive decision scope

Table of contents

1. Purpose 2. In scope 3. Out of scope 4. Decision classes 5. Stakeholders and RACI 6. Peer skill handoffs 7. Deliverables 8. Governance cadence

Purpose

This skill addresses operational and policy decisions where acting on a weak or noisy signal causes disproportionate harm: customer friction, wrongful blocks, analyst burnout, wrongful escalations, or audit findings from undocumented dispositions.

The agent applies structured judgment—not ad-hoc leniency—so teams can defend why they did not act as clearly as why they did.

In scope

Out of scope

Decision classes

Classify each workflow by reversibility and impact:

Anti-FP discipline matters most for classes C and D, where a single weak signal must rarely trigger action alone.

Stakeholders and RACI

Document who can approve threshold moves that increase false negatives (stricter blocking) vs decrease false positives (looser blocking)—both directions need governance.

Peer skill handoffs

Return to this skill when the question is “what bar before we act?” not “how do we implement the control?”

Deliverables

Typical outputs when applying this skill:

1. Decision policy one-pager — action, evidence bar, tiers, owners 2. Threshold change packet — hypothesis, metrics, approval, effective date 3. Disposition rationale template — fields required in case tooling 4. Calibration plan — sample sizes, review frequency, success criteria 5. Escalation matrix — tier → role → SLA

Governance cadence

Operating principles

1. Prefer delay over wrong irreversible action when base rate is low and cost of FP is high. 2. Never optimize on alert volume alone—volume can rise with better detection or worse noise. 3. Corroborate with independent signal types before Tier 3–4. 4. Document negatives (why not escalated) as rigorously as positives. 5. Segment populations; one global threshold often fails retail and corporate alike.

Red flags (process failure)

• “We blocked because the model said so” with no secondary check for Class D
• Threshold changes in production without versioned approval
• 90%+ benign closure with one-line notes
• Analysts incentivized only on alert closure speed
• No FN review when FP rate drops sharply (may indicate under-detection)

Evidence bars and corroboration

Table of contents

1. Evidence tiers 2. Corroboration rules 3. Independence 4. Disconfirming evidence 5. Screening-specific patterns 6. [Security alert patterns](#security-alert patterns) 7. Rationale template 8. Anti-patterns

Evidence tiers

Define tiers per decision class (see anti_fp_decision_scope.md):

Map tiers to response tiers:

• E0–E1 → observe or soft queue
• E2 → review before customer impact
• E3 → contain pending review
• E4 → irreversible path with approvals

Corroboration rules

Corroboration = at least two pieces of evidence that jointly increase confidence before action.

Minimum rules (examples—adapt per policy)

Write rules as AND across independent families, not duplicate counts of the same data.

Independence

Evidence families should not share a single failure point:

Not independent: two rules using the same raw field; model score + derived bucket from same score.

Disconfirming evidence

Before closing or escalating, explicitly check falsifiers:

Record what you looked for and what you found in the case note.

Screening-specific patterns

Sanctions and PEP screening (operational—not legal advice):

Pair with aml-compliance for program design; this skill sets bars, not vendor config alone.

Security alert patterns

Use defensive-security-analyst for detection content; use this reference for disposition bar.

Rationale template

Required fields for material dispositions:

Alert ID / case ID:
Decision: [monitor | soft | hold | escalate | close benign | confirm]
Evidence tier reached: E_
Signals (independent families):
  1.
  2.
Disconfirming checks performed:
Outcome of checks:
Residual uncertainty:
Approver (if Tier 3+):
Next review date (if monitor):

Anti-patterns

Escalation when evidence is insufficient

If tier required for policy but evidence is only E1:

1. Do not downgrade policy silently 2. Escalate to risk owner for exception or temporary monitor tier 3. Time-box enhanced monitoring 4. Log exception with expiry

FP/FN trade-offs and base rates

Table of contents

1. Definitions 2. Why base rate matters 3. Cost asymmetry 4. Operating points 5. Segmentation 6. Bayesian intuition (practical) 7. Worksheet 8. Common mistakes

Definitions

In high-stakes workflows, precision at disposition (after analyst review) often matters more than raw rule precision.

Why base rate matters

Base rate (prevalence): fraction of the population that truly has the condition.

When prevalence is low, even a highly specific rule can produce mostly false positives among alerts:

• Example: 0.1% fraud rate, rule with 99% specificity on random population still yields ~90% FPs among alerts if sensitivity forces volume (illustrative—always model with your data).

Never judge a rule only by “we get 10,000 alerts/month” without prevalence and disposition outcomes.

Cost asymmetry

Build a cost matrix before moving thresholds:

FP costs to enumerate:

• Customer churn, support load, wrongful freeze
• Analyst hours × fully loaded rate
• Reputation and regulatory complaints
• Opportunity cost (delayed wire, failed checkout)

FN costs to enumerate:

• Direct loss, breach impact, fine exposure
• Safety harm, sanctions exposure

Risk appetite states which cell the org will tolerate more of, by segment.

Operating points

Threshold selection is choosing an operating point on a precision–recall curve (or ROC for scores):

Document the intended move in every tuning packet:

• “Reduce FP on retail wires by 30% at acceptable 5% recall drop”
• Not “reduce alerts” without FN review

Segmentation

Global thresholds fail when subpopulations differ:

Use segment-specific thresholds and evidence bars where policy allows; document exceptions.

Bayesian intuition (practical)

After an alert fires, ask:

Given this evidence, what is the posterior probability of true harm?

Operational proxies:

• Single weak feature → low posterior → Tier 1–2 only
• Multiple independent features → higher posterior → Tier 3
• Human-confirmed pattern → may justify Tier 4 with approvals

You do not need formal Bayes in production if you enforce independent corroboration and calibration sampling.

Worksheet

Copy for each decision class:

Decision class: _______________
Population size (monthly): _______________
Estimated prevalence (band): ___% to ___%
Current alerts/month: _______________
Disposition TP / FP / FN (last 90d): ___ / ___ / ___
Estimated cost per FP: $________
Estimated cost per FN: $________
Target precision at disposition: ___%
Minimum acceptable recall: ___%
Approved operating point change: _______________
Approver: _______________  Date: _______________

Common mistakes

Review triggers

Revisit trade-offs when:

• Product or geography launch changes population mix
• Adversary adapts (sudden FN spike)
• Regulatory feedback on over-blocking
• Major vendor or model change
• FP cost event (public complaint, executive escalation)

Metrics, calibration, and documentation

Table of contents

1. Metric hierarchy 2. Precision, recall, and FDR 3. Labels and lag 4. Calibration program 5. Sampling methodology 6. Documentation artifacts 7. Audit and regulatory readiness 8. Dashboards 9. Failure modes

Metric hierarchy

Anti-FP programs optimize L3–L4, not L1 alone.

Precision, recall, and FDR

Report confidence intervals on sampled metrics when populations are small.

Disposition precision

Precision_at_disposition = Confirmed_TP_after_review / All_closed_as_TP

Separate rule-fired precision from analyst-confirmed—analysts may rescue weak rules or bury FPs in benign closes.

Labels and lag

Ground truth is delayed:

Run matured cohort reviews quarterly on older alert vintages.

Calibration program

Calibration = periodic check that scores, tiers, and dispositions match reality.

Cadence

Calibration meeting agenda (60 min)

1. Metric dashboard vs targets 2. Top 5 FP drivers (root cause) 3. FN themes (from incidents + sample) 4. Open tuning packets status 5. Exceptions nearing expiry 6. Actions with owners

Sampling methodology

Minimum sample sizes depend on volume; document confidence when n is small.

Stratification dimensions

• Rule / scenario ID
• Tier at disposition
• Analyst / team
• Customer segment
• Geography

Documentation artifacts

Maintain versioned artifacts:

Store in systems of record—not only wiki.

Audit and regulatory readiness

Auditors and regulators often ask:

• How do you know alerts are appropriately dispositioned?
• Show threshold changes and approvals
• Show sample of benign closes with rationale
• Prove irreversible actions had sufficient evidence

Prepare audit pack per scenario family:

1. Policy excerpt 2. Metric trend (precision, FN incidents) 3. Sample workpapers (redacted) 4. Tuning log excerpt

Use auditor for workpaper format; use this reference for quality metrics narrative.

Dashboards

Minimum viable dashboard panels:

Avoid ranking analysts on volume closed only.

Failure modes

Integration with build validation

Before launching new automated checks in product or CI, ask (build-validator complement):

• What tier can this signal trigger?
• What evidence bar is required?
• What metric proves it is not noisy?

Document answers in the launch checklist.

Security and compliance alert tuning

Table of contents

1. Tuning vs detection 2. Governed change process 3. Security operations (SOC) 4. Compliance monitoring 5. Fraud and AML TM 6. Screening systems 7. Enrichment and deduplication 8. Metrics for tuning 9. Post-change review

Tuning vs detection

Do not conflate “fewer alerts” with success if FN rate rises without review.

Governed change process

Every material tuning change follows:

1. Hypothesis → 2. Analysis → 3. Approval → 4. Deploy → 5. Post-review

1. Hypothesis

Document:

• Problem statement (FP pain, FN incident, new product)
• Affected rules/scenarios and population
• Expected direction (precision ↑, recall ↓, etc.)

2. Analysis

Include base rate and disposition breakdown (see fp_fn_tradeoffs_and_base_rates.md).

3. Approval

4. Deploy

• Version ID and effective timestamp
• Feature flag or gradual rollout where possible
• Rollback plan documented

5. Post-review

30- and 90-day checkpoints: precision at disposition, FN themes, customer complaints, SLA breach.

Security operations (SOC)

Pair with soc-analyst for shift ops; defensive-security-analyst for rule content.

Compliance monitoring

Continuous control monitoring (CCM) alerts differ from threat alerts:

Use compliance-engineer for evidence automation design; this reference for alert disposition policy.

Fraud and AML TM

Transaction monitoring tuning (operational):

Hand off program-level scenario libraries to aml-compliance; use governed process here.

Screening systems

Enrichment and deduplication

Often the highest ROI for FP reduction without lowering detection:

Metrics for tuning

Track before and after deploy:

Post-change review

Template:

Change ID:
Effective date:
Hypothesis result: [met | partial | failed]
Volume delta: ___%
Precision at disposition delta: ___%
FN incidents (if any):
Customer impact signals:
Decision: [keep | rollback | iterate]

When not to tune

• During active incident (stabilize first)
• Without baseline metrics
• To meet arbitrary “alert reduction OKR”
• By disabling detection without risk acceptance

Route systemic detection gaps to engineering and detection owners; route policy gaps to risk owner.

Tiered response and human-in-the-loop gates

Table of contents

1. Response tier model 2. Mapping evidence to tiers 3. HITL gate types 4. SLAs and capacity 5. Automation guardrails 6. Dual control and approvals 7. Customer communication 8. Exception handling 9. Examples by domain

Response tier model

Default posture for low base rate + high FP cost: start at Tier 0–1; earn Tier 3–4 with evidence tiers (see evidence_bars_and_corroboration.md).

Mapping evidence to tiers

Escalation up tiers requires new evidence or approver exception—never downgrade evidence requirement silently.

HITL gate types

Document gates in runbooks; do not rely on tribal knowledge.

SLAs and capacity

Capacity planning prevents fake precision from rushed closes:

If SLA routinely missed:

• Do not lower evidence bar
• Add enrichment, dedup, or tier-0 routing
• Escalate capacity to operations leadership

Pair with incident-management-engineer for major incident severity—not daily alert queues.

Automation guardrails

Allowed automation only with:

1. Written policy mapping tier → action 2. Kill switch and rollback 3. Audit log (who/what/when/version) 4. Rate limits per customer / global 5. Periodic sample QA (G1)

Prohibited without explicit risk acceptance:

• Tier 4 without human
• Permanent suppress without expiry
• Cross-customer threshold changes without segment analysis

Dual control and approvals

Record approver identity in case system—not chat only.

Customer communication

Tier 3–4 often requires controlled messaging:

Route wording to compliance/legal for regulated text; this skill defines when to contact, not copy.

Exception handling

Temporary exception (risk acceptance):

Exception ID:
Scope (customers, rules, duration):
Evidence bar waived to tier: ___
Approver:
Expiry (mandatory):
Review date:

Exceptions must expire automatically; no permanent “VIP skip” without executive register.

Examples by domain

Payments fraud

Sanctions screening

Security (account compromise)

AI / automated agent actions

For agentic systems that act on alerts:

• Require tool policy: max tier without human token
• Log chain-of-thought decisions not raw model rationale for audit
• Use ai-risk-governance for model tier; use this reference for action tier

De-escalation

When evidence weakens:

1. Release contain at Tier 3 when disconfirming checks pass 2. Document de-escalation rationale 3. Retain monitor tier (0–1) for cooling period if policy requires