
Legal Red Team
- 52 installs
- 451 repo stars
- Updated July 21, 2026
- borghei/claude-skills
legal-red-team is a skill that adversarially verifies AI-generated legal content by fact-checking claims, validating citations, checking arithmetic, and scoring distribution readiness.
About
This skill is an adversarial verification framework for AI-generated legal content. It checks factual accuracy, legal citations, arithmetic, source verification, speculation, and disclaimer adequacy across six categories, then scores document quality and distribution readiness. Legal teams use it to fact-check and detect hallucinations in AI-drafted legal documents before distribution. It is explicitly experimental and not legal advice.
- Adversarially verifies AI-generated legal content across 6 categories
- Fact-checks claims, validates citations, checks arithmetic, and detects speculation
- Scores document quality and distribution readiness with a severity taxonomy
Legal Red Team by the numbers
- 52 all-time installs (skills.sh)
- Ranked #569 of 1,352 Code Review & Quality skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
legal-red-team capabilities & compatibility
- Capabilities
- fact checking · hallucination detection · citation validation · quality scoring
- Use cases
- code review · research · web search
- Pricing
- Free
What legal-red-team says it does
Adversarial verification for AI-generated legal content. Use when fact-checking legal documents, validating citations, detecting hallucinations, scoring document quality, or assessing distribution
Every AI-generated legal document must be checked across 6 categories.
If a claim cannot be verified from an official source, flag it. Do not assume accuracy.
npx skills add https://github.com/borghei/claude-skills --skill legal-red-teamAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 52 |
|---|---|
| repo stars | ★ 451 |
| Last updated | July 21, 2026 |
| Repository | borghei/claude-skills ↗ |
What it does
Fact-check and adversarially verify AI-generated legal content, flagging hallucinations before distribution.
Who is it for?
Legal teams fact-checking AI-drafted legal documents and detecting hallucinations before distribution.
Skip if: Anyone needing actual legal advice; the skill is experimental and explicitly not legal advice.
When should I use this skill?
You are fact-checking legal documents, validating citations, or detecting hallucinations in AI-generated legal content.
What you get
Produces a verification report flagging unverifiable claims, citation errors, and a distribution-readiness quality score.
- Verification report
- Quality score
- Flagged hallucination list
By the numbers
- 6 verification categories
- Six-step verification methodology
- Two Python tools: legal fact checker and legal quality scorer
Files
⚠️ EXPERIMENTAL — This skill is provided for educational and informational purposes only. It does NOT constitute legal advice. All responsibility for usage rests with the user. Consult qualified legal professionals before acting on any output.
Legal Red Team
Production-ready adversarial verification framework for AI-generated legal content. Covers factual accuracy, citation validation, arithmetic checking, speculation detection, and distribution readiness scoring.
---
Table of Contents
- Verification Categories
- Tools
- Six-Step Methodology
- Severity Taxonomy
- Quality Score
- Known Hallucination Patterns
- Reference Guides
- Workflows
- Troubleshooting
- Success Criteria
- Scope and Limitations
- Anti-Patterns
- Tool Reference
---
Verification Categories
Every AI-generated legal document must be checked across 6 categories.
| # | Category | What to Check | Red Flags |
|---|---|---|---|
| 1 | Factual Accuracy | Dates, references, numbers, entity names, timelines | Wrong effective dates, confused entity names, incorrect amounts |
| 2 | Legal Authority Citations | Primary/secondary sources, format, hierarchy, currency | Non-existent articles, wrong section numbers, outdated citations |
| 3 | Arithmetic Validation | Timelines, percentages, financial calculations, deadlines | Date math errors, percentage miscalculations, compounding mistakes |
| 4 | Source Verification | Verifiable claims, official sources, cross-referencing | Unverifiable assertions stated as fact, single-source claims |
| 5 | Speculation Detection | Opinion vs fact, uncertainty language, predictive claims | Predictions stated as certainty, guidance treated as binding law |
| 6 | Disclaimer Adequacy | Legal advice disclaimers, jurisdiction, date, professional consultation | Missing disclaimers, overly broad claims, no jurisdiction limits |
---
Tools
Legal Fact Checker
Scans legal text for verifiable claims and flags potential hallucination patterns.
# Check a legal document
python scripts/legal_fact_checker.py --input document.txt
# Check with JSON output
python scripts/legal_fact_checker.py --input memo.txt --json
# Check inline text
python scripts/legal_fact_checker.py --text "Under GDPR Article 83(5), fines can reach EUR 20 million..."
# Save verification report
python scripts/legal_fact_checker.py --input document.txt --output report.jsonLegal Quality Scorer
Scores legal document quality across all 6 verification categories.
# Score a document
python scripts/legal_quality_scorer.py --input document.txt
# Score with JSON output
python scripts/legal_quality_scorer.py --input document.txt --json
# Score with detailed breakdown
python scripts/legal_quality_scorer.py --input document.txt --verbose
# Save quality assessment
python scripts/legal_quality_scorer.py --input document.txt --output assessment.json---
Six-Step Methodology
Step 1: Initial Review
Read the entire document with an adversarial mindset. For each claim, ask:
- Is this verifiable?
- Does this sound too specific to be generated without a source?
- Does this sound too confident for an uncertain area?
Mark every factual assertion, citation, date, number, and predictive statement.
Step 2: Source Verification (ALWAYS Web Search)
For every verifiable claim, attempt to verify against official sources.
| Source Type | Verification Method | Examples |
|---|---|---|
| EU legislation | EUR-Lex official database | eur-lex.europa.eu |
| US federal law | congress.gov, govinfo.gov | Official code and statutes |
| US regulations | eCFR, Federal Register | ecfr.gov |
| UK legislation | legislation.gov.uk | Official statute database |
| Court decisions | Court databases, Westlaw, LexisNexis | Official reporters |
| Agency guidance | Agency official website | Direct download from .gov/.europa.eu |
| International treaties | UN Treaty Collection | treaties.un.org |
Rule: If a claim cannot be verified from an official source, flag it. Do not assume accuracy.
Step 3: Arithmetic Verification
Check every calculation, date computation, and numerical claim.
| Check Type | Method |
|---|---|
| Timeline calculations | Count days/months/years between stated dates |
| Percentage calculations | Recalculate from base figures |
| Financial computations | Verify arithmetic and compounding |
| Deadline calculations | Confirm against statutory text |
| Penalty ranges | Cross-check against statute |
Step 4: Citation Validation
For every legal citation, verify:
| Element | Check |
|---|---|
| Source exists | Does the cited statute/article/section actually exist? |
| Content matches | Does the cited provision say what the document claims? |
| Citation format | Is the citation in correct format for the jurisdiction? |
| Currency | Is this the current, in-force version? |
| Hierarchy correct | Is the source characterized at the right authority level? |
Step 5: Speculation Identification
Distinguish fact from opinion, certainty from prediction.
| Language Pattern | Classification | Action |
|---|---|---|
| "The law requires..." | Factual claim | Verify against statutory text |
| "Courts will likely..." | Speculation | Flag; add uncertainty qualifier |
| "It is recommended..." | Guidance | Verify source; clarify if binding |
| "Best practice suggests..." | Opinion | Label as opinion; cite source |
| "This means that..." | Interpretation | Flag if stated as fact without authority |
| "Companies must..." | Obligation claim | Verify statutory basis |
Step 6: Disclaimer Review
Every AI-generated legal document must include:
| Required Element | Description |
|---|---|
| Not legal advice | Clear statement that content is informational only |
| Jurisdiction limitations | Which jurisdictions are and are not covered |
| Date of preparation | When the content was prepared (law changes) |
| Professional consultation | Recommendation to consult qualified legal counsel |
| AI-generated disclosure | Statement that content was generated or assisted by AI |
| Accuracy limitations | Acknowledgment that verification is recommended |
---
Severity Taxonomy
| Severity | Definition | Examples | Action |
|---|---|---|---|
| CRITICAL | Factually wrong in a way that could cause legal harm | Wrong article number creating false obligation, incorrect penalty amount, non-existent legal requirement | Must fix before any distribution |
| HIGH | Materially misleading or unverifiable | Guidance stated as binding law, unverifiable timeline, confident but unsourced claim | Must fix or add prominent caveat |
| MODERATE | Imprecise or potentially confusing | Ambiguous language, minor date discrepancy, incomplete citation | Should fix; acceptable with caveat |
| LOW | Style or formatting issue | Citation format inconsistency, missing cross-reference, minor redundancy | Fix if time permits |
---
Quality Score
| Score | Rating | Distribution Status | Criteria |
|---|---|---|---|
| 5/5 | Distribution Ready | Safe to distribute | Zero CRITICAL/HIGH issues; all citations verified; disclaimers complete |
| 4/5 | Minor Revisions | Safe after small fixes | Zero CRITICAL; 1-2 HIGH issues with clear fixes; most citations verified |
| 3/5 | Moderate Revisions | Needs work before distribution | Zero CRITICAL; 3+ HIGH issues; some unverified citations |
| 2/5 | Major Revisions | Not safe to distribute | 1+ CRITICAL issues; multiple HIGH issues; significant unverified content |
| 1/5 | Not Distribution Ready | Requires complete rework | Multiple CRITICAL issues; pervasive inaccuracies; unreliable throughout |
---
Known Hallucination Patterns
AI models exhibit 5 recurring patterns when generating legal content.
| # | Pattern | Description | Detection Technique |
|---|---|---|---|
| 1 | Plausible but wrong article numbers | AI generates article/section numbers that sound correct but do not exist (e.g., "Article 42(5)" when only 42(1)-(4) exist) | Cross-reference every article number against official statute text |
| 2 | Confident but incorrect dates | Implementation timelines, effective dates, or deadlines stated with false confidence (off by weeks or months) | Verify every date against official timeline from the statute or implementing body |
| 3 | Mixing guidance and legal requirements | Treating non-binding recommendations as binding obligations (e.g., stating ENISA recommendations as NIS2 requirements) | Check whether cited source is binding legislation vs guidance; verify authority level |
| 4 | Outdated legal references | Citing superseded or repealed provisions without noting they are no longer in force | Verify currency of every cited provision; check for amendments and repeals |
| 5 | Arithmetic errors in timeline calculations | Miscounting days, months, or years between dates; wrong deadline calculations | Independently calculate every timeline; do not trust AI date math |
See references/hallucination_patterns.md for detailed examples and prevention strategies.
---
Reference Guides
| Guide | Path | Description |
|---|---|---|
| Verification Methodology | references/verification_methodology.md | Complete 6-step methodology with source hierarchy and citation formats |
| Hallucination Patterns | references/hallucination_patterns.md | 5 patterns with examples, detection, and prevention strategies |
---
Workflows
Workflow 1: Full Adversarial Review
1. Run scripts/legal_fact_checker.py on the document. 2. Review flagged items and verify each against official sources. 3. Run scripts/legal_quality_scorer.py for category scores. 4. For each CRITICAL/HIGH finding, document: the error, the correct information, and the source. 5. Produce a verification report with findings by severity. 6. Assign quality score and distribution readiness assessment. 7. Validation: Every verifiable claim checked, score assigned, recommendations provided.
Workflow 2: Quick Citation Check
1. Run scripts/legal_fact_checker.py on the document. 2. Focus on citation extraction results. 3. Verify each extracted citation against official source. 4. Flag any citation that cannot be verified. 5. Validation: All citations verified or flagged.
Workflow 3: Pre-Distribution Gate
1. Run scripts/legal_quality_scorer.py on the final document. 2. Review composite score. 3. If score < 4/5, document must not be distributed. 4. If score >= 4/5, verify CRITICAL count is zero. 5. Confirm all disclaimers are present and adequate. 6. Validation: Quality score >= 4/5, zero CRITICAL issues, disclaimers complete.
---
Troubleshooting
| Problem | Likely Cause | Resolution |
|---|---|---|
| Too many false positives | Regex patterns matching non-legal text | Narrow input to legal content only; use context-aware review |
| Cannot verify citation | Source not freely accessible | Note as "unverifiable from public sources"; do not assume correct |
| AI-generated text has no citations | Content is entirely unsourced | Flag entire document as unverified; score as 2/5 or lower |
| Hallucination pattern detected | AI confabulation of legal details | Replace with verified information from official source |
| Document mixes jurisdictions | No clear jurisdiction scope | Flag as HIGH; recommend splitting by jurisdiction |
| Quality score seems too high | Automated scoring has limits | Always supplement automated scoring with manual review |
---
Success Criteria
| Criterion | Target |
|---|---|
| Citations verified | 100% of legal citations checked against official sources |
| Hallucination patterns scanned | All 5 known patterns checked |
| Arithmetic validated | Every calculation independently verified |
| Severity assigned | Every finding classified CRITICAL/HIGH/MODERATE/LOW |
| Quality score calculated | Composite score with per-category breakdown |
| Disclaimers verified | All 6 required disclaimer elements present |
| Distribution decision | Clear go/no-go recommendation with rationale |
---
Scope & Limitations
In scope: Verifying factual claims in legal text, validating citations, detecting hallucination patterns, scoring document quality, assessing distribution readiness.
Out of scope: Verifying legal conclusions or interpretations, assessing litigation strategy, replacing professional legal review, accessing paid legal databases (Westlaw, LexisNexis).
Disclaimer: This skill provides a structured adversarial verification methodology. It catches common AI errors but cannot guarantee complete accuracy. Professional legal review remains essential for high-stakes documents.
---
Anti-Patterns
| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Trusting AI-generated citations without verification | AI models routinely generate plausible but non-existent legal citations; unverified citations in distributed documents create serious credibility and legal risk | Verify every citation against official sources; assume wrong until proven right |
| Relying solely on automated checking | Automated tools catch patterns but miss contextual errors, mischaracterizations, and subtle hallucinations | Use automated tools for first pass, then conduct manual review of all flagged items and a sample of unflagged items |
| Skipping the "adversarial mindset" | Confirmation bias leads reviewers to accept plausible-sounding content; legal text that "sounds right" may still be wrong | Actively seek to disprove every claim; assume error until verified; question every specific number, date, and citation |
| Distributing with score 3/5 or below | MODERATE and HIGH issues in distributed documents undermine credibility and may cause legal harm | Set a firm distribution threshold at 4/5; no exceptions without documented risk acceptance by a qualified reviewer |
---
Tool Reference
| Tool | Input | Output | Use Case |
|---|---|---|---|
legal_fact_checker.py | Legal document text | Verification report with flagged claims, citations, dates, hallucination alerts | First-pass automated scanning of legal content |
legal_quality_scorer.py | Legal document text | Quality score (1-5) with per-category breakdown and severity-classified findings | Pre-distribution quality gate |
AI Hallucination Patterns in Legal Content
5 known hallucination patterns with descriptions, real-world examples, detection techniques, and prevention strategies. Plus quality score rubric and adversarial review principles.
---
Table of Contents
- Overview
- Pattern 1: Plausible but Wrong Article Numbers
- Pattern 2: Confident but Incorrect Dates
- Pattern 3: Mixing Guidance and Legal Requirements
- Pattern 4: Outdated Legal References
- Pattern 5: Arithmetic Errors in Timeline Calculations
- Detection Quick Reference
- Quality Score Rubric
- Adversarial Review Principles
---
Overview
AI language models generate legal content that sounds authoritative but frequently contains errors that could mislead readers or cause legal harm. These errors follow predictable patterns.
| Pattern | Severity | Frequency | Detection Difficulty |
|---|---|---|---|
| Wrong article numbers | CRITICAL | High | Medium -- requires source checking |
| Incorrect dates | HIGH | High | Medium -- requires calendar verification |
| Guidance as law | HIGH | Very High | Hard -- requires understanding authority hierarchy |
| Outdated references | HIGH | Medium | Medium -- requires currency checking |
| Timeline arithmetic | MODERATE | High | Easy -- requires basic math |
Why these patterns occur:
- AI models are trained on large text corpora that include outdated, incorrect, and informal legal content.
- Models generate plausible-sounding text by statistical pattern matching, not by reasoning from source material.
- Legal content requires precision that exceeds what statistical text generation can reliably produce.
- Models cannot verify their own outputs against current, authoritative sources.
---
Pattern 1: Plausible but Wrong Article Numbers
Description
AI models generate article, section, or paragraph numbers that sound correct for the statute being discussed but do not actually exist or refer to a different topic than claimed.
Why It Happens
Models learn that certain statutes have articles in certain ranges and that specific topics appear at specific article numbers. They generate numbers that fit the statistical pattern but are not always correct.
Examples
| AI-Generated Claim | Actual Reality | Error Type |
|---|---|---|
| "AI Act Article 42(5) requires..." | Article 42 has only paragraphs (1)-(4) | Non-existent subsection |
| "GDPR Article 83(7) provides for..." | Article 83 has only paragraphs (1)-(6) | Non-existent paragraph |
| "Under NIS2 Article 35..." | NIS2 has 46 articles; Art. 35 exists but covers a different topic | Wrong topic attribution |
| "DORA Article 55(3)(b)..." | Article 55 does not have a paragraph 3(b) | Fabricated sub-reference |
| "Section 1798.150(a)(1)(B) of the CCPA..." | Subsection structure is different | Wrong internal structure |
Detection Techniques
1. Cross-reference every article number against the official statute text. 2. Check maximum article/section number for the statute (e.g., GDPR has 99 articles; any reference above 99 is immediately wrong). 3. Verify internal structure (paragraph numbers, subsections, letters). 4. Confirm topic match -- even if the article number exists, verify it covers the topic claimed. 5. Be especially suspicious of high subsection numbers (e.g., paragraph (7) or higher is uncommon).
Prevention Strategies
- Always provide article text directly from the official source when drafting.
- Cross-reference every article number before including it in output.
- When uncertain about a specific subsection, cite the article broadly rather than fabricating precision.
- Use EUR-Lex, govinfo.gov, or other official databases for real-time verification.
---
Pattern 2: Confident but Incorrect Dates
Description
AI models state implementation timelines, effective dates, and compliance deadlines with false confidence. Dates are typically close to correct but off by weeks, months, or occasionally years.
Why It Happens
Training data contains multiple draft versions, proposals, and commentary that discuss various proposed dates. Models conflate proposed dates with final dates, or confuse entry-into-force with application dates.
Examples
| AI-Generated Claim | Actual Reality | Error |
|---|---|---|
| "The AI Act applies from August 1, 2026" | Phased application: prohibited practices Feb 2025, high-risk Aug 2026, other provisions Aug 2027 | Oversimplified; missed phased timeline |
| "GDPR entered into force on May 25, 2018" | Entered into force May 24, 2016; became applicable May 25, 2018 | Confused entry-into-force with application date |
| "NIS2 transposition deadline is October 2024" | Transposition deadline is October 17, 2024 | Imprecise (missing day) |
| "DORA applies from January 2025" | DORA applies from January 17, 2025 | Imprecise (missing day) |
| "CCPA was enacted in 2019" | Signed into law June 28, 2018; effective January 1, 2020 | Wrong year for both enactment and effectiveness |
Detection Techniques
1. Verify every date against the official source (Official Journal, government gazette). 2. Check for phased timelines -- most modern legislation has multiple application dates. 3. Distinguish entry-into-force from application date (EU law routinely has a 2-year gap). 4. Verify the specific day, not just month and year. 5. Cross-reference timeline with official implementation guidance from the relevant authority.
Prevention Strategies
- Always cite the specific article that establishes the timeline (e.g., "Article 113 of the AI Act").
- Present phased timelines as tables, not single dates.
- When uncertain about a specific date, state the uncertainty: "approximately" or "subject to verification."
- Maintain a reference table of verified key dates for frequently discussed statutes.
---
Pattern 3: Mixing Guidance and Legal Requirements
Description
AI models frequently treat non-binding guidance, recommendations, and best practices as if they were binding legal requirements. This is one of the most common and dangerous hallucination patterns.
Why It Happens
Training data often discusses guidance and legislation together without clearly distinguishing authority levels. Models generate text that conflates "ENISA recommends" with "NIS2 requires" or "EDPB guidance states" with "GDPR mandates."
Examples
| AI-Generated Claim | Actual Reality | Error |
|---|---|---|
| "ENISA requires organizations to conduct annual penetration testing" | ENISA publishes guidance; NIS2 Art. 21 requires risk management measures but does not mandate annual pen testing | Guidance presented as statutory requirement |
| "The EDPB mandates that consent pop-ups must include a reject button of equal size" | EDPB issues guidelines; specific UI requirements may come from national DPA enforcement | Advisory guidance stated as mandate |
| "NIST CSF requires organizations to maintain an asset inventory" | NIST CSF is a voluntary framework; it recommends, not requires | Voluntary framework stated as requirement |
| "The ICO requires a DPIA for all AI systems" | ICO provides guidance on when DPIAs are needed; GDPR Art. 35 sets the legal threshold | Guidance-level interpretation stated as rule |
| "ISO 27001 legally requires annual management reviews" | ISO 27001 is a voluntary standard; management reviews are a certification requirement, not a legal requirement | Standard requirement conflated with legal obligation |
Detection Techniques
1. Check the source type: Is the cited source a statute/regulation (binding) or guidance/recommendation (non-binding)? 2. Verify the verb: Does the source actually "require" or does it "recommend/suggest/encourage"? 3. Check the authority chain: Does the guidance body have rule-making authority, or only advisory authority? 4. Look for conditional language: Guidance often uses "should" while law uses "shall/must." 5. Verify the enforcement mechanism: Non-binding guidance typically has no enforcement mechanism.
Authority Level Quick Reference
| Source Type | Binding? | Examples |
|---|---|---|
| Statute/Regulation | Yes | GDPR, AI Act, CCPA, NIS2, DORA |
| Delegated/Implementing act | Yes | Commission implementing regulations |
| Agency rule (US) | Yes (after notice and comment) | FTC rules, SEC rules |
| Court decision | Yes (within jurisdiction) | CJEU rulings, Supreme Court decisions |
| Agency guidance | No | EDPB guidelines, ICO guidance, CNIL guides |
| International standard | No (unless mandated by statute) | ISO 27001, NIST CSF |
| Industry body recommendation | No | ENISA reports, CIS benchmarks |
Prevention Strategies
- Always verify whether the cited source has binding legal authority.
- Use precise language: "EDPB recommends..." not "EDPB requires..."
- Clearly distinguish statutory requirements from guidance-level best practices.
- When guidance informs compliance, state: "While not legally binding, [guidance body] recommends..."
---
Pattern 4: Outdated Legal References
Description
AI models cite provisions that have been repealed, superseded, or substantially amended. The model's training data includes historical versions of laws that are no longer in force.
Why It Happens
Training corpora contain legal texts from many time periods. Models may cite the Data Protection Directive (95/46/EC) rather than GDPR, or reference pre-amendment versions of statutes.
Examples
| AI-Generated Claim | Actual Reality | Error |
|---|---|---|
| "Under the Data Protection Directive 95/46/EC, Article 25..." | Directive 95/46/EC was repealed and replaced by GDPR in 2018 | Citing repealed legislation |
| "The Safe Harbor framework governs EU-US data transfers" | Safe Harbor was invalidated by Schrems I (2015); replaced by Privacy Shield, then by the EU-US Data Privacy Framework | Citing invalidated framework |
| "Under the FDA's Quality System Regulation (21 CFR 820)..." | QSR is being replaced by QMSR aligned with ISO 13485 | Citing regulation under active replacement |
| "PCI-DSS 3.2.1 requires..." | PCI-DSS 4.0 was released in 2022 with transition period | Citing superseded version |
| "Under the pre-amendment CCPA..." | CPRA amendments are in effect as of January 2023 | Citing pre-amendment version |
Detection Techniques
1. Check the date of the cited instrument. Older instruments are more likely to be superseded. 2. Search for "repealed by" or "amended by" on the official source. 3. Verify on the official database (EUR-Lex shows consolidated versions with amendment history). 4. Check for successor instruments -- most repeals come with replacement legislation. 5. Verify version numbers for standards (ISO, PCI-DSS, NIST).
Prevention Strategies
- Always verify that the cited provision is current and in force.
- Use consolidated texts from official sources (EUR-Lex, eCFR).
- When citing standards, specify the version number and date.
- Note when a provision is in transition (e.g., "QSR/QMSR transition period").
---
Pattern 5: Arithmetic Errors in Timeline Calculations
Description
AI models make mathematical errors when calculating dates, deadlines, and timelines. This includes miscounting days, months, or years, and confusing business days with calendar days.
Why It Happens
AI models generate text token by token and do not perform actual arithmetic. Date calculations require calendar awareness that models lack. Models approximate rather than calculate.
Examples
| AI-Generated Claim | Actual Reality | Error |
|---|---|---|
| "72 hours from Friday 5pm is Monday 5pm" | 72 hours from Friday 5pm is Monday 5pm -- correct in this case, but models often get this wrong with holidays | May not account for time zones or holidays |
| "18 months from January 2025 is July 2026" | 18 months from January 2025 is July 2026 -- correct | Verify independently; models often miscalculate |
| "4% of EUR 50 billion is EUR 200 million" | 4% of EUR 50 billion is EUR 2 billion | Order of magnitude error |
| "The deadline is 60 business days from notification" | Statute says "60 days" (calendar days), not business days | Calendar vs business day confusion |
| "3 years from entry into force (2024) means compliance by 2026" | 3 years from 2024 is 2027 | Basic arithmetic error |
Detection Techniques
1. Recalculate every timeline independently using a calendar. 2. Verify whether the statute specifies calendar days or business days. 3. Check for time zone issues in cross-border timelines. 4. Verify percentage calculations by doing the math. 5. Watch for order-of-magnitude errors (million vs billion, days vs months). 6. Cross-reference calculated dates against published official timelines.
Prevention Strategies
- Calculate all dates manually and verify against the statutory text.
- Present timelines in table format with start date, rule, and calculated end date.
- Specify whether days are calendar or business days.
- Use specific dates rather than relative calculations when possible.
---
Detection Quick Reference
| What to Check | How to Check | Tool |
|---|---|---|
| Article numbers | Cross-reference against official statute | EUR-Lex, eCFR, official code |
| Dates | Verify against Official Journal or statute text | Official gazette database |
| Authority level | Confirm binding vs advisory status | Check source type and issuing body |
| Currency of law | Check for amendments and repeals | Consolidated text databases |
| Arithmetic | Recalculate independently | Calculator, calendar |
| Entity names | Verify official name and jurisdiction | Organization's official website |
| Penalty amounts | Cross-reference against penalty provision | Statutory text |
---
Quality Score Rubric
| Score | Rating | Criteria |
|---|---|---|
| 5/5 | Distribution Ready | Zero CRITICAL or HIGH findings. All citations verified against official sources. All dates confirmed. All arithmetic checked. Complete disclaimers. Speculation properly qualified. Suitable for client distribution. |
| 4/5 | Minor Revisions | Zero CRITICAL findings. Maximum 1-2 HIGH findings with clear, simple fixes. Most citations verified. Minor imprecisions in dates or numbers. Adequate disclaimers. Safe to distribute after targeted corrections. |
| 3/5 | Moderate Revisions | Zero CRITICAL findings. 3-5 HIGH findings. Some unverified citations. Some dates or numbers not confirmed. Incomplete disclaimers. Needs focused revision before distribution. |
| 2/5 | Major Revisions | 1 or more CRITICAL findings (wrong article numbers, materially incorrect claims). Multiple HIGH findings. Significant unverified content. Missing key disclaimers. Not safe to distribute. |
| 1/5 | Not Distribution Ready | Multiple CRITICAL findings. Pervasive inaccuracies across the document. Hallucination patterns evident throughout. No disclaimers. Requires complete rework from verified sources. |
Score Decision Tree
Are there any CRITICAL findings?
├── YES → Score 2 or 1 (depending on count and pervasiveness)
└── NO → Count HIGH findings
├── 0 HIGH → Score 5 (if all verified) or 4 (if minor gaps)
├── 1-2 HIGH → Score 4 (if simple fixes) or 3 (if complex)
└── 3+ HIGH → Score 3 or 2 (depending on severity and scope)---
Adversarial Review Principles
The Red Team Mindset
| Principle | Application |
|---|---|
| You are the opponent | Your job is to find every weakness, not to confirm the document is good |
| Assume the worst | Every unverified claim is wrong until proven right |
| Follow the money | Errors in financial calculations, penalties, and deadlines cause the most harm |
| Check the basics first | The most obvious facts (dates, names, article numbers) are where most errors hide |
| Trust nothing | AI confidence is not correlated with accuracy; the most confident claims are often the most wrong |
| Document everything | Every finding, every verification, every source -- create a complete audit trail |
| Think like the reader | What would a reader who trusts this document do? What harm would an error cause? |
Verification Prioritization
When time is limited, verify in this order:
| Priority | Category | Rationale |
|---|---|---|
| 1 | Legal citations (article numbers) | Most likely to be hallucinated; creates false obligations |
| 2 | Dates and deadlines | Wrong deadlines cause compliance failures |
| 3 | Penalty amounts | Wrong penalties cause incorrect risk assessment |
| 4 | Binding vs non-binding characterization | Confusing guidance for law wastes resources or creates false security |
| 5 | Arithmetic | Calculation errors compound through analysis |
| 6 | Entity names and jurisdictions | Less likely to cause direct harm but undermines credibility |
Verification Methodology Reference
Complete adversarial verification framework for AI-generated legal content. Covers the 6-step methodology, source hierarchy, citation validation procedures, and verification statistics.
---
Table of Contents
- Adversarial Mindset Principles
- Six-Step Verification Methodology
- Source Hierarchy for Legal Verification
- Citation Validation Procedures
- Cross-Referencing Techniques
- Verification Statistics Tracking
- Common Verification Failures
---
Adversarial Mindset Principles
The core principle: assume error until proven correct. AI-generated legal content is inherently unreliable until verified.
| Principle | Description |
|---|---|
| Assume error | Every factual claim, citation, date, and number is wrong until you verify it |
| Seek contradictory evidence | Do not just confirm -- actively try to disprove claims |
| Question every number | Dates, amounts, percentages, and article numbers are the highest-risk elements |
| Demand sources | Any claim without a verifiable source is suspect |
| Test logical consistency | Do the claims in the document make sense together? Do timelines add up? |
| Beware of confidence | AI models are most dangerous when they sound most certain |
| Check the obvious | The most basic facts (names, dates, locations) are often where errors hide |
---
Six-Step Verification Methodology
Step 1: Initial Review
Goal: Identify all verifiable claims in the document.
| Element to Identify | What to Mark |
|---|---|
| Legal citations | Article numbers, section references, regulation names |
| Dates | Effective dates, deadlines, implementation timelines |
| Numbers | Monetary amounts, percentages, thresholds, penalties |
| Entity names | Organizations, agencies, courts, legislative bodies |
| Factual assertions | "The law requires...", "Companies must...", "The regulation provides..." |
| Predictive claims | "Courts will likely...", "It is expected that..." |
| Source attributions | "According to [source]...", "As stated by [authority]..." |
Output: A marked-up document with every verifiable element highlighted and categorized.
Step 2: Source Verification (ALWAYS Web Search)
Goal: Verify every factual claim against official sources.
Mandatory rule: Always attempt to verify from the official source. Do not rely on the AI's own confidence or on secondary summaries.
| Verification Target | Primary Source | Secondary Source |
|---|---|---|
| EU legislation text | EUR-Lex (eur-lex.europa.eu) | National transposition databases |
| EU implementation dates | Official Journal of the EU | European Commission website |
| US federal statute text | United States Code (uscode.house.gov) | Cornell LII (law.cornell.edu) |
| US federal regulation text | eCFR (ecfr.gov) | Federal Register (federalregister.gov) |
| UK legislation text | legislation.gov.uk | Explanatory notes |
| Court decisions | Official reporters, court websites | Westlaw, LexisNexis |
| Agency guidance | Agency official website (.gov, .europa.eu) | Published FAQs and guidance documents |
| International treaties | UN Treaty Collection | Government treaty databases |
| Penalty amounts | Statutory text (primary source) | Enforcement action press releases |
| Organizational facts | Organization's official website | Official filings and reports |
Process for each claim: 1. Identify the claim and its source attribution. 2. Locate the official source. 3. Read the relevant provision directly. 4. Compare the claim to the source text. 5. Record: Verified / Incorrect / Unverifiable / Partially Correct. 6. If incorrect, note the correct information and the source.
Step 3: Arithmetic Verification
Goal: Independently verify every calculation.
| Calculation Type | Verification Method |
|---|---|
| Timeline (days/months) | Count manually from start date to end date |
| Percentage of revenue | Recalculate from stated base figures |
| Fine ranges | Cross-reference against statutory text |
| Deadline calculations | Verify using the statute's own counting rules |
| Compounding | Recalculate step by step |
| Currency conversions | Use the date-specific exchange rate |
Common arithmetic errors in AI content:
| Error Type | Example |
|---|---|
| Off-by-one in month counting | "18 months from May 2025" stated as "November 2026" (should be November 2026 -- correct, but check carefully) |
| Wrong year in multi-year timeline | "3 years from 2024" stated as "2026" (should be 2027) |
| Percentage miscalculation | "4% of EUR 50 billion" stated as "EUR 200 million" (should be EUR 2 billion) |
| Mixing financial year and calendar year | Revenue calculated on wrong period |
| Inconsistent rounding | Numbers don't add up due to premature rounding |
Step 4: Citation Validation
Goal: Verify every legal citation exists and says what the document claims.
| Validation Step | Check |
|---|---|
| 1. Citation format | Is the citation in the correct format for its jurisdiction? |
| 2. Source exists | Does the cited statute, article, or section actually exist? |
| 3. Content accuracy | Does the cited provision say what the document claims? |
| 4. Currency | Is this the current, in-force version? Has it been amended or repealed? |
| 5. Authority level | Is the source characterized at the correct level (binding vs guidance)? |
| 6. Jurisdiction match | Does the citation apply to the jurisdiction discussed? |
Step 5: Speculation Identification
Goal: Distinguish fact from opinion, certainty from prediction.
| Category | Language Signals | Appropriate? |
|---|---|---|
| Fact | "The regulation requires..." | Yes, if verified |
| Qualified interpretation | "This likely means..." | Yes, if properly qualified |
| Prediction | "Courts will probably hold..." | Acceptable if clearly marked as prediction |
| Unqualified prediction | "This will be enforced starting..." | Not acceptable -- must add qualifier |
| Opinion stated as fact | "Companies must [non-statutory obligation]" | Not acceptable -- reframe or cite source |
| Guidance stated as law | "EDPB requires..." | Not acceptable -- clarify as guidance |
Step 6: Disclaimer Review
Goal: Verify all required disclaimers are present and adequate.
| Required Disclaimer | Minimum Content |
|---|---|
| Not legal advice | "This document does not constitute legal advice and should not be relied upon as such." |
| Jurisdiction limitations | "This analysis covers [specific jurisdictions]. Laws may differ in other jurisdictions." |
| Date of preparation | "Current as of [date]. Laws and regulations may have changed since this date." |
| Professional consultation | "Consult qualified legal counsel for advice specific to your circumstances." |
| AI-generated disclosure | "This document was generated/assisted by artificial intelligence." |
| Accuracy limitations | "While efforts have been made to ensure accuracy, independent verification is recommended." |
---
Source Hierarchy for Legal Verification
Tier 1: Primary Legislation (Highest Authority)
| Source Type | Examples | Verification Status |
|---|---|---|
| Constitution | US Constitution, EU Treaties | Definitive |
| Statutes | GDPR, AI Act, CCPA, Clean Air Act | Definitive when from official source |
| Regulations | CFR, EU implementing regulations | Definitive when from official source |
Tier 2: Official Interpretive Sources
| Source Type | Examples | Verification Status |
|---|---|---|
| Court decisions | Supreme Court, CJEU, High Court | Binding within jurisdiction |
| Agency decisions | FTC enforcement actions, DPA decisions | Authoritative within scope |
Tier 3: Authoritative Guidance
| Source Type | Examples | Verification Status |
|---|---|---|
| Agency guidance | EDPB guidelines, ICO guidance, FTC guidance | Persuasive; not binding |
| Implementing standards | ISO standards, NIST frameworks | Reference standards; not law |
| Official FAQs | European Commission FAQs, agency FAQs | Informative; can change |
Tier 4: Secondary Sources
| Source Type | Examples | Verification Status |
|---|---|---|
| Legal treatises | Practitioner texts, annotated codes | Scholarly; not authoritative |
| Law review articles | Academic legal journals | Scholarly; not authoritative |
| Legal news | Law firm alerts, legal publications | Informational; verify underlying claims |
Tier 5: Informal Sources (Lowest Reliability)
| Source Type | Examples | Verification Status |
|---|---|---|
| Blog posts | Law firm blogs, tech blogs | May contain errors; never cite as authority |
| News articles | General media coverage of legal topics | Frequently inaccurate on legal details |
| Social media | LinkedIn posts, Twitter/X threads | Unreliable; never cite |
| AI-generated summaries | ChatGPT, Gemini, Claude outputs | Must be independently verified |
---
Citation Validation Procedures
Bluebook (US) Citation Format
| Element | Format | Example |
|---|---|---|
| US Code | [Title] U.S.C. § [Section] | 15 U.S.C. § 7702 |
| CFR | [Title] C.F.R. § [Section] | 16 C.F.R. § 312.2 |
| Federal statute (session) | Pub. L. No. [Number], § [Section], [Statutes] | Pub. L. No. 117-328, § 101 |
| Court case | [Party] v. [Party], [Volume] [Reporter] [Page] ([Year]) | Brown v. Board, 347 U.S. 483 (1954) |
EU Citation Format
| Element | Format | Example |
|---|---|---|
| Regulation | Regulation (EU) [Year]/[Number] | Regulation (EU) 2016/679 |
| Directive | Directive (EU) [Year]/[Number] | Directive (EU) 2022/2555 |
| Article reference | Article N([letter]) | Article 83(5)(a) |
| CJEU case | Case C-[Number]/[Year] | Case C-311/18 (Schrems II) |
OSCOLA (UK) Citation Format
| Element | Format | Example |
|---|---|---|
| UK statute | [Short Title] [Year], s [Section] | Data Protection Act 2018, s 3 |
| SI | [Title] [Year] (SI [Year]/[Number]) | The GDPR (SI 2019/419) |
| Case | [Party] v [Party] [[Year]] [Court] [Number] | [2019] UKSC 22 |
---
Cross-Referencing Techniques
Triangulation
Verify claims using at least 2 independent sources.
| Method | Description |
|---|---|
| Source triangulation | Verify the same fact from 2+ independent sources |
| Method triangulation | Use different verification methods for the same claim |
| Temporal triangulation | Check if the claim was true at the stated date (not just currently) |
Consistency Checking
| Check | Method |
|---|---|
| Internal consistency | Do claims within the document contradict each other? |
| External consistency | Do claims match what other authoritative sources say? |
| Logical consistency | Do the conclusions follow from the premises? |
| Mathematical consistency | Do the numbers add up across the document? |
---
Verification Statistics Tracking
Track verification results for quality assessment.
| Metric | How to Calculate |
|---|---|
| Verification rate | Claims verified / Total claims identified |
| Accuracy rate | Claims verified correct / Claims verified |
| Citation accuracy | Citations confirmed / Citations checked |
| Date accuracy | Dates confirmed / Dates checked |
| Hallucination rate | Hallucinations found / Total claims |
| Severity distribution | Count of CRITICAL / HIGH / MODERATE / LOW findings |
Sample Verification Log
| # | Claim | Category | Source Checked | Result | Severity |
|---|---|---|---|---|---|
| 1 | "GDPR Art. 83(5) fines up to EUR 20M" | Citation | EUR-Lex | Verified | -- |
| 2 | "AI Act applies from August 2026" | Date | EUR-Lex | Partially correct (phased) | MODERATE |
| 3 | "ENISA requires annual testing" | Authority | ENISA website | Incorrect (guidance, not requirement) | HIGH |
---
Common Verification Failures
| Failure Mode | Why It Happens | Prevention |
|---|---|---|
| Trusting AI confidence | Model sounds certain, so reviewer assumes correctness | Always verify regardless of AI's confidence level |
| Checking only flagged items | Automated tools miss contextual errors | Manually review a random sample of unflagged claims |
| Verifying against another AI | Using one AI to fact-check another | Always verify against official, non-AI sources |
| Stopping at first confirmation | Found one source supporting the claim; stopped looking | Check at least 2 independent sources for important claims |
| Ignoring absence of evidence | Could not find supporting source; assumed claim is too obscure | If you cannot verify it, flag it as unverified |
| Time pressure shortcuts | Deadline pressure reduces verification thoroughness | Budget adequate time; prioritize CRITICAL items if time is limited |
#!/usr/bin/env python3
"""
Legal Fact Checker
Scans legal text for verifiable claims: article/section references, dates,
numerical values, monetary amounts, entity names. Cross-references article
numbering against known statute structures. Flags potential hallucination
patterns. Outputs a verification report with confidence levels.
Usage:
python legal_fact_checker.py --input document.txt
python legal_fact_checker.py --text "Under GDPR Article 83(5), fines..."
python legal_fact_checker.py --input document.txt --json
python legal_fact_checker.py --input document.txt --output report.json
"""
import argparse
import json
import re
import sys
from typing import Any, Dict, List, Optional, Tuple
# Known statute structures for cross-referencing
KNOWN_STATUTES: Dict[str, Dict[str, Any]] = {
"GDPR": {
"full_name": "Regulation (EU) 2016/679",
"max_article": 99,
"key_articles": {
5: "Principles", 6: "Lawful basis", 7: "Consent", 9: "Special categories",
12: "Transparency", 13: "Info at collection", 14: "Info not from data subject",
15: "Right of access", 16: "Rectification", 17: "Erasure",
20: "Portability", 21: "Object", 22: "Automated decisions",
25: "Data protection by design", 28: "Processor", 30: "Records",
32: "Security", 33: "Breach notification authority", 34: "Breach notification data subject",
35: "DPIA", 37: "DPO designation", 44: "Transfer principles",
83: "Fines", 99: "Entry into force",
},
"patterns": [r"\bGDPR\b", r"\bRegulation\s*\(?EU\)?\s*2016/679\b"],
},
"EU_AI_ACT": {
"full_name": "Regulation (EU) 2024/1689",
"max_article": 113,
"key_articles": {
3: "Definitions", 5: "Prohibited practices", 6: "Classification rules",
9: "Risk management", 10: "Data governance", 11: "Technical documentation",
13: "Transparency", 14: "Human oversight", 15: "Accuracy/robustness",
50: "Transparency obligations", 51: "GPAI providers",
52: "GPAI systemic risk", 72: "Post-market monitoring",
83: "Penalties", 99: "Codes of practice", 113: "Entry into force",
},
"patterns": [r"\bAI\s+Act\b", r"\bRegulation\s*\(?EU\)?\s*2024/1689\b"],
},
"NIS2": {
"full_name": "Directive (EU) 2022/2555",
"max_article": 46,
"key_articles": {
6: "Definitions", 7: "National strategy", 21: "Risk management measures",
23: "Reporting obligations", 26: "Jurisdiction", 34: "Penalties",
},
"patterns": [r"\bNIS\s*2\b", r"\bDirective\s*\(?EU\)?\s*2022/2555\b"],
},
"DORA": {
"full_name": "Regulation (EU) 2022/2554",
"max_article": 64,
"key_articles": {
3: "Definitions", 5: "Governance", 6: "ICT risk framework",
17: "ICT incident reporting", 24: "Testing", 28: "Third-party risk",
},
"patterns": [r"\bDORA\b", r"\bRegulation\s*\(?EU\)?\s*2022/2554\b"],
},
}
# Citation extraction patterns
CITATION_PATTERNS = [
("eu_article", re.compile(
r"(?:Article|Art\.?)\s+(\d+)(?:\((\d+)\))?(?:\(([a-z])\))?"
)),
("us_section", re.compile(
r"(?:Section|Sec\.?|§)\s*(\d+(?:\.\d+)?)(?:\(([a-z0-9]+)\))?"
)),
("eu_regulation", re.compile(
r"(?:Regulation|Directive)\s+\(?(?:EU|EC)\)?\s*(?:No\.?\s*)?(\d{4})/(\d+)"
)),
("us_cfr", re.compile(
r"(\d+)\s+(?:C\.?F\.?R\.?)\s+(?:§\s*)?(\d+(?:\.\d+)?)"
)),
("us_usc", re.compile(
r"(\d+)\s+U\.?S\.?C\.?\s+(?:§\s*)?(\d+)"
)),
("recital", re.compile(
r"(?:Recital|recital)\s+\(?(\d+)\)?"
)),
("annex", re.compile(
r"(?:Annex|Schedule|Appendix)\s+([IVXLCDM]+|\d+)"
)),
]
# Date extraction
DATE_PATTERN = re.compile(
r"\b(\d{1,2}[/-]\d{1,2}[/-]\d{2,4}|\d{4}[/-]\d{1,2}[/-]\d{1,2}|"
r"(?:January|February|March|April|May|June|July|August|September|October|November|December)"
r"\s+\d{1,2},?\s+\d{4}|\d{1,2}\s+"
r"(?:January|February|March|April|May|June|July|August|September|October|November|December)"
r"\s+\d{4})\b",
re.IGNORECASE,
)
# Monetary amounts
MONEY_PATTERN = re.compile(
r"(?:EUR|USD|GBP)?\s*[€$£]?\s*(\d[\d,]*(?:\.\d{1,2})?)\s*"
r"(?:million|billion|thousand|[MBK])?\s*(?:EUR|USD|GBP)?",
re.IGNORECASE,
)
# Hallucination flag patterns
HALLUCINATION_FLAGS = [
{
"id": "H1",
"name": "Plausible but suspicious article number",
"pattern": re.compile(r"(?:Article|Art\.?)\s+(\d+)\((\d+)\)", re.IGNORECASE),
"check": "high_subsection",
"description": "Article with high subsection number (>6) may not exist",
},
{
"id": "H2",
"name": "Confident implementation date",
"pattern": re.compile(
r"(?:will\s+(?:apply|enter\s+into\s+force|become\s+applicable)|effective|deadline)\s+"
r"(?:from|on|by)\s+\d{1,2}\s+\w+\s+\d{4}",
re.IGNORECASE,
),
"check": "date_confidence",
"description": "Specific implementation date stated with high confidence -- verify against official timeline",
},
{
"id": "H3",
"name": "Guidance stated as requirement",
"pattern": re.compile(
r"(?:(?:ENISA|EDPB|CNIL|ICO|FTC|NIST)\s+(?:requires?|mandates?|obligates?))|"
r"(?:guidance|recommendation|guideline|best\s+practice)\s+(?:requires?|mandates?|must)",
re.IGNORECASE,
),
"check": "guidance_as_law",
"description": "Non-binding guidance or recommendations stated as binding legal requirements",
},
{
"id": "H4",
"name": "Outdated reference indicator",
"pattern": re.compile(
r"(?:Directive\s+95/46|Data\s+Protection\s+Directive|Safe\s+Harbor)|"
r"(?:repealed|superseded|replaced\s+by|no\s+longer\s+in\s+(?:force|effect))",
re.IGNORECASE,
),
"check": "outdated_ref",
"description": "Reference to potentially repealed or superseded legal instrument",
},
{
"id": "H5",
"name": "Arithmetic in timeline",
"pattern": re.compile(
r"(\d+)\s+(?:days?|months?|years?)\s+(?:from|after|before|until)\s+",
re.IGNORECASE,
),
"check": "timeline_arithmetic",
"description": "Timeline calculation that should be independently verified",
},
]
# Speculation language patterns
SPECULATION_PATTERNS = [
(re.compile(r"\b(?:will\s+likely|is\s+likely\s+to|probably|may\s+well)\b", re.I), "prediction"),
(re.compile(r"\b(?:it\s+is\s+(?:expected|anticipated|believed))\b", re.I), "expectation"),
(re.compile(r"\b(?:should\s+be\s+(?:interpreted|understood|read))\b", re.I), "interpretation"),
(re.compile(r"\b(?:in\s+(?:our|my)\s+(?:view|opinion|assessment))\b", re.I), "opinion"),
(re.compile(r"\b(?:courts?\s+(?:will|would|might|could)\s+(?:likely|probably))\b", re.I), "judicial_prediction"),
]
def extract_sentences(text: str) -> List[str]:
"""Split text into sentences."""
abbrevs = ["Art.", "Sec.", "No.", "para.", "e.g.", "i.e.", "et al.", "cf.", "v.", "Ltd.", "Inc."]
protected = text
for abbr in abbrevs:
protected = protected.replace(abbr, abbr.replace(".", "<DOT>"))
sentences = re.split(r'(?<=[.!?])\s+', protected)
return [s.replace("<DOT>", ".").strip() for s in sentences if s.strip()]
def identify_statutes(text: str) -> List[str]:
"""Identify which known statutes are referenced in the text."""
found = []
for statute_key, info in KNOWN_STATUTES.items():
for pat in info["patterns"]:
if re.search(pat, text, re.IGNORECASE):
found.append(statute_key)
break
return found
def extract_citations(text: str) -> List[Dict[str, Any]]:
"""Extract all legal citations from text."""
citations = []
sentences = extract_sentences(text)
for sent in sentences:
for cit_type, pattern in CITATION_PATTERNS:
for m in pattern.finditer(sent):
citation = {
"type": cit_type,
"full_match": m.group(),
"groups": [g for g in m.groups() if g is not None],
"context": sent.strip()[:200],
"verification_status": "unverified",
"flags": [],
}
citations.append(citation)
return citations
def validate_citations(citations: List[Dict], referenced_statutes: List[str]) -> List[Dict]:
"""Validate citations against known statute structures."""
for citation in citations:
if citation["type"] == "eu_article" and citation["groups"]:
article_num = int(citation["groups"][0])
subsection = int(citation["groups"][1]) if len(citation["groups"]) > 1 else None
for statute_key in referenced_statutes:
info = KNOWN_STATUTES.get(statute_key, {})
max_art = info.get("max_article", 999)
if article_num > max_art:
citation["flags"].append({
"severity": "CRITICAL",
"message": f"Article {article_num} exceeds maximum article ({max_art}) "
f"in {info.get('full_name', statute_key)}",
})
citation["verification_status"] = "likely_invalid"
elif article_num in info.get("key_articles", {}):
citation["verification_status"] = "plausible"
citation["known_topic"] = info["key_articles"][article_num]
if subsection and subsection > 6:
citation["flags"].append({
"severity": "HIGH",
"message": f"Subsection ({subsection}) unusually high -- verify existence",
})
return citations
def check_hallucination_patterns(text: str) -> List[Dict[str, Any]]:
"""Check for known AI hallucination patterns."""
flags = []
sentences = extract_sentences(text)
for sent in sentences:
for h in HALLUCINATION_FLAGS:
for m in h["pattern"].finditer(sent):
flag = {
"pattern_id": h["id"],
"pattern_name": h["name"],
"matched_text": m.group(),
"context": sent.strip()[:200],
"description": h["description"],
"action": "Verify against official source",
}
# Additional checks per pattern type
if h["check"] == "high_subsection":
groups = m.groups()
if len(groups) > 1 and groups[1]:
sub = int(groups[1])
if sub > 6:
flag["severity"] = "HIGH"
else:
flag["severity"] = "MODERATE"
else:
flag["severity"] = "MODERATE"
elif h["check"] == "guidance_as_law":
flag["severity"] = "HIGH"
elif h["check"] == "outdated_ref":
flag["severity"] = "HIGH"
elif h["check"] == "date_confidence":
flag["severity"] = "MODERATE"
elif h["check"] == "timeline_arithmetic":
flag["severity"] = "MODERATE"
else:
flag["severity"] = "MODERATE"
flags.append(flag)
return flags
def extract_dates_with_context(text: str) -> List[Dict[str, str]]:
"""Extract dates for verification."""
dates = []
sentences = extract_sentences(text)
for sent in sentences:
for m in DATE_PATTERN.finditer(sent):
dates.append({
"date": m.group().strip(),
"context": sent.strip()[:200],
"status": "requires_verification",
})
return dates
def detect_speculation(text: str) -> List[Dict[str, str]]:
"""Detect speculative or opinion language."""
speculations = []
sentences = extract_sentences(text)
for sent in sentences:
for pattern, spec_type in SPECULATION_PATTERNS:
for m in pattern.finditer(sent):
speculations.append({
"type": spec_type,
"matched_text": m.group(),
"context": sent.strip()[:200],
"action": "Verify if stated as fact or properly qualified",
})
return speculations
def check_disclaimers(text: str) -> Dict[str, Any]:
"""Check for required disclaimers."""
text_lower = text.lower()
checks = {
"not_legal_advice": bool(re.search(
r"(?:not\s+(?:constitute|intended\s+as)\s+legal\s+advice|"
r"does\s+not\s+constitute\s+legal\s+advice|"
r"for\s+informational\s+purposes\s+only)",
text_lower,
)),
"jurisdiction_limitation": bool(re.search(
r"(?:jurisdiction|applicable\s+(?:in|to)|"
r"may\s+(?:vary|differ)\s+(?:by|across)\s+jurisdiction)",
text_lower,
)),
"date_of_preparation": bool(re.search(
r"(?:as\s+of|prepared\s+(?:on|as\s+of)|current\s+as\s+of|"
r"last\s+(?:updated|reviewed))",
text_lower,
)),
"professional_consultation": bool(re.search(
r"(?:consult\s+(?:a\s+)?(?:qualified\s+)?(?:legal\s+)?(?:counsel|lawyer|attorney|professional|adviser)|"
r"seek\s+(?:legal\s+)?(?:advice|counsel))",
text_lower,
)),
"ai_generated_disclosure": bool(re.search(
r"(?:ai[- ]generated|generated\s+(?:by|using)\s+ai|"
r"artificial\s+intelligence|machine\s+generated|"
r"assisted\s+by\s+ai)",
text_lower,
)),
"accuracy_limitations": bool(re.search(
r"(?:accuracy\s+(?:not\s+)?guaranteed|verify\s+(?:independently|against)|"
r"no\s+(?:warranty|guarantee)\s+(?:of|as\s+to)\s+accuracy)",
text_lower,
)),
}
present = sum(1 for v in checks.values() if v)
checks["score"] = f"{present}/6"
checks["adequate"] = present >= 4
return checks
def compute_summary(
citations: List, hallucination_flags: List, dates: List,
speculations: List, disclaimers: Dict
) -> Dict[str, Any]:
"""Compute verification summary."""
severity_counts = {"CRITICAL": 0, "HIGH": 0, "MODERATE": 0, "LOW": 0}
for cit in citations:
for flag in cit.get("flags", []):
sev = flag.get("severity", "MODERATE")
severity_counts[sev] = severity_counts.get(sev, 0) + 1
for h in hallucination_flags:
sev = h.get("severity", "MODERATE")
severity_counts[sev] = severity_counts.get(sev, 0) + 1
if not disclaimers.get("adequate"):
severity_counts["MODERATE"] += 1
return {
"total_citations": len(citations),
"total_dates": len(dates),
"total_speculations": len(speculations),
"total_hallucination_flags": len(hallucination_flags),
"findings_by_severity": severity_counts,
"disclaimer_score": disclaimers.get("score", "0/6"),
}
def format_human_report(result: Dict[str, Any]) -> str:
"""Format as human-readable report."""
lines = []
lines.append("=" * 72)
lines.append("LEGAL FACT-CHECK REPORT")
lines.append("=" * 72)
s = result["summary"]
lines.append(f"\nCitations found: {s['total_citations']}")
lines.append(f"Dates found: {s['total_dates']}")
lines.append(f"Speculation instances: {s['total_speculations']}")
lines.append(f"Hallucination flags: {s['total_hallucination_flags']}")
lines.append(f"Disclaimer score: {s['disclaimer_score']}")
sev = s["findings_by_severity"]
lines.append(f"\nFindings: CRITICAL={sev['CRITICAL']} HIGH={sev['HIGH']} "
f"MODERATE={sev['MODERATE']} LOW={sev['LOW']}")
if result["hallucination_flags"]:
lines.append(f"\n--- HALLUCINATION FLAGS ({len(result['hallucination_flags'])}) ---")
for h in result["hallucination_flags"]:
lines.append(f"\n [{h['severity']}] {h['pattern_name']}")
lines.append(f" Matched: {h['matched_text']}")
lines.append(f" Context: {h['context'][:120]}")
lines.append(f" Action: {h['action']}")
flagged_citations = [c for c in result["citations"] if c.get("flags")]
if flagged_citations:
lines.append(f"\n--- FLAGGED CITATIONS ({len(flagged_citations)}) ---")
for c in flagged_citations:
lines.append(f"\n Citation: {c['full_match']}")
lines.append(f" Status: {c['verification_status']}")
for f in c["flags"]:
lines.append(f" [{f['severity']}] {f['message']}")
if result["speculations"]:
lines.append(f"\n--- SPECULATION ({len(result['speculations'])}) ---")
for sp in result["speculations"]:
lines.append(f" [{sp['type']}] {sp['matched_text']}")
lines.append(f" {sp['context'][:120]}")
d = result["disclaimers"]
lines.append(f"\n--- DISCLAIMERS ---")
for key in ["not_legal_advice", "jurisdiction_limitation", "date_of_preparation",
"professional_consultation", "ai_generated_disclosure", "accuracy_limitations"]:
status = "PRESENT" if d.get(key) else "MISSING"
lines.append(f" {key:30s} {status}")
lines.append(f" Adequate: {'YES' if d.get('adequate') else 'NO'}")
lines.append("\n" + "=" * 72)
return "\n".join(lines)
def main() -> None:
parser = argparse.ArgumentParser(
description="Scan legal text for verifiable claims and flag hallucination patterns."
)
parser.add_argument("--input", "-i", type=str, help="Path to legal document")
parser.add_argument("--text", "-t", type=str, help="Inline legal text")
parser.add_argument("--output", "-o", type=str, help="Path to save output (JSON)")
parser.add_argument("--json", action="store_true", help="Output in JSON format")
args = parser.parse_args()
if not args.input and not args.text:
parser.print_help()
sys.exit(1)
try:
if args.input:
with open(args.input, "r", encoding="utf-8") as f:
text = f.read()
else:
text = args.text
if not text or not text.strip():
print("Error: Empty input text.", file=sys.stderr)
sys.exit(1)
referenced_statutes = identify_statutes(text)
citations = extract_citations(text)
citations = validate_citations(citations, referenced_statutes)
hallucination_flags = check_hallucination_patterns(text)
dates = extract_dates_with_context(text)
speculations = detect_speculation(text)
disclaimers = check_disclaimers(text)
summary = compute_summary(citations, hallucination_flags, dates, speculations, disclaimers)
result = {
"referenced_statutes": referenced_statutes,
"citations": citations,
"hallucination_flags": hallucination_flags,
"dates": dates,
"speculations": speculations,
"disclaimers": disclaimers,
"summary": summary,
}
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
json.dump(result, f, indent=2, ensure_ascii=False)
print(f"Report saved to {args.output}")
elif args.json:
print(json.dumps(result, indent=2, ensure_ascii=False))
else:
print(format_human_report(result))
except FileNotFoundError:
print(f"Error: File not found: {args.input}", file=sys.stderr)
sys.exit(1)
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Legal Quality Scorer
Scores legal document quality across 6 verification categories: Factual
Accuracy, Legal Authority Citations, Arithmetic Validation, Source
Verification, Speculation Detection, and Disclaimer Adequacy. Calculates
composite quality score and distribution readiness.
Usage:
python legal_quality_scorer.py --input document.txt
python legal_quality_scorer.py --input document.txt --json
python legal_quality_scorer.py --input document.txt --verbose
python legal_quality_scorer.py --input document.txt --output assessment.json
"""
import argparse
import json
import re
import sys
from typing import Any, Dict, List, Tuple
# Scoring weights for each category
CATEGORY_WEIGHTS = {
"factual_accuracy": 0.25,
"legal_authority": 0.25,
"arithmetic": 0.15,
"source_verification": 0.15,
"speculation_control": 0.10,
"disclaimer_adequacy": 0.10,
}
QUALITY_RATINGS = {
5: {"label": "Distribution Ready", "description": "Safe to distribute. Zero CRITICAL/HIGH issues."},
4: {"label": "Minor Revisions", "description": "Safe after small fixes. Zero CRITICAL, 1-2 HIGH."},
3: {"label": "Moderate Revisions", "description": "Needs work before distribution. 3+ HIGH issues."},
2: {"label": "Major Revisions", "description": "Not safe to distribute. 1+ CRITICAL issues."},
1: {"label": "Not Distribution Ready", "description": "Requires complete rework. Multiple CRITICAL."},
}
# Pattern sets for scoring
CITATION_PATTERN = re.compile(
r"(?:Article|Art\.?|Section|Sec\.?|§)\s*\d+(?:\(\d+\))?(?:\([a-z]\))?",
re.IGNORECASE,
)
SPECIFIC_CITATION_PATTERN = re.compile(
r"(?:Regulation|Directive)\s+\(?(?:EU|EC)\)?\s*(?:No\.?\s*)?\d{4}/\d+|"
r"\d+\s+(?:C\.?F\.?R\.?|U\.?S\.?C\.?)\s+(?:§\s*)?\d+",
re.IGNORECASE,
)
DATE_PATTERN = re.compile(
r"\b(?:\d{1,2}[/-]\d{1,2}[/-]\d{2,4}|\d{4}[/-]\d{1,2}[/-]\d{1,2}|"
r"(?:January|February|March|April|May|June|July|August|September|October|November|December)"
r"\s+\d{1,2},?\s+\d{4})\b",
re.IGNORECASE,
)
NUMBER_PATTERN = re.compile(
r"(?:EUR|USD|GBP)?\s*[€$£]?\s*\d[\d,]*(?:\.\d+)?\s*"
r"(?:million|billion|thousand|%|percent|days?|months?|years?|hours?)?\b",
re.IGNORECASE,
)
SPECULATION_PATTERNS = [
re.compile(r"\b(?:will\s+likely|is\s+likely\s+to|probably|may\s+well)\b", re.I),
re.compile(r"\b(?:it\s+is\s+(?:expected|anticipated|believed))\b", re.I),
re.compile(r"\b(?:should\s+be\s+(?:interpreted|understood))\b", re.I),
re.compile(r"\b(?:in\s+(?:our|my)\s+(?:view|opinion|assessment))\b", re.I),
re.compile(r"\b(?:courts?\s+(?:will|would|might|could)\s+(?:likely|probably))\b", re.I),
]
CERTAINTY_PATTERNS = [
re.compile(r"\b(?:certainly|definitely|undoubtedly|clearly|obviously|always|never)\b", re.I),
re.compile(r"\b(?:there\s+is\s+no\s+doubt)\b", re.I),
re.compile(r"\b(?:it\s+is\s+(?:certain|clear|obvious|evident))\b", re.I),
]
HEDGING_PATTERNS = [
re.compile(r"\b(?:may|might|could|potentially|possibly|arguably)\b", re.I),
re.compile(r"\b(?:it\s+(?:appears|seems)\s+(?:that|to))\b", re.I),
re.compile(r"\b(?:subject\s+to\s+(?:interpretation|debate))\b", re.I),
]
DISCLAIMER_CHECKS = {
"not_legal_advice": re.compile(
r"(?:not\s+(?:constitute|intended\s+as)\s+legal\s+advice|"
r"for\s+informational\s+purposes\s+only)", re.I
),
"jurisdiction_limitation": re.compile(
r"(?:jurisdiction|applicable\s+(?:in|to)|"
r"may\s+(?:vary|differ)\s+(?:by|across)\s+jurisdiction)", re.I
),
"date_of_preparation": re.compile(
r"(?:as\s+of|prepared\s+(?:on|as\s+of)|current\s+as\s+of|"
r"last\s+(?:updated|reviewed))", re.I
),
"professional_consultation": re.compile(
r"(?:consult\s+(?:a\s+)?(?:qualified\s+)?(?:legal\s+)?(?:counsel|lawyer|attorney|professional)|"
r"seek\s+(?:legal\s+)?(?:advice|counsel))", re.I
),
"ai_generated_disclosure": re.compile(
r"(?:ai[- ]generated|generated\s+(?:by|using)\s+ai|"
r"artificial\s+intelligence|assisted\s+by\s+ai)", re.I
),
"accuracy_limitations": re.compile(
r"(?:accuracy\s+(?:not\s+)?guaranteed|verify\s+(?:independently|against)|"
r"no\s+(?:warranty|guarantee))", re.I
),
}
# High-subsection article pattern (hallucination indicator)
HIGH_SUBSECTION = re.compile(r"(?:Article|Art\.?)\s+\d+\(([7-9]|\d{2,})\)", re.I)
# Guidance-as-law pattern
GUIDANCE_AS_LAW = re.compile(
r"(?:(?:ENISA|EDPB|CNIL|ICO|FTC|NIST)\s+(?:requires?|mandates?|obligates?))|"
r"(?:guidance|recommendation|guideline|best\s+practice)\s+(?:requires?|mandates?|must)",
re.I,
)
def extract_sentences(text: str) -> List[str]:
"""Split into sentences."""
abbrevs = ["Art.", "Sec.", "No.", "para.", "e.g.", "i.e.", "et al.", "cf.", "v."]
protected = text
for abbr in abbrevs:
protected = protected.replace(abbr, abbr.replace(".", "<DOT>"))
sentences = re.split(r'(?<=[.!?])\s+', protected)
return [s.replace("<DOT>", ".").strip() for s in sentences if s.strip()]
def score_factual_accuracy(text: str, sentences: List[str]) -> Tuple[int, List[Dict]]:
"""Score factual accuracy (dates, numbers, entities)."""
findings = []
deductions = 0
# Check for high-subsection articles (hallucination pattern)
for m in HIGH_SUBSECTION.finditer(text):
findings.append({
"severity": "HIGH",
"category": "factual_accuracy",
"detail": f"Unusually high subsection number: {m.group()}",
"action": "Verify article subsection exists in source statute",
})
deductions += 1
# Check for guidance stated as binding law
for m in GUIDANCE_AS_LAW.finditer(text):
findings.append({
"severity": "HIGH",
"category": "factual_accuracy",
"detail": f"Non-binding guidance possibly stated as requirement: {m.group()}",
"action": "Verify if cited source is binding legislation vs guidance",
})
deductions += 1
# Check for overcertainty about predictions
for sent in sentences:
for p in CERTAINTY_PATTERNS:
if p.search(sent):
findings.append({
"severity": "MODERATE",
"category": "factual_accuracy",
"detail": f"Overcertain language: {sent[:100]}",
"action": "Verify claim or soften language",
})
deductions += 0.5
break
score = max(1, 5 - deductions)
return min(5, round(score)), findings
def score_legal_authority(text: str, sentences: List[str]) -> Tuple[int, List[Dict]]:
"""Score citation quality and presence."""
findings = []
citations = CITATION_PATTERN.findall(text)
specific_citations = SPECIFIC_CITATION_PATTERN.findall(text)
word_count = len(text.split())
# Citation density
citation_density = len(citations) / max(word_count / 500, 1)
if len(citations) == 0 and word_count > 200:
findings.append({
"severity": "CRITICAL",
"category": "legal_authority",
"detail": "No legal citations found in document",
"action": "Add specific statutory references for all legal claims",
})
return 1, findings
if citation_density < 0.5 and word_count > 500:
findings.append({
"severity": "HIGH",
"category": "legal_authority",
"detail": f"Low citation density: {len(citations)} citations in {word_count} words",
"action": "Add more specific citations to support legal claims",
})
if len(specific_citations) == 0 and word_count > 300:
findings.append({
"severity": "MODERATE",
"category": "legal_authority",
"detail": "No full statute references (e.g., Regulation (EU) 2024/1689)",
"action": "Include complete statute references for traceability",
})
deductions = 0
for f in findings:
if f["severity"] == "CRITICAL":
deductions += 3
elif f["severity"] == "HIGH":
deductions += 1.5
elif f["severity"] == "MODERATE":
deductions += 0.5
return max(1, min(5, round(5 - deductions))), findings
def score_arithmetic(text: str, sentences: List[str]) -> Tuple[int, List[Dict]]:
"""Score arithmetic and numerical content."""
findings = []
numbers = NUMBER_PATTERN.findall(text)
dates = DATE_PATTERN.findall(text)
# Flag any timeline calculations for verification
timeline_pattern = re.compile(
r"(\d+)\s+(?:days?|months?|years?)\s+(?:from|after|before|until)\s+",
re.IGNORECASE,
)
for m in timeline_pattern.finditer(text):
findings.append({
"severity": "MODERATE",
"category": "arithmetic",
"detail": f"Timeline calculation requires verification: {m.group()[:80]}",
"action": "Independently calculate and verify",
})
# Flag percentage calculations
pct_pattern = re.compile(r"(\d+(?:\.\d+)?)\s*%\s*of\s+", re.I)
for m in pct_pattern.finditer(text):
findings.append({
"severity": "LOW",
"category": "arithmetic",
"detail": f"Percentage calculation: {m.group()[:80]}",
"action": "Verify arithmetic",
})
# Score based on findings
high_count = sum(1 for f in findings if f["severity"] in ("CRITICAL", "HIGH"))
mod_count = sum(1 for f in findings if f["severity"] == "MODERATE")
if high_count > 0:
score = max(1, 3 - high_count)
elif mod_count > 3:
score = 3
elif mod_count > 0:
score = 4
else:
score = 5
return score, findings
def score_source_verification(text: str, sentences: List[str]) -> Tuple[int, List[Dict]]:
"""Score source verifiability."""
findings = []
# Check for unsourced definitive claims
definitive_claims = re.compile(
r"(?:the\s+law\s+(?:requires|mandates|prohibits))|"
r"(?:(?:companies|organizations|entities)\s+(?:must|shall|are\s+required))",
re.I,
)
citations_present = len(CITATION_PATTERN.findall(text)) > 0
claim_count = 0
for sent in sentences:
if definitive_claims.search(sent) and not CITATION_PATTERN.search(sent):
claim_count += 1
if claim_count <= 5: # Limit reported findings
findings.append({
"severity": "MODERATE",
"category": "source_verification",
"detail": f"Legal claim without citation: {sent[:100]}",
"action": "Add specific statutory reference",
})
if claim_count > 5:
findings.append({
"severity": "HIGH",
"category": "source_verification",
"detail": f"{claim_count} legal claims without specific citations",
"action": "Systematically add citations for all legal claims",
})
deductions = claim_count * 0.3
return max(1, min(5, round(5 - deductions))), findings
def score_speculation(text: str, sentences: List[str]) -> Tuple[int, List[Dict]]:
"""Score speculation control -- proper hedging vs overcertainty."""
findings = []
speculation_count = 0
hedging_count = 0
for sent in sentences:
for p in SPECULATION_PATTERNS:
if p.search(sent):
speculation_count += 1
break
for p in HEDGING_PATTERNS:
if p.search(sent):
hedging_count += 1
break
total_sentences = len(sentences)
speculation_ratio = speculation_count / max(total_sentences, 1)
if speculation_ratio > 0.3:
findings.append({
"severity": "HIGH",
"category": "speculation_control",
"detail": f"High speculation ratio: {speculation_ratio:.0%} of sentences contain speculative language",
"action": "Replace speculation with verified facts or clearly mark as opinion",
})
elif speculation_ratio > 0.15:
findings.append({
"severity": "MODERATE",
"category": "speculation_control",
"detail": f"Moderate speculation: {speculation_count} speculative statements",
"action": "Verify speculative claims or add qualifiers",
})
# Unqualified predictions are worse than hedged ones
for sent in sentences:
for p in CERTAINTY_PATTERNS:
if p.search(sent):
for sp in SPECULATION_PATTERNS:
if sp.search(sent):
findings.append({
"severity": "HIGH",
"category": "speculation_control",
"detail": f"Prediction stated with false certainty: {sent[:100]}",
"action": "Add uncertainty qualifier or remove certainty language",
})
break
high = sum(1 for f in findings if f["severity"] in ("CRITICAL", "HIGH"))
if high >= 2:
score = 2
elif high == 1:
score = 3
elif speculation_ratio > 0.15:
score = 4
else:
score = 5
return score, findings
def score_disclaimers(text: str) -> Tuple[int, List[Dict]]:
"""Score disclaimer adequacy."""
findings = []
present = 0
missing = []
for name, pattern in DISCLAIMER_CHECKS.items():
if pattern.search(text):
present += 1
else:
missing.append(name)
for m in missing:
label = m.replace("_", " ").title()
severity = "HIGH" if m in ("not_legal_advice", "professional_consultation") else "MODERATE"
findings.append({
"severity": severity,
"category": "disclaimer_adequacy",
"detail": f"Missing disclaimer: {label}",
"action": f"Add {label} disclaimer",
})
if present >= 5:
score = 5
elif present >= 4:
score = 4
elif present >= 3:
score = 3
elif present >= 2:
score = 2
else:
score = 1
return score, findings
def calculate_composite_score(category_scores: Dict[str, int]) -> Tuple[int, float]:
"""Calculate weighted composite score."""
weighted = sum(
category_scores[cat] * weight
for cat, weight in CATEGORY_WEIGHTS.items()
)
rounded = max(1, min(5, round(weighted)))
return rounded, round(weighted, 2)
def format_human_report(result: Dict[str, Any], verbose: bool = False) -> str:
"""Format as human-readable report."""
lines = []
lines.append("=" * 72)
lines.append("LEGAL QUALITY ASSESSMENT")
lines.append("=" * 72)
cs = result["composite_score"]
rating = QUALITY_RATINGS[cs]
lines.append(f"\nQuality Score: {cs}/5 -- {rating['label']}")
lines.append(f"Weighted Score: {result['weighted_score']}/5.00")
lines.append(f"Assessment: {rating['description']}")
lines.append(f"Distribution: {'APPROVED' if cs >= 4 else 'NOT APPROVED'}")
lines.append("\n--- CATEGORY SCORES ---")
for cat, score in result["category_scores"].items():
label = cat.replace("_", " ").title()
weight = CATEGORY_WEIGHTS[cat]
lines.append(f" {label:30s} {score}/5 (weight: {weight:.0%})")
sev = result["severity_summary"]
lines.append(f"\n--- FINDINGS ---")
lines.append(f" CRITICAL: {sev['CRITICAL']} HIGH: {sev['HIGH']} "
f"MODERATE: {sev['MODERATE']} LOW: {sev['LOW']}")
if verbose or cs < 4:
# Show all findings for low scores, or when verbose
lines.append("\n--- DETAILED FINDINGS ---")
for f in result["findings"]:
lines.append(f"\n [{f['severity']}] ({f['category']})")
lines.append(f" {f['detail']}")
lines.append(f" Action: {f['action']}")
lines.append("\n--- RECOMMENDATIONS ---")
for rec in result["recommendations"]:
lines.append(f" - {rec}")
lines.append("\n" + "=" * 72)
return "\n".join(lines)
def main() -> None:
parser = argparse.ArgumentParser(
description="Score legal document quality across 6 verification categories."
)
parser.add_argument("--input", "-i", type=str, help="Path to legal document")
parser.add_argument("--text", "-t", type=str, help="Inline legal text")
parser.add_argument("--output", "-o", type=str, help="Path to save output (JSON)")
parser.add_argument("--json", action="store_true", help="Output in JSON format")
parser.add_argument("--verbose", "-v", action="store_true", help="Show detailed findings")
args = parser.parse_args()
if not args.input and not args.text:
parser.print_help()
sys.exit(1)
try:
if args.input:
with open(args.input, "r", encoding="utf-8") as f:
text = f.read()
else:
text = args.text
if not text or not text.strip():
print("Error: Empty input text.", file=sys.stderr)
sys.exit(1)
sentences = extract_sentences(text)
all_findings: List[Dict] = []
fa_score, fa_findings = score_factual_accuracy(text, sentences)
la_score, la_findings = score_legal_authority(text, sentences)
ar_score, ar_findings = score_arithmetic(text, sentences)
sv_score, sv_findings = score_source_verification(text, sentences)
sp_score, sp_findings = score_speculation(text, sentences)
di_score, di_findings = score_disclaimers(text)
all_findings.extend(fa_findings)
all_findings.extend(la_findings)
all_findings.extend(ar_findings)
all_findings.extend(sv_findings)
all_findings.extend(sp_findings)
all_findings.extend(di_findings)
# Sort by severity
severity_order = {"CRITICAL": 0, "HIGH": 1, "MODERATE": 2, "LOW": 3}
all_findings.sort(key=lambda x: severity_order.get(x["severity"], 4))
category_scores = {
"factual_accuracy": fa_score,
"legal_authority": la_score,
"arithmetic": ar_score,
"source_verification": sv_score,
"speculation_control": sp_score,
"disclaimer_adequacy": di_score,
}
composite, weighted = calculate_composite_score(category_scores)
# Override: CRITICAL findings force score down
critical_count = sum(1 for f in all_findings if f["severity"] == "CRITICAL")
if critical_count > 0 and composite > 2:
composite = 2
severity_summary = {"CRITICAL": 0, "HIGH": 0, "MODERATE": 0, "LOW": 0}
for f in all_findings:
severity_summary[f["severity"]] = severity_summary.get(f["severity"], 0) + 1
# Generate recommendations
recommendations = []
if critical_count > 0:
recommendations.append("Fix all CRITICAL issues before any distribution")
if severity_summary["HIGH"] > 0:
recommendations.append(f"Address {severity_summary['HIGH']} HIGH-severity findings")
if la_score < 4:
recommendations.append("Improve citation coverage and specificity")
if di_score < 4:
recommendations.append("Add missing disclaimers")
if sp_score < 4:
recommendations.append("Reduce speculation or add proper qualifiers")
if composite >= 4:
recommendations.append("Document is suitable for distribution after final review")
result = {
"composite_score": composite,
"weighted_score": weighted,
"rating": QUALITY_RATINGS[composite]["label"],
"distribution_ready": composite >= 4 and critical_count == 0,
"category_scores": category_scores,
"severity_summary": severity_summary,
"findings": all_findings,
"recommendations": recommendations,
"statistics": {
"word_count": len(text.split()),
"sentence_count": len(sentences),
"total_findings": len(all_findings),
},
}
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
json.dump(result, f, indent=2, ensure_ascii=False)
print(f"Assessment saved to {args.output}")
elif args.json:
print(json.dumps(result, indent=2, ensure_ascii=False))
else:
print(format_human_report(result, verbose=args.verbose))
except FileNotFoundError:
print(f"Error: File not found: {args.input}", file=sys.stderr)
sys.exit(1)
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
Related skills
FAQ
What does it verify?
Six categories: factual accuracy, legal authority citations, arithmetic, source verification, speculation detection, and disclaimer adequacy.
Is this legal advice?
No. The skill is explicitly experimental, for educational purposes only, and not legal advice.