Arize Compliance Audit

Arize Compliance Audit Skill

Use this skill when the user wants to audit their AI agent or LLM application for regulatory compliance. The skill scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces a tailored checklist with optional remediation.

Triggers: "audit my app for compliance", "EU AI Act requirements", "NIST AI RMF checklist", "GDPR for AI", "is my AI app compliant", "compliance checklist", "regulatory audit", "ISO 42001", "AI management system", "AIMS certification".

Disclaimer

Before doing anything else, present this disclaimer verbatim to the user:

---

⚠️ Legal disclaimer

>

This audit is for guidance only and does not constitute legal advice or a complete compliance assessment. It identifies common technical patterns and gaps based on publicly available regulatory frameworks, but cannot assess your organisation's specific legal obligations, contractual commitments, data processing agreements, or operational processes.

>

Do not rely on this output as a substitute for qualified legal counsel. Regulatory compliance is a complex, jurisdiction-specific, and fact-dependent determination. Always engage a qualified attorney or compliance specialist for binding assessments.

---

Core principles

• Prefer inspection over mutation — understand the codebase before suggesting changes.
• Be practical, not legal — produce developer-actionable items, not legal opinions.
• Tailor to jurisdiction and use case — a chatbot has different obligations than a hiring tool. Do not dump the entire regulatory framework.
• Cross-reference instrumentation — compliance requires audit trails; check whether Arize tracing captures what regulators expect.
• Offer remediation, always confirm — after presenting the checklist, offer to implement specific fixes, but never modify code without explicit user confirmation.
• Keep output concise and production-focused — do not generate extra documentation or summary files unless requested.
• Never embed literal credential values — always reference environment variables.

Phase 0: Framework selection and use case

Before scanning code, determine which compliance frameworks apply.

Step 1 — Framework selection

Use the AskUserQuestion tool to ask the user which frameworks apply. Do not infer or auto-select — always ask explicitly.

Ask:

Which compliance frameworks should this audit cover?
Select all that apply (reply with numbers, e.g. "1, 3"):

1. EU frameworks — EU AI Act, GPAI Code of Practice, GDPR
   (choose if end-users or data subjects are located in the EU)

2. US frameworks — NIST AI RMF, state laws (Colorado AI Act, NYC LL144),
   HIPAA (if processing health data)
   (choose if operating in the United States)

3. ISO 42001 — International AI Management System standard
   (choose if pursuing ISO 42001 certification, operating globally,
   or wanting an internationally recognised baseline)

You can select any combination. If unsure, select all that seem relevant
and we can narrow down during the audit.

Based on the selection:

• 1 selected — EU AI Act, GPAI Code of Practice, GDPR apply. See references/eu-ai-act-gpai.md.
• 2 selected — NIST AI RMF, Colorado AI Act, NYC LL144, HIPAA may apply. See references/us-ai-compliance.md.
• 3 selected — ISO 42001 AIMS controls apply. See references/iso-42001.md. Note: ISO 42001 is an organisational management system — the audit will cover technically-auditable controls only; purely organisational clauses (leadership review, internal audits) are flagged separately.
• Multiple selected — all selected frameworks apply; the audit covers the union of requirements, with cross-references where frameworks overlap.

Step 2 — Determine use case category

Use the AskUserQuestion tool to ask: What does your AI application do?

• General chatbot / assistant — Limited risk (EU), general obligations (US)
• Hiring / HR — High risk (EU Art. 6, Annex III); Colorado AI Act applies; NYC LL144 applies if NYC
• Healthcare — High risk (EU); HIPAA applies if processing PHI
• Credit / financial — High risk (EU); Colorado AI Act applies
• Education — High risk (EU)
• Content generation — Limited risk (EU Art. 50 transparency); general obligations (US)
• GPAI model provider — GPAI Code of Practice applies (EU)

Step 3 — Determine risk tier

Based on the use case and selected frameworks:

• EU selected: Classify as Unacceptable / High / Limited / Minimal per references/eu-ai-act-gpai.md
• US selected: Classify as High-risk (consequential decisions per Colorado AI Act) or General
• ISO 42001 selected: Risk tier is not a formal classification in ISO 42001, but note whether the system is high-stakes (which elevates the priority of impact assessment and bias controls)

Phase 0 output

Present a brief summary:

Frameworks selected: {EU / US / ISO 42001 / combination}
Use case:            {category}
Risk tier:           {EU tier if applicable} / {US tier if applicable}
Applicable:          {list of specific regulations and standards}
ISO 42001 note:      {if selected} Audit covers technically-auditable controls only;
                     organisational clauses will be flagged but not code-audited.

Then proceed directly to Phase 1.

Phase 1: Codebase audit (read-only)

Do not write any code or create any files during this phase.

Systematically scan the codebase for evidence of compliance and gaps across seven domains. For each domain, run the listed searches and record findings.

A. Transparency and disclosure

What to look for:

• User-facing strings disclosing AI involvement: search for terms like AI, artificial intelligence, automated, bot, machine learning, generated by, powered by in UI templates, API responses, and user-facing code
• Content labelling: markers on AI-generated output (text, images, audio)
• Terms of service, privacy policy references in the codebase

Signals of concern: Absence of any AI disclosure in user-facing code, especially if the application generates content or makes recommendations.

B. Data protection and privacy

What to look for:

• PII field names in code: email, phone, ssn, social_security, date_of_birth, address, name in prompts, context, or retrieved documents
• PII in trace span attributes: check if input.value or output.value could contain personal data sent to Arize without redaction
• Consent mechanisms: consent, opt-in, opt-out, gdpr, ccpa references
• DPIA or privacy assessment references
• Data retention and deletion handlers
• Data subject rights: right_to_access, right_to_erasure, data_subject_request, data_protection_officer

C. Security

What to look for:

• Prompt injection defences: input validation, guardrail libraries (guardrails-ai, nemo-guardrails, rebuff, lakera), content filtering, system prompt protection
• Data loss prevention: output scanning before returning to users, sensitive data detection
• Tool/function calling controls: permission boundaries, allowlists, sandboxing for tool execution
• Rate limiting and authentication on AI endpoints
• Hardcoded secrets: api_key, secret, password, token literals in source files (not env var references)

D. Testing and evaluation

What to look for:

• Bias and fairness testing: references to demographic parity, impact ratios, fairness metrics
• Red teaming or adversarial test suites: prompt injection tests, jailbreak tests
• Evaluation frameworks: Arize evaluators, custom eval scripts, pytest-based evals, experiment infrastructure
• A/B testing or model comparison infrastructure

E. Documentation

What to look for:

• Model cards: MODEL_CARD.md, model_card.json, model_card.yaml, or similar
• System architecture documentation
• Change logs or version tracking for prompts and model updates
• Incident response documentation

F. Monitoring and observability

What to look for:

• Arize tracing setup: arize-otel, register(), TracerProvider, opentelemetry, openinference imports
• If tracing exists, check coverage:
• All LLM calls traced (not just some)
• Session IDs for conversation continuity
• User IDs for data subject request support
• Error tracking and exception spans
• Alerting and drift detection configuration
• Trace retention configuration

G. Vendor management

What to look for:

• Third-party AI API usage: OpenAI, Anthropic, Google, Azure, Bedrock, Cohere imports or client instantiation
• Model versioning: are specific model versions pinned (e.g., gpt-4-0613) or using latest / unversioned identifiers
• Fallback and failover logic between providers

Phase 1 output

Present a two-part report:

Part 1 — Summary table

Part 2 — Gap detail (required for every Non-compliant or Partial rating)

For each domain rated Non-compliant or Partial, write a dedicated subsection that includes:

1. The exact code path — file path(s), line number(s), and the relevant code snippet showing where the gap exists. Do not paraphrase; quote the actual code. 2. Why it matters in this specific app — explain the concrete risk in the context of this codebase (e.g. which tools could be abused, which data flows are exposed, what an attacker or regulator would find). 3. What is missing — a precise description of the control or code that should exist but does not (e.g. "a span attribute processor that hashes user_email before the OTLP exporter fires", not just "add PII redaction").

Minimum one subsection per Non-compliant/Partial domain. Do not omit this section — it is the primary value of the audit for engineering teams.

Then proceed directly to Phase 2.

Phase 2: Compliance checklist

Using the Phase 1 findings and the template in references/compliance-checklist-template.md, generate a tailored compliance checklist.

Rules for checklist generation

1. Only include relevant sections. If the user is US-only, skip GDPR-specific items. If not healthcare, skip HIPAA. If not hiring in NYC, skip LL144. 2. Mark items from Phase 1. Items where evidence was found: mark as Compliant. Items with gaps: mark as Non-compliant with a concrete remediation suggestion. 3. Prioritise correctly. Critical = enforcement risk or system prohibition. High = required by regulation. Medium = recommended by framework. Low = best practice. 4. Be specific in remediation. Instead of "implement input validation", say "add a guardrail library like guardrails-ai to validate LLM inputs and outputs against your content policy". 5. Include the instrumentation cross-reference table from the template. If Arize tracing is not set up, flag this as a Critical gap — audit trails are required by EU Art. 12 and NIST MAN-2.1.

Final report

Present a single consolidated report with four sections:

Section 1 — Audit scope (Phase 0 summary)

• Frameworks selected, use case, risk tier, applicable regulations

Section 2 — Codebase findings (Phase 1 summary table)

• The domain table (A–G) with evidence, gaps, and ratings

Section 3 — Gap detail (Phase 1 expanded)

• One subsection per Non-compliant or Partial domain, each containing: exact file paths and line numbers, quoted code snippets, app-specific risk explanation, and a precise description of what is missing. This section is mandatory — never omit it.

Section 4 — Compliance checklist (Phase 2)

• The tailored checklist with status and remediation suggestions, instrumentation cross-reference table, priority summary, and recommended next steps

When the user asks for a report file, write a single markdown file to /tmp/<app-name>-compliance-audit-<YYYY-MM-DD>.md containing all four sections.

After presenting the report, offer Phase 3 remediation.

Phase 3: Remediation (optional)

After presenting the checklist, offer to implement specific fixes. Always use the `AskUserQuestion` tool to confirm before making any changes.

Remediation categories

Add dependencies — offer to install:

• Guardrail libraries for input/output validation (e.g., guardrails-ai, nemo-guardrails)
• PII detection/redaction packages (e.g., presidio-analyzer, scrubadub)
• Content safety classifiers

Insert code — offer to add:

• AI disclosure strings in user-facing output (templates, API responses)
• PII redaction filters on span attributes before export to Arize
• Input validation/sanitisation on AI endpoints
• User ID attributes on trace spans for data subject request support

Create documentation templates — offer to scaffold:

• Model card template (markdown file with standard sections)
• Incident response plan template
• Data processing record template

Configure monitoring — offer to set up via related skills:

• Arize evaluators for bias detection and content safety (via arize-evaluator skill)
• Tracing for audit trail coverage (via arize-instrumentation skill)

Remediation rules

• Present each remediation as a discrete, confirmable action. Never batch-apply changes.
• Show exactly what will change (file, code diff concept) then use the AskUserQuestion tool to get confirmation before applying.
• Follow existing code style and project conventions.
• Never embed credentials — always use environment variables.
• Test that the application still builds after changes.

Skill orchestration

When gaps identified in Phase 1 or 2 require capabilities from other Arize skills, offer to invoke them. Always use the `AskUserQuestion` tool to ask before invoking another skill and explain why it is relevant to the compliance gap.

Instrumentation cross-reference

If Arize tracing is already set up, verify it meets compliance requirements:

If Arize tracing is not set up, this is a significant compliance gap. Offer: "Shall I run the arize-instrumentation skill to set up audit-trail tracing? Regulatory frameworks (EU AI Act Art. 12, NIST AI RMF MAN-2.1) require event logging for AI systems."

Reference links

Reference files

• references/eu-ai-act-gpai.md — EU AI Act and GPAI Code of Practice developer guide
• references/us-ai-compliance.md — US compliance landscape (NIST AI RMF, Colorado, NYC LL144, HIPAA)
• references/iso-42001.md — ISO/IEC 42001:2023 AI Management Systems developer guide (technically-auditable controls only)
• references/compliance-checklist-template.md — Reusable checklist template for Phase 2 output

Compliance Checklist Template

Use this template to generate the Phase 2 compliance checklist. Fill in the status and notes columns based on Phase 1 audit findings. Only include sections relevant to the user's jurisdiction and use case — do not dump the entire template.

Header

Priority legend

---

1. Governance and documentation

2. Data protection and privacy

3. Security

4. Testing and evaluation

5. Transparency and user communication

6. Monitoring and continuous compliance

7. Vendor management

---

Instrumentation cross-reference

Next steps

1. Address all Critical items immediately 2. Remediate High items before production deployment 3. Schedule Medium items for the next development cycle 4. Track Low items in backlog 5. Schedule next compliance audit for: {date + 3 months or per regulatory requirement} 6. Consult qualified legal counsel for binding compliance assessment

EU AI Act and GPAI Code of Practice — Developer Reference

This reference covers the EU AI Act (Regulation 2024/1689) and the Code of Practice for General-Purpose AI (GPAI). It is written for developers building AI agents and LLM applications, not for legal professionals. Always consult qualified legal counsel for binding compliance assessments.

Official sources:

• EU AI Act full text: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• GPAI Code of Practice: https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
• Code of Practice portal: https://code-of-practice.ai/

Risk classification

The EU AI Act classifies AI systems into four risk tiers. Your obligations depend on which tier your application falls into.

Unacceptable risk (prohibited — Art. 5)

These AI uses are banned outright:

• Social scoring by public authorities
• Real-time remote biometric identification in public spaces (with narrow exceptions)
• Exploitation of vulnerabilities of specific groups (age, disability)
• Subliminal manipulation causing harm
• Emotion recognition in workplaces and educational institutions (with exceptions)
• Untargeted scraping of facial images for facial recognition databases

High risk (Art. 6–7, Annex III)

Subject to the strictest requirements. Includes AI used for:

• Employment: recruitment, CV screening, interview evaluation, promotion decisions, task allocation
• Credit scoring and loan decisions
• Education: student assessment, admissions, exam proctoring
• Healthcare: diagnostic assistance, treatment recommendations, triage
• Law enforcement: risk assessment, evidence evaluation
• Migration and border control: document authentication, risk assessment
• Critical infrastructure: energy, water, transport management

Limited risk (Art. 50 — transparency obligations)

AI systems that interact with people or generate content must:

• Disclose to users that they are interacting with AI
• Label AI-generated content (text, audio, image, video) as such
• Label deepfakes clearly

This tier covers most chatbots, virtual assistants, and content-generation tools.

Minimal risk

No specific obligations. Includes spam filters, AI-enhanced video games, inventory management systems.

High-risk system requirements (Art. 8–15)

If your application is classified as high-risk, you must implement:

Risk management system (Art. 9)

• Continuous, iterative risk identification and mitigation throughout the AI system lifecycle
• Testing with metrics appropriate to the intended purpose
• Evaluation of risks from foreseeable misuse
• Risk mitigation measures that cannot be fully eliminated must be communicated to deployers

Data governance (Art. 10)

• Training, validation, and testing datasets must be relevant, sufficiently representative, and as error-free as possible
• Statistical properties including biases must be examined
• Data must be appropriate for the geographic, contextual, and functional setting

Technical documentation (Art. 11)

• General description of the AI system and its intended purpose
• Design specifications, development methodology, and development choices
• Monitoring, functioning, and control of the system
• Risk management documentation
• Description of changes made throughout the lifecycle
• Must be drawn up before the system is placed on the market and kept up to date

Record-keeping (Art. 12)

• Automatic logging of events ("logs") throughout the system's lifecycle
• Logs must enable tracing of the system's operation to identify risks
• Logging must be appropriate to the intended purpose
• Retention: Logs must be kept for a period appropriate to the intended purpose, at least 6 months unless provided otherwise by law

Transparency and information to deployers (Art. 13)

• Clear, adequate information to deployers about the system's capabilities and limitations
• Intended purpose, level of accuracy, robustness, and cybersecurity
• Known or foreseeable circumstances that may lead to risks
• Performance metrics including for specific persons or groups
• Technical capabilities and limitations (input data specifications, output interpretation guidance)

Human oversight (Art. 14)

• Designed to allow effective oversight by natural persons during use
• Deployers must assign human oversight to competent individuals
• Individuals must be able to: understand the system, detect anomalies, decide not to use the system, intervene or stop the system

Accuracy, robustness, cybersecurity (Art. 15)

• Appropriate level of accuracy declared in documentation
• Resilience to errors, faults, and inconsistencies
• Resilience against manipulation by unauthorised third parties
• Technical redundancy solutions where appropriate

GPAI Code of Practice obligations

The GPAI Code of Practice applies to providers of general-purpose AI models. If you are building on top of a GPAI model (e.g., using OpenAI, Anthropic, or open-source LLMs), these obligations primarily fall on the model provider, but you should understand them for your own documentation and risk management.

Transparency (Commitments 1–7)

• Prepare and maintain a Model Documentation Form covering model capabilities, limitations, training data summaries, and contact information
• Publish a sufficiently detailed summary of training data content
• Provide documentation to downstream providers and deployers
• Retention: Documentation must be maintained for a minimum of 10 years after the model is placed on the market

Copyright (Commitments 8–11)

• Establish a copyright compliance policy aligned with EU copyright law (Directive 2019/790)
• Implement measures to mitigate risk of copyright-infringing outputs
• Implement a complaints-handling process for rights holders
• Document how training data was sourced and validated for copyright compliance

Safety and security (Commitments 12–22, for systemic risk models)

• Implement a Safety and Security Framework (SSF) covering governance, risk assessment, and technical mitigations
• Identify and evaluate systemic risks before major deployment decisions
• Conduct model evaluations (capability, safety, societal risk)
• Implement safeguards for identified risks
• Conduct adversarial testing (red teaming)
• Report serious incidents to the AI Office within prescribed timelines
• Track and document model capabilities over time

GDPR intersection

If your AI application processes personal data of EU residents, GDPR applies regardless of where you are based.

When AI processing triggers GDPR

• Prompts or context containing personal data (names, emails, health info)
• User conversation logs stored for fine-tuning or evaluation
• Retrieval-augmented generation (RAG) over documents containing personal data
• Trace data sent to observability platforms containing PII in span attributes

Key GDPR requirements for AI developers

• Lawful basis (Art. 6): Consent, contract, legitimate interest, or other lawful basis for processing
• Data Protection Impact Assessment (Art. 35): Required for high-risk processing, including systematic profiling and large-scale processing of sensitive data
• Data Processing Agreements (Art. 28): Required with every processor (LLM providers, cloud infrastructure, observability platforms)
• Data subject rights (Art. 15–22): Right to access, rectification, erasure, restriction, portability, and to object. For automated decision-making: right to human intervention and explanation
• Data minimisation (Art. 5(1)(c)): Only process data that is adequate, relevant, and limited to what is necessary
• Breach notification (Art. 33–34): Notify supervisory authority within 72 hours; notify affected individuals without undue delay if high risk
• International transfers (Art. 44–49): Standard Contractual Clauses or Data Privacy Framework for transfers outside EEA

Developer action mapping

Penalties

Timeline

ISO/IEC 42001:2023 — AI Management Systems Developer Reference

This reference covers ISO/IEC 42001:2023, the international standard for Artificial Intelligence Management Systems (AIMS). It is written for developers building AI agents and LLM applications, not for legal or audit professionals.

Official source: https://www.iso.org/standard/42001.html

Scope note: ISO 42001 is primarily an organisational management system standard. Many of its requirements (leadership commitment, internal audits, management reviews) cannot be verified from code alone. This reference focuses on clauses and annex controls that produce code or documentation artifacts — things a codebase audit can actually find evidence of or flag as missing.

What ISO 42001 is

ISO 42001 is to AI what ISO 27001 is to information security: a framework for establishing, implementing, maintaining, and continually improving an AI management system. It applies to any organisation that develops, provides, or uses AI systems.

Unlike the EU AI Act or NIST AI RMF, ISO 42001 is a certifiable standard — organisations can be independently audited and certified. It is internationally applicable (not jurisdiction-specific) and complements other frameworks rather than replacing them.

Relationship to other frameworks

ISO 42001 Annex C maps the standard's controls to the EU AI Act and NIST AI RMF. If you comply with EU AI Act Art. 8–15, you satisfy most ISO 42001 technical requirements.

Clause structure overview

Technically auditable clauses and controls

Clause 5.2 — AI policy

Requirement: The organisation shall establish an AI policy that includes commitments to satisfy applicable requirements and to continual improvement.

Code artifact to look for: A policy document referenced in the codebase (AI_POLICY.md, docs/ai-policy.md, GOVERNANCE.md) or a link to a policy in README.md or CONTRIBUTING.md.

Gap signal: No policy document or reference found anywhere in the repository.

Clause 6.1.2 — AI risk assessment

Requirement: Documented process for identifying and assessing AI-specific risks (bias, safety, privacy, security).

Code artifact to look for: Risk assessment or DPIA documents, risk register files, or references to a risk management process.

Gap signal: No risk or impact assessment documentation.

Clause 8.3 — AI impact assessment

Requirement: Conduct and document an impact assessment for AI systems, covering intended use, potential harms, and affected stakeholders.

Code artifact to look for: AI_IMPACT_ASSESSMENT.md, docs/impact-assessment.*, or similar. May be combined with a DPIA for GDPR overlap.

Gap signal: No impact assessment document found.

Clause 8.4 — AI system lifecycle

Requirement: Manage the AI system through a documented lifecycle: design, development, testing, deployment, monitoring, decommission. Changes are documented and controlled.

Code artifact to look for:

• Changelog or version history for models and prompts
• Model card (MODEL_CARD.md, model_card.json)
• Pinned model versions (not latest) in code
• Deployment notes or release documentation

Gap signal: No model card, no changelog, or model versions unpinned.

Clause 8.6 — AI system operation and monitoring

Requirement: Monitor AI system performance, detect degradation, and act on findings.

Code artifact to look for:

• Arize tracing and evaluation setup
• Drift detection configuration
• Alerting rules
• Periodic re-evaluation schedule

Gap signal: No monitoring or evaluation infrastructure.

Clause 9.1 — Monitoring, measurement, analysis, evaluation

Requirement: Determine what needs to be monitored, methods, when to analyse, when to report results.

Code artifact to look for: Evaluation pipelines, Arize evaluators, scheduled jobs for performance review, dashboards.

Gap signal: No evaluation or measurement infrastructure beyond basic logging.

Annex A controls (technically auditable subset)

A.2.2 — Policies for AI use (organisational AI use policy)

Policies governing acceptable AI use, including approved use cases, prohibited uses, and responsible use guidelines.

Artifact: docs/ai-use-policy.md, ACCEPTABLE_USE.md, or policy links in README.

A.3.2 — Internal audit of the AIMS

Documented audit programme and results.

Artifact: Audit log files, scheduled audit tooling config. Usually organisational — note as not code-auditable.

A.4.2 — Addressing bias

Documented processes for identifying, assessing, and mitigating bias in training data and model outputs.

Artifact: Bias testing scripts, fairness metric definitions, Arize evaluators for fairness, demographic parity tests.

A.5.2 — AI system impact assessment

See clause 8.3 above.

A.6.1 — Allocation of AI roles and responsibilities

Accountable owners for AI systems identified and documented.

Artifact: CODEOWNERS, README.md ownership sections, or GOVERNANCE.md with role assignments.

A.6.2 — AI system lifecycle management

Processes for managing the full lifecycle. See clause 8.4.

A.7 — Data for AI systems

A.7.1 — Data governance: Processes ensuring data quality, provenance, and fitness for purpose.

A.7.2 — Data acquisition and collection: Documented data sources, collection methods, consent basis.

A.7.3 — Data preparation: Data cleaning, preprocessing, and validation documented.

A.7.4 — Data quality: Data quality checks implemented in pipeline.

Code artifact to look for: Data validation scripts, schema definitions, data provenance documentation, data quality checks.

A.8 — Information for interested parties

Requirements for transparency to users and stakeholders — overlaps with EU AI Act Art. 13 and 50.

Artifact: User-facing disclosures, model documentation, capability and limitation statements.

A.9 — Use of AI systems

Guidance for users of AI systems, including known limitations and appropriate use. Often in user documentation.

Artifact: User guide, USAGE.md, or in-app help text describing limitations.

A.10 — Third-party and supplier relationships

Requirement: Supplier AI policy covering third-party AI providers and data processors.

Code artifact to look for: Documented AI vendor list, vendor risk assessments, DPA references, supplier questionnaires.

Gap signal: Third-party AI APIs used (OpenAI, Anthropic, etc.) with no vendor documentation.

Organisational-only requirements (not code-auditable)

These clauses require organisational evidence that cannot be found in a codebase. Flag them for the user to address outside the technical audit:

Checklist priority guidance

US AI Compliance Landscape — Developer Reference

This reference covers US regulatory frameworks applicable to AI agent and LLM application development. The US does not have a single comprehensive federal AI law; compliance is a patchwork of federal frameworks, state laws, and sector-specific regulations. Always consult qualified legal counsel for binding compliance assessments.

Official sources:

• NIST AI RMF: https://www.nist.gov/artificial-intelligence/ai-risk-management-framework
• Colorado AI Act (SB24-205): https://leg.colorado.gov/bills/sb24-205
• NYC Local Law 144: https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page
• HIPAA: https://www.hhs.gov/hipaa/index.html

NIST AI Risk Management Framework (AI RMF 1.0)

The NIST AI RMF is voluntary but increasingly the de facto standard for AI governance. Federal agencies require alignment for AI procurement (OMB M-24-10), and state laws like the Colorado AI Act reference NIST-aligned frameworks. Aligning with NIST AI RMF is the strongest foundation for US compliance.

Govern function

Establish organisational AI governance. Developer actions:

Map function

Identify and contextualise AI risks. Developer actions:

Measure function

Quantify and test AI risks. Developer actions:

Manage function

Implement controls and mitigation. Developer actions:

Colorado AI Act (SB24-205)

Scope

Applies to developers and deployers of "high-risk AI systems" — those making or substantially assisting consequential decisions in:

• Employment, promotion, termination
• Education admissions and opportunities
• Financial services (lending, credit, insurance)
• Healthcare services
• Housing
• Legal services
• Government services

Developer obligations (effective 1 February 2026)

Safe harbour

Compliance with a recognised risk management framework (NIST AI RMF or ISO 42001) establishes a rebuttable presumption of reasonable care.

NYC Local Law 144 — Automated Employment Decision Tools (AEDT)

Scope

Applies to employers and employment agencies in New York City using AI for hiring, promotion, or other employment decisions.

Requirements

Audit methodology

• Calculate selection/scoring rates for each demographic category
• Calculate impact ratios (selection rate of each group vs the most-selected group)
• Flag categories with impact ratios below 0.8 (four-fifths rule) for further analysis

HIPAA for AI applications

Applies if your AI application processes Protected Health Information (PHI). This includes clinical data, diagnostic information, treatment records, or any data linkable to a specific patient.

When HIPAA applies to AI

• AI system processes, stores, or transmits PHI
• AI system is used by or on behalf of a covered entity (healthcare provider, health plan, healthcare clearinghouse)
• AI system is provided by a business associate of a covered entity

Requirements

PHI in traces and spans

If using Arize tracing in a healthcare context:

• Ensure PHI is redacted from input.value and output.value span attributes before export
• Verify Arize's BAA covers trace data storage
• Implement PII/PHI detection filters on span data before it leaves the application
• Configure trace retention to meet HIPAA's 6-year retention requirement for audit logs

State privacy laws intersection

Several state privacy laws impose additional obligations when AI processes personal data:

Common obligations across state privacy laws:

• Data protection assessments for AI-driven profiling
• Opt-out mechanisms for automated decision-making
• Transparency about automated processing
• Data minimisation in AI contexts

Developer action summary