Auditing Experiments Flags

Auditing experiments and feature flags

This skill teaches you how to run configuration audits on experiments and feature flags. All checks use the experiment and feature flag read tools (experiment-get, experiment-list, feature-flag-get-definition, feature-flag-get-all) — no SQL queries are needed for Phase 1 checks.

Usage modes

Quick check (single entity)

When the user asks about a specific experiment or flag:

1. Fetch the entity via experiment-get (experiment ID) or feature-flag-get-definition (numeric flag ID). 2. Apply the relevant checks from experiment checks or flag checks. 3. Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO). 4. Include entity links as [Experiment: name](/experiments/id) or [Flag: key](/feature_flags/id).

Scoped audit (one domain)

When the user asks to audit all experiments or all flags:

1. Bulk-fetch via experiment-list or feature-flag-get-all. 2. Run all checks for that domain against each entity. 3. Group findings by severity, then by entity. 4. Report as inline markdown.

Full audit (comprehensive)

When the user asks for a comprehensive audit of both experiments and flags:

1. Fetch all experiments via experiment-list and all flags via feature-flag-get-all. 2. Run all experiment checks and all flag checks. 3. Apply recurring patterns to identify patterns across multiple findings. 4. If there are more than 5 entities with findings, output as a notebook artifact via notebooks-create for easier navigation. Otherwise report inline.

Output format

For each finding, include:

• Severity badge: 🔴 CRITICAL, 🟡 WARNING, or 🔵 INFO
• Check name: Which check produced this finding
• Entity link: Markdown link to the entity
• What's wrong: One-sentence description
• Action: What to do about it (see remediation actions)

Example:

🟡 WARNING — Flag integration · Experiment: checkout-redesign

The linked feature flag is inactive (paused). Traffic is not being split.

Action: Re-enable the flag or end the experiment.

Handling unavailable data

Some checks require activity logs (feature-flags-activity-retrieve for flags), which may not be available in every session. If activity log data is unavailable:

• Skip checkActivityHistory (experiment check) entirely.
• Skip the "toggle instability" and "never activated" sub-checks in flag lifecycle checks.
• In your report, note which checks were skipped and why:

_Skipped: Activity history checks (activity logs not available via current tools)_

Partial failures

If a fetch call fails for some entities:

• Continue with the entities you could fetch.
• Report which entities could not be assessed and why.
• Do not silently omit entities from the audit.

Reference files

• Experiment checks — experiment configuration checks
• Flag checks — feature flag checks
• Finding types — severity and category definitions
• Recurring patterns — patterns across multiple findings
• Remediation actions — what to do about each finding

Experiment checks

Run these checks against each experiment fetched via experiment-get or experiment-list.

For each check, the "Look at" section tells you which fields to inspect on the experiment object. The "Findings" section lists what to report and at what severity.

---

1. Metric setup

Verifies the experiment has a valid primary metric configuration.

Look at: metrics, metrics_secondary

Findings:

• No metrics at all: Both metrics and metrics_secondary are empty or missing.
• Severity: CRITICAL · Category: Correctness
• Report: "This experiment has no metrics configured. Results cannot be measured."
• Action: Add at least one primary metric before launching.

• Secondary metrics only: metrics is empty but metrics_secondary has entries.
• Severity: WARNING · Category: Process
• Report: "This experiment has secondary metrics but no primary metric. There is no primary success criterion."
• Action: Promote one secondary metric to primary or add a new primary metric.

---

2. Flag integration

Verifies the experiment's linked feature flag is valid and correctly configured.

Look at: feature_flag (the linked flag object or ID), and fetch the flag via feature-flag-get-definition if only an ID is available.

Findings:

• Missing flag: feature_flag is null or missing.
• Severity: CRITICAL · Category: Correctness
• Report: "This experiment has no linked feature flag. Traffic cannot be split."
• Action: Create and link a feature flag.

• Inactive flag: The linked flag exists but active is false.
• Severity: WARNING · Category: Correctness
• Report: "The linked feature flag is inactive (paused). Traffic is not being split."
• Action: Re-enable the flag or end the experiment.

• Deleted flag: The linked flag has deleted set to true.
• Severity: CRITICAL · Category: Correctness
• Report: "The linked feature flag has been deleted."
• Action: Create a new flag and re-link it, or archive the experiment.

• Uneven variant split: The linked flag's variant rollout percentages differ from the experiment's expected split by more than 5 percentage points.

Compare the flag's filters.multivariate.variants rollout percentages to the experiment's parameters.feature_flag_variants.

• Severity: WARNING · Category: Correctness
• Report: "Variant rollout percentages on the flag don't match the experiment's expected split."
• Action: Adjust the flag's variant percentages to match the experiment configuration.

• Variant mismatch: The variant keys in the experiment's parameters.feature_flag_variants don't match the variant keys in the flag's filters.multivariate.variants.
• Severity: CRITICAL · Category: Correctness
• Report: "Variant keys differ between the experiment and its linked flag."
• Action: Align variant keys between the experiment and its flag.

---

3. State consistency

Checks for contradictions between an experiment's conclusion and its current flag state.

Look at: end_date (non-null means concluded), archived, parameters.recommended_variant, and the linked flag's active state and variant configuration.

Findings:

• Conclusion contradicts shipped variant: The experiment concluded with a recommended variant (in parameters.recommended_variant), but the flag is rolled out to a _different_ variant at 100%.
• Severity: WARNING · Category: Correctness
• Report: "The experiment concluded recommending variant 'X' but the flag is rolled out to variant 'Y'."
• Action: Review and align the flag's rollout with the experiment conclusion.

• Concluded but still splitting: The experiment has an end_date (it's concluded) but the linked flag still has multiple variants with non-zero rollout (traffic is still being split).
• Severity: WARNING · Category: Waste
• Report: "This experiment has concluded but its flag is still splitting traffic between variants."
• Action: Roll out the winning variant or disable the flag.

---

4. Lifecycle

Checks for experiments stuck in unproductive states.

Look at: created_at, start_date, end_date, description (for hypothesis)

Findings:

• Stale draft: start_date is null (never launched) and created_at is more than 7 days ago.
• Severity: INFO · Category: Cleanup
• Report: "This experiment has been in draft for N days without being launched."
• Action: Launch the experiment or delete it.

• No hypothesis: description is empty or missing, and the experiment has been launched (start_date is set).
• Severity: INFO · Category: Process
• Report: "This launched experiment has no hypothesis documented in its description."
• Action: Add a hypothesis to document what you expect to learn.

---

5. Stopped with active flag

Checks for experiments that have ended but whose flags are still active and splitting.

Look at: end_date, archived, and the linked flag's active status and variant rollout.

Findings:

• Ended but flag still active and splitting: end_date is set (experiment ended), but the linked flag is still active: true and has multiple variants with non-zero rollout percentages.
• Severity: WARNING · Category: Waste
• Report: "This experiment ended on [date] but its flag is still actively splitting traffic."
• Action: Roll out the winning variant at 100% or disable the flag.

Note: This is related to but distinct from "concluded but still splitting" in check 3. Check 3 focuses on the contradiction with the conclusion; this check focuses on the resource waste of an ended experiment still consuming flag evaluations.

---

6. Minimum duration

Checks whether a running experiment has collected enough data.

Look at: start_date, end_date

Findings:

• Very short run: start_date is set, end_date is set, and the duration is less than 7 days.
• Severity: WARNING · Category: Process
• Report: "This experiment ran for only N days. Results may not be statistically significant."
• Action: Consider whether the sample size was sufficient before drawing conclusions.

• Short run: Duration is between 7 and 14 days.
• Severity: INFO · Category: Process
• Report: "This experiment ran for N days. Consider whether the sample size is sufficient."
• Action: Review statistical significance before concluding.

---

7. Stats config

Checks for unusual statistical configuration.

Look at: start_date, end_date (or current date if still running), parameters.stats_config

Findings:

• Long-running experiment: The experiment has been running for more than 30 days (calculated from start_date to end_date or today if still running).
• Severity: INFO · Category: Process
• Report: "This experiment has been running for N days. Long-running experiments can accumulate confounding factors."
• Action: Review whether this experiment still needs to run or if a conclusion can be drawn.

---

8. Activity history

Checks for flag modifications that may have affected experiment integrity. These checks require activity logs. If activity logs are not available, skip this entire check and note it was skipped.

Look at: Activity log entries for the linked feature flag, filtered by the experiment's run period (start_date to end_date or today).

Findings:

• Pre-run flag changes: The flag was modified between experiment creation and launch.
• Severity: INFO · Category: Process
• Report: "The flag was modified N times before the experiment launched."
• Action: Informational — verify the flag was in the intended state at launch.

• Mid-run rollout changes: The flag's rollout percentages were changed while the experiment was running.
• Severity: WARNING · Category: Correctness
• Report: "The flag's rollout percentages were changed during the experiment run."
• Action: This may have affected results. Note the change date and consider its impact on the data.

• Mid-run variant changes: Variants were added or removed from the flag while the experiment was running.
• Severity: CRITICAL · Category: Correctness
• Report: "Variants were added or removed from the flag during the experiment run."
• Action: This likely invalidated the experiment. Consider restarting with a clean flag.

• Mid-run flag toggles: The flag was toggled on/off during the experiment run.
• Severity: WARNING · Category: Correctness
• Report: "The flag was toggled on/off during the experiment run, creating periods with no traffic splitting."
• Action: Review whether the interruption affected results significantly.

• Mid-run targeting changes: The flag's targeting conditions (properties, groups) were modified during the run.
• Severity: WARNING · Category: Correctness
• Report: "The flag's targeting conditions were changed mid-experiment, altering the eligible population."
• Action: Review whether the targeting change affected the experiment's statistical validity.

Finding types

Severities

Finding categories

Severity caps

Never assign a severity higher than the category's max:

• A Cleanup finding is always INFO, never WARNING or CRITICAL.
• A Correctness finding can be CRITICAL, WARNING, or INFO depending on impact.
• A Waste or Process finding caps at WARNING.

Finding format

When reporting a finding, always include:

1. Severity — one of CRITICAL, WARNING, INFO 2. Category — one of the categories above 3. Check name — which check produced this (e.g., "Metric setup", "Flag integration") 4. Entity — which experiment or flag, as a markdown link 5. Description — one sentence explaining what's wrong 6. Action — one sentence explaining what to do (reference remediation actions)

Ordering

When listing multiple findings:

1. Sort by severity: CRITICAL first, then WARNING, then INFO. 2. Within the same severity, group by entity. 3. Within the same entity, list by category: Correctness → Waste → Process → Complexity → Cleanup → Security.

Feature flag checks

Run these checks against each flag fetched via feature-flag-get-definition or feature-flag-get-all.

---

1. Staleness: fully rolled out

Detects active boolean flags that are effectively permanent and can be removed from code.

Look at: active, filters.multivariate (should be absent or null for boolean flags), filters.groups, last_called_at

Findings:

• Fully rolled out boolean flag: Flag is active: true, has no multivariate config, and at least one release condition (filters.groups entry) with rollout_percentage: 100 and no properties (empty array or missing). This flag always evaluates to true.
• Severity: INFO · Category: Cleanup
• Report: "This boolean flag is rolled out to 100% with no targeting conditions. It always evaluates to true."
• Action: Remove the flag from code and hardcode the value.

• Possibly unused: Flag has last_called_at that is more than 30 days ago, regardless of rollout configuration.
• Severity: INFO · Category: Cleanup
• Report: "This flag hasn't been evaluated in N days. It may no longer be referenced in code."
• Action: Check if the flag is still referenced in your codebase. If not, delete it.

---

2. Staleness: stale draft

Detects flags that were created but never activated.

Look at: active, created_at, activity logs (if available)

Findings:

• Stale draft flag: active is false, flag is more than 30 days old (based on created_at), and activity logs confirm it was never activated.
• Severity: INFO · Category: Cleanup
• Report: "This flag has been inactive for N days and was never activated."
• Action: Delete the flag if it's no longer needed, or activate it.

Note: The "never activated" check requires activity logs. If activity logs are unavailable, skip this sub-check and only report based on the flag being inactive and old:

• Report: "This flag has been inactive for N days. Could not verify whether it was ever activated (activity logs unavailable)."

---

3. Staleness: orphaned experiment flag

Detects flags whose linked experiments are all done.

Look at: experiment_set (list of linked experiment IDs), and for each experiment, check its end_date and archived status via experiment-get.

Findings:

• Orphaned experiment flag: Flag is active: true, has entries in experiment_set, and ALL linked experiments have end_date set (completed) or archived: true.
• Severity: INFO · Category: Cleanup
• Report: "This flag's linked experiments are all completed or archived. The flag is no longer serving an active experiment."
• Action: Roll out the winning variant at 100% and remove the flag from code, or disable the flag.

---

4. Rollout integrity: variant sum

Checks multivariate flag rollout percentages for correctness.

Look at: filters.multivariate.variants (array of {key, rollout_percentage, ...}), experiment_set

Findings:

• Variant sum != 100%: The sum of all rollout_percentage values across filters.multivariate.variants does not equal 100.
• Severity: WARNING · Category: Correctness
• Report: "Multivariate rollout percentages sum to N%, not 100%. Traffic distribution is incorrect."
• Action: Adjust variant rollout percentages to sum to 100%.

• Dead variant (0% rollout): A variant has rollout_percentage: 0 on a flag that is NOT linked to an experiment (empty experiment_set).
• Severity: INFO · Category: Cleanup
• Report: "Variant 'X' has 0% rollout and receives no traffic."
• Action: Either give the variant traffic or remove it.

• Dead condition (0% rollout): A release condition in filters.groups has rollout_percentage: 0.
• Severity: INFO · Category: Cleanup
• Report: "A release condition has 0% rollout and is not serving any traffic."
• Action: Either increase the rollout or remove the condition.

• Manual rollout on experiment flag: A flag with entries in experiment_set has release conditions where rollout_percentage is not the expected even split. This suggests someone manually adjusted the rollout outside the experiment.
• Severity: INFO · Category: Process
• Report: "This experiment flag has manual rollout overrides that differ from the experiment's expected split."
• Action: Remove manual overrides and let the experiment control the variant split.

---

5. Lifecycle

Checks for flags with unstable or high-churn configurations. These checks require activity logs. If unavailable, skip and note it.

Look at: Activity log entries for the flag, created_at

Findings:

• Toggle instability: The flag has been toggled on/off (active → inactive or vice versa) more than 3 times based on activity logs.
• Severity: WARNING · Category: Complexity
• Report: "This flag has been toggled on/off N times. Frequent toggling suggests it may be used as a kill switch or there's uncertainty about its state."
• Action: Consider whether the flag is being used as intended. If it's a kill switch, document that purpose.

• High config churn: The flag has more than 20 activity log entries AND the average rate exceeds 0.5 changes per day (calculated from first to last activity log entry).
• Severity: WARNING · Category: Complexity
• Report: "This flag has been modified N times at a rate of X changes/day. High churn can indicate instability."
• Action: Consider stabilizing the configuration or splitting into multiple simpler flags.

Remediation actions

For each finding type, recommend the appropriate action. Phase 1 is read-only — all actions require the user to make changes manually.

Experiment actions

Flag actions

Recurring patterns

Apply these patterns during full audits (both experiments and flags). Each pattern looks for a cluster of related findings that together suggest a bigger problem.

Experiment setup gaps

Trigger: 3+ experiments have PROCESS-category findings (missing hypothesis, no metrics, no conclusion).

Message:

Multiple experiments lack key setup steps (hypothesis, metrics, or conclusions).

This suggests the team may benefit from an experiment setup checklist or template.

Flag hygiene debt

Trigger: 5+ flags have CLEANUP-category findings (stale drafts, fully rolled out, orphaned experiment flags).

Message:

There are many flags that could be cleaned up. Consider scheduling a flag cleanup session

to remove stale flags from the codebase and reduce unnecessary flag evaluations.

Experiment-flag disconnection

Trigger: At least one experiment has a "stopped with active flag" finding AND at least one has a "mid-run flag change" finding.

Message:

Some experiments have flags that were modified during their run, and others were stopped

but their flags are still active. This suggests the experiment-flag lifecycle is not well-coordinated.

Consider establishing a post-experiment cleanup process.

Reporting patterns

When a pattern triggers:

1. Add a "Recurring patterns" section after individual findings. 2. List each triggered pattern with its message. 3. These are always INFO severity — they are observations, not individual findings.