Diagnosing Experiment Results

Diagnosing experiment results

This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on?

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged [HIGH], [MEDIUM], or [LOW] based on how strongly it's verified — [HIGH] is verified directly in PostHog code, [MEDIUM] is partially or team-source verified, [LOW] describes SDK/external behavior that wasn't verified here. Treat [LOW] items as hypotheses to test, not facts to assert.

Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the finding-experiments skill first to resolve it to a concrete ID.

Call experiment-get and pull these fields. They are inputs for almost every diagnostic:

• parameters.feature_flag_variants[].rollout_percentage — the variant split
• parameters.rollout_percentage — the overall rollout (% of users entering the experiment)
• exposure_criteria.multiple_variant_handling — defaults to "exclude" if absent
• exposure_criteria.exposure_event — null means default $feature_flag_called
• exposure_criteria.filterTestAccounts — defaults to true
• feature_flag.active, status (draft / running / paused / stopped), start_date, end_date
• feature_flag.filters.groups[].variant — any non-null value is a forced-variant override on the

matched cohort (release-condition assignment, not randomized). Surfaces A7 by default.

• stats_config — Bayesian (default) or Frequentist

Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in references/diagnostic-snapshot.md. Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.

Step 2 — Match symptom to diagnostic

If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes — do not guess.

Step 3 — Surface every diagnostic the evidence supports

After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic that applies before recommending an action.

Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a _systematic_ bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a _statistical_ pattern (e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the bundled advice.

Only list mechanisms that have a path to verification in the project state — config (from experiment-get), snapshot data, activity log, or repo source. Config-derived mechanisms count: an 80/20 split with default multiple_variant_handling="exclude" is visible in experiment-get and is therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean variant ratio) is not.

Diagnostic groups

A — Bias & skew

Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants. Covers the uneven-split + Exclude interaction, SRM, identity fragmentation, bootstrap × /decide mismatch, and flag/experiment state inconsistency.

→ See references/bias-and-skew.md

B — Empty experiment / 0 exposures / "not enough data"

A frequent pain point. Covers SDK call (wrong evaluation method, identify() timing, dedup), exposure capture (custom event missing variant property, required properties, ad-blockers), and exposure-criteria match (test-account filter, eligibility ordering, events firing before exposure).

→ See references/empty-experiment.md

C — Significance / interpretation traps

Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple comparisons, low-volume variance, peeking / early stopping. Includes the legacy stats issue (A/A tests historically over-fired before the new Bayesian module) and how the win-probability methodology changed in Jan 2025 (single test vs control, not control vs all variants).

→ See references/interpretation.md

D — Numbers don't match (PostHog vs the user's SQL / raw count)

The experiment page applies an exposure scope, $multiple exclusion, test-account filter, and date range that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats), breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user confusion, and the recordings-panel-vs-stats divergence.

→ See references/numbers-vs-sql.md

E — Surprises after mid-run changes (incl. lifecycle and retention quirks)

Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding metrics mid-run is p-hacking; ship-variant can rewrite the flag in surprising ways; reset clears results not the flag. Also covers retention-metric quirks (first-event-must-be-after-exposure design), "matured users" filtering, and long-term vs short-term metric divergence.

→ See references/mid-run-changes.md

Step 4 — Calibrate recommendations to experiment state

Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the experiment's current state permits.

• Draft — config changes are free; recommend and apply.
• Running — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe?

user-visible?) before recommending. See configuring-experiment-rollout and its reference file references/changing-distribution-after-launch.md for the mid-run rules.

• Stopped / archived — the experiment AND its feature flag represent the documented outcome of

the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for the _next_ experiment, or (c) explaining what happened.

On a stopped or archived experiment, don't preemptively offer reversal of a state mutation (ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?", explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't signal. Conditional offers like _"if this wasn't intended, you could…"_ or _"want me to revert it?"_ count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?", "can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface reversal mechanics.

Use consistent terminology: variant _split_ (between variants) is distinct from _rollout_ (overall % entering); the $feature_flag_called exposure event is distinct from a _custom exposure event_; the _Exclude_ / _First seen_ options control multivariate handling, not exposure.

Bias & skew on a running experiment

Variant counts don't match the configured split, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants.

Before diagnosing

Pull three signals first:

1. SRM result on the Exposures tab. A green SRM check at ≥100 exposures rules out real imbalance — the visible split is normal small-sample variance (see C2 in interpretation.md). A red SRM means there is a real assignment or capture problem; proceed below. 2. `$multiple` share. If non-zero, identity fragmentation (A3/A4) is on the table. 3. Configured split. Read experiment-get's parameters.feature_flag_variants[].rollout_percentage — uneven splits amplify whichever bias source is present.

If the symptom is "metric count is far smaller than exposures" (e.g. 10× or 100× gap), walk this file before numbers-vs-sql.md — that shape of divergence is most often a bucketing / identity problem (A3/A4), not a query-scope problem.

Contents

• A1 — Multi-variant exclusion bias on uneven split (the in-app banner)
• A2 — Sample ratio mismatch (SRM)
• A3 — Identity fragmentation (users in both control and test)
• A4 — Bootstrap × /decide variant disagreement
• A5 — Flag/experiment state inconsistency
• A6 — Mid-run flag edits that rebucket already-exposed users
• A7 — Non-randomized assignment via release conditions
• A8 — Migrating the distinct_id strategy during a running experiment

A1 — Multi-variant exclusion bias on uneven split [HIGH]

This is the in-app bias-warning banner's signal. Triggers when all three hold:

• multiple_variant_handling == "exclude" (the default)
• variant rollouts are uneven
• there are _any_ observed $multiple exposures (the backend warning fires above 0.1%

multi-variant share — MULTIPLE_VARIANT_BIAS_THRESHOLD in products/experiments/backend/analysis_health.py)

The warning-vs-visible gap. The backend warning banner fires at > 0.1% $multiple share, but the Exposures tab in the UI hides the $multiple row when share is ≤ 0.5% (MULTIPLE_VARIANT_WARNING_THRESHOLD in frontend/src/scenes/experiments/utils.ts). So a user can see the bias-warning banner _while_ the Exposures tab shows a clean variant split with no $multiple row — they'll ask "why is the warning firing when no users are in $multiple?". When the user reports this disconnect, lead with: the warning is real; the row is hidden because the share is between 0.1% and 0.5%. Pull the exact share from the Step 1.5 snapshot so the explanation is concrete, not abstract.

Mechanism. Multi-variant users are dropped, but the smaller variant loses a _larger fraction_ of its assignments than the larger variant. Multi-device / multi-session / signup-flow users tend to be high-intent — so the smaller variant keeps a low-intent slice and looks worse than it should. This is asymmetric exclusion bias, not a UI bug.

Recommend (in this order):

1. Switch to an equal split. See configuring-experiment-rollout. On a draft experiment this is free. Mid-run it's an anti-pattern — prefer reset or end+restart over changing the split mid-run. 2. Switch `multiple_variant_handling` to `"first_seen"`. See configuring-experiment-analytics. Mid-run this is the low-disruption option — no users switch variants, all already-collected data stays in the analysis. first_seen is less biased than `exclude` for this specific shape, not unbiased: it counts the first variant a user saw and ignores later ones, which still asymmetrically discounts engaged multi-session users. There is no clean fix for the underlying problem; the trade-off is between which bias the user prefers.

A2 — Sample ratio mismatch (SRM) [HIGH]

Open the Exposures tab. PostHog runs a chi-squared test once total exposures ≥ 100 and flags SRM at p < 0.001. The $multiple bucket is excluded from the SRM check (so a high $multiple share is _not_ what triggers SRM — it's that the visible variants don't match the configured rollout).

Verify directly. The exposure-shape query from Step 1.5 already gives the counts. Compare observed vs expected (using parameters.feature_flag_variants[].rollout_percentage) and apply χ². Treat p < 0.001 as SRM.

What it means. The actual variant distribution differs significantly from the configured split — something is biasing variant assignment or exposure capture. Note: low-volume variance can produce splits that _look_ off without being SRM. The chi-squared test accounts for that, so trust the SRM check over the visual ratio.

Investigate, in order. Each item has a _Detect_ (how the agent can verify it from MCP / by asking) and a _Fix path_ (the specific action to recommend — the agent cannot mutate flag conditions via MCP, only read them, so most fixes are precise guidance not direct action):

1. Bot traffic hashing into a single variant. Server-side flag evaluations from bots are deterministic by distinct_id — a single crawler hitting the same path repeatedly hashes into the same variant and skews the visible split. The public troubleshooting docs rank this as the

_Detect:_ check whether the exposure events come from server-side evaluations ( $lib values like posthog-python, posthog-node, posthog-ruby, posthog-go, posthog-php). High server-side share + no bot filter is the signature.

_Fix path:_ enable the Bot detector Hog Function template at _Settings → Data pipeline → Transformations_. Filters known crawler user agents before ingestion. Cannot be enabled via MCP — guide the user to the UI.

<!-- Source for maintainers: docs at https://posthog.com/docs/experiments/troubleshooting#diagnosing-sample-ratio-mismatch-srm (item 1, "ranked by frequency"). Template lives in posthog/api/test/__data__/hog_function_templates.json — search for "known_bot_filter_list". -->

2. `identify()` timing. Late identify() fragments users into multiple distinct_ids and skews exposure. (See A3 for the mechanism.)

_Detect:_ the distinct_ids / persons ratio from Step 1.5 — noticeably > 1 is the signal.

_Fix path:_ call identify() _before_ the flag is evaluated. Do not call reset() between sessions (only on logout). SDK code change — guide the user.

3. Wrong evaluation method. Single-flag accessors fire $feature_flag_called; bulk and payload-only accessors don't. See B1 in empty-experiment.md for the per-SDK table.

_Detect:_ ask the user which method they call to read the flag.

_Fix path:_ switch to getFeatureFlag() / get_feature_flag() / framework hook.

4. Complex release conditions. Property-based targeting can create uneven assignment when the property is missing or evaluates differently at flag-call time.

_Detect:_ call feature-flag-get-definition and inspect filters.groups[].properties — does any condition reference a property that might be missing or late-loaded? Note that non-randomized release conditions (forced overrides) also produce pre-exposure bias.

_Fix path:_ simplify conditions, or test with a clean 50/50 rollout and no property conditions to isolate. Cannot mutate flag conditions via MCP — guide the user.

5. Ad-blockers / network drops. Prevent flag calls from reaching PostHog at all.

_Detect:_ indirect — partial-data hint. If the user's expected traffic is much higher than captured exposures and other causes are ruled out, this is the residual.

_Fix path:_ set up a reverse proxy so capture requests come from the user's own domain. Typical capture lift: 10–30%. Infrastructure change — guide the user.

6. Bootstrap × `/decide` disagreement. Server-rendered apps with bootstrap enabled can emit two $feature_flag_called events for the same person under different IDs. See A4.

_Detect:_ the $used_bootstrap_value discriminator query in A4.

_Fix path:_ pass distinctID in the bootstrap payload when the server knows the identity; use bootstrap with server-side local evaluation, not alone. Code change — guide the user.

7. Server-side / local-evaluation drift. Local-evaluation flag definitions refresh on an SDK-specific interval (typically tens of seconds). If the flag was edited mid-run, exposures captured during the refresh window use the old definition.

_Detect:_ hard to verify from data alone — timing-based. Cross-check with feature-flags-activity-retrieve to find recent edits, then ask whether the user's server-side fleet is configured for local evaluation.

_Fix path:_ lower the local-eval refresh interval, or avoid mid-run flag edits. Cannot reach via MCP — guide.

8. Flag-persistence-across-auth (experience continuity). This setting (ensure_experience_continuity on the flag) is incompatible with local evaluation; mixing them produces inconsistent assignments. Native mobile auth flows that combine both are particularly susceptible.

_Detect:_ feature-flag-get-definition returns ensure_experience_continuity. If true _and_ local evaluation is in use server-side, that's the conflict.

_Fix path:_ pick one. For pre-auth experiments, device-ID bucketing is often the better fix (see A3) — this option is easy to miss; surface it explicitly. Flag mutation needed — cannot via MCP, guide the user.

9. Flag-condition changes via the API. The experiment UI locks flag conditions on a launched experiment, but the API does not enforce the same restriction. A tooling pipeline can quietly skew the split.

_Detect:_ feature-flags-activity-retrieve { id: <feature_flag_id> } — scan results[].detail.changes[] for field == "filters" after the experiment's start_date. The diff shows the exact condition change.

_Fix path:_ revert via the flag UI (cannot mutate flag conditions via MCP). If the change was substantial, treat the post-change window as contaminated and consider reset + relaunch.

10. Server-side SDK dedup cache overflow [LOW]. Server-side SDKs (Node, Python) dedup exposure events using an in-memory cache of ~50,000 distinct (distinct_id, flag, variant) entries. On high-throughput servers, earlier entries are evicted and those users fire duplicate exposures after a worker restart — inflating one variant's count if traffic isn't symmetric across the fleet.

_Detect:_ hard from data alone. Ask: roughly how many distinct users does each worker see between restarts? > 50k is the danger zone.

_Fix path:_ shorter worker restart cadence, or fire $feature_flag_called yourself with a custom exposure event you control the dedup window for. Code/infra change — guide the user.

<!-- Source for maintainers: docs at https://posthog.com/docs/experiments/troubleshooting#diagnosing-sample-ratio-mismatch-srm (item 4). Tagged [LOW] until directly verified. -->

A3 — Identity fragmentation (users in both control and test) [MEDIUM]

This is an identity problem, not a bias problem. The user has two (or more) distinct_ids that haven't been linked — PostHog sees them as separate persons, each correctly assigned a variant. The symptom — same human appearing in both control and test — usually shows up as elevated $multiple share.

Verify directly. Two signatures worth checking before recommending:

-- Persons exposed to more than one variant (excluding the synthetic $multiple bucket)
SELECT
  person_id,
  count(DISTINCT properties.$feature_flag_response) AS variants_seen,
  count(DISTINCT distinct_id) AS distinct_ids,
  groupArray(DISTINCT properties.$feature_flag_response) AS variants
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND properties.$feature_flag_response != '$multiple'
  AND timestamp >= '<start_date>'
GROUP BY person_id
HAVING variants_seen > 1
ORDER BY variants_seen DESC
LIMIT 50

A non-trivial count of rows here, or a distinct_ids / persons ratio noticeably above 1 in the Step 1.5 snapshot, points at fragmentation. Pick one or two person_ids and use persons-retrieve to confirm whether they look like cross-device/cross-auth journeys vs the SDK-ordering bugs below.

Common causes:

• reset() was called between sessions (other than on logout)
• identify() ran after the flag was already evaluated
• Cross-device usage without identity stitching
• Cookies cleared between visits, incognito / stealth browsing
• Anonymous → identified transition without flag persistence enabled
• The same user has different anonymous IDs client-side vs server-side, so the flag hash bucket

differs

• Native mobile auth flows where the flag is read before the SDK identifies the user, or where

authentication crosses an SDK boundary (e.g. web → in-app webview)

A note on what's fundamentally fixable vs not. Stitched-identity issues from identify() ordering, cross-domain cookies, and bootstrap timing are real bugs that can be fixed. Multi-device usage and incognito / stealth browsing are _not_ fixable from PostHog's side — and the users who exhibit them tend to be more engaged on average, so excluding the $multiple bucket pulls a non-random slice out of the analysis. There is no clean fix; the recommendation is to _contain_ the problem (scope exposure to the relevant flow so the denominator stays meaningful) rather than eliminate it.

Recommend:

• Audit identify() and reset() ordering — reset() only on explicit logout, identify() before

flag evaluation.

• For experiments spanning logged-out → logged-in flows, consider one of:
• Persist flag across authentication steps (tradeoffs: requires person_profiles: 'always',

incompatible with local evaluation and bootstrapping, adds slight latency)

• Device-ID bucketing — appropriate for landing/marketing/anonymous flows. Keeps the variant

stable across the anonymous→identified transition without the flag-persistence tradeoffs. Many users don't realize this option exists; surface it explicitly when the symptom is cross-auth bucketing.

• For pre-auth experiments, ensure cookies/localStorage persistence is configured (cookies preferred

for cross-subdomain).

• For mobile flows, consider evaluating the flag server-side (local evaluation) once the user is

authenticated rather than on first app open.

A4 — Bootstrap × /decide variant disagreement [MEDIUM]

Specific scenario: server-rendered app with bootstrapping enabled. The $multiple share in this shape can become substantial — well above the trickle you'd expect from normal cross-device traffic alone. Website-only flags (no bootstrap) are unaffected.

Mechanism. The server bootstraps flags using the server-known distinct_id, but the bootstrap payload doesn't include distinctID — so posthog-js initializes with whatever's in persistence (often the anonymous ID). The bootstrap value gets reported under the anonymous ID; then posthog-js calls /decide with whatever ID it has after identify(). When the IDs differ, hash(anonymous_id) ≠ hash(user_id) → different variant bucket → two $feature_flag_called events for two variants.

Verify directly. $feature_flag_called carries two source-discriminator properties:

• $used_bootstrap_value — true when the event came from the client's bootstrap payload.
• locally_evaluated — true when the event came from server-side local evaluation.

A4's signature is a single person who emitted both a bootstrap-sourced event and a non-bootstrap event for the same flag with different $feature_flag_response values:

WITH per_person AS (
  SELECT
    person_id,
    countDistinctIf(properties.$feature_flag_response, properties.$used_bootstrap_value = true) AS bootstrap_variants,
    countDistinctIf(properties.$feature_flag_response, properties.$used_bootstrap_value != true) AS non_bootstrap_variants,
    groupUniqArrayIf(properties.$feature_flag_response, properties.$used_bootstrap_value = true) AS bootstrap_variant_keys,
    groupUniqArrayIf(properties.$feature_flag_response, properties.$used_bootstrap_value != true) AS non_bootstrap_variant_keys
  FROM events
  WHERE event = '$feature_flag_called'
    AND properties.$feature_flag = '<flag-key>'
    AND properties.$feature_flag_response != '$multiple'
    AND timestamp >= '<start_date>'
  GROUP BY person_id
)
SELECT *
FROM per_person
WHERE bootstrap_variants > 0
  AND non_bootstrap_variants > 0
  AND bootstrap_variant_keys != non_bootstrap_variant_keys
LIMIT 50

Non-trivial row count here distinguishes A4 from A3: A3 is identity fragmentation regardless of source, A4 is specifically the bootstrap-vs-/decide mismatch. If the query returns no rows _and_ no events for the flag have $used_bootstrap_value = true anywhere, bootstrap is likely not in play and A4 is unlikely — but absence isn't definitive (older SDKs may not stamp the property). Cross-check by asking whether the user's app is server-rendered with bootstrapping enabled.

Recommend:

• Pass distinctID in the bootstrap payload when the server already knows the identity (e.g. logged-in

users).

• Bootstrap should be used together with server-side local evaluation, not alone.

A5 — Flag/experiment state inconsistency [HIGH]

The experiment view shows a warning banner for any of these states. Each has a specific fix:

Use managing-experiment-lifecycle for the correct lifecycle action.

A6 — Mid-run flag edits that rebucket already-exposed users [MEDIUM]

Any flag edit that changes the inputs to the variant hash rebuckets already-exposed users on their next flag evaluation. Affected users flip variants and get stamped $multiple on subsequent flag calls — driving up the $multiple share and (depending on uneven-split + exclude) feeding A1. The four common shapes:

• Variant rollout change — e.g. taking a variant from 10% to 0%. Users who were bucketed to

the dropped variant get re-hashed into the remaining variants. Residual $feature_flag_response values for the dropped variant in the snapshot (despite a 0% configured rollout) are the fingerprint.

• Bucketing identifier change — user-bucketing ↔ device-bucketing; or changing

bucketing_identifier on the flag. All assignments are re-bucketed because the hash input changes.

• Release-condition change — adding or tightening release conditions can change which group a

user matches, leading the rollout-percentage logic to evaluate differently. Particularly visible when conditions reference late-loaded person properties.

• Variant key rename — renaming a variant key changes the hash input space and rebuckets

everyone. Rare but high-impact.

Detect. feature-flags-activity-retrieve { id: <feature_flag_id> } is authoritative for the diff (the higher-fidelity activity endpoint). activity-log-list { scope: "FeatureFlag" } only shows _who/when_, not _what_ — but a cluster of edits around or after start_date is the fingerprint to pursue further.

Recommend: treat any of these like changing the variant split mid-run — anti-pattern. The already-collected data after the edit window is contaminated by re-bucketed users. Reset and relaunch is the cleaner fix; switching multiple_variant_handling to first_seen is the low-disruption mid-run option (per A1).

Note on sticky flags + device bucketing: these have known tradeoffs. Device bucketing is designed for initially-anonymous users and is incompatible with the standard sticky-flag pattern (which stores a flag value as a person property — anonymous users have no profile to attach it to). If the user wants both, it requires person_profiles: 'always', which is more expensive.

A7 — Non-randomized assignment via release conditions [MEDIUM]

If the user is using release conditions to target specific cohorts to specific variants (e.g. iOS users see test, Android users see control), the resulting assignment is not random. PostHog's statistics assume randomization, so this invalidates the standard significance interpretation.

PostHog doesn't prevent this in the UI — but the user should understand that significance calculations are misleading in this setup.

Verify directly. In experiment-get's response, scan feature_flag.filters.groups[] for any entry where variant is non-null. That field is the per-release-group variant override: any user matching that group's properties[] is forced to that variant rather than being randomly bucketed. A variant: null (or missing field) means the group is randomized normally and A7 doesn't apply.

When the override exists, also check whether the targeted cohort overlaps the project's test-account exclusion list. If the cohort is _in_ the exclusion list, those users are filtered out of the analysis and the override is mostly a no-op for the metric (they were never going to count). If the cohort is _not_ excluded (e.g. an external partner's email domain), the override contaminates the variant assignment for real users.

Recommend: if they need to compare cohorts, run separate experiments per cohort, or use a single random assignment and analyze the cohorts as breakdowns of the same experiment (with the multiple- comparisons caveats from references/interpretation.md). If the override exists by accident (left over from QA / pre-launch validation), remove it: set variant: null on the affected release group, or delete the group entirely. On a young experiment with little accumulated data, reset + relaunch after the edit; on an experiment with significant clean data from before the issue was noticed, treat the post-launch window as contaminated and consider end + relaunch.

A8 — Migrating the distinct_id strategy during a running experiment [HIGH]

If the user is changing how distinct_id is sent (e.g. anonymous → identified user ID, or email-as-ID → stable user ID, or a different identifier altogether) while an experiment is running, every affected person re-buckets the next time the flag is evaluated. The flag's variant assignment is hash(flag_key + distinct_id) — different input, different bucket, possible variant flip mid-experiment.

Recommend:

• Finish or end the running experiment before the identifier migration, then start a fresh

experiment under the new strategy.

• If they have to migrate during the run, expect inflated $multiple and treat the affected window

as contaminated — use reset + relaunch once the migration is complete.

• An "experience continuity" / flag-persistence approach can paper over anonymous → identified

transitions but is not a general substitute for the migration above (see A3 tradeoffs).

Diagnostic snapshot

Before asking clarifying questions, gather evidence directly. Most diagnostics in this skill can be confirmed or ruled out by data — the agent has execute-sql, experiment-stats, feature-flags-activity-retrieve, and activity-log-list and should use them. Treat user-facing questions as a fallback for when MCP cannot answer.

Run this snapshot once and reuse the results across the dispatch table in SKILL.md.

Exposure shape

Powers A1/A2, B0, C2.

-- Default exposure event ($feature_flag_called):
SELECT
  properties.$feature_flag_response AS variant,
  count() AS exposures,
  count(DISTINCT person_id) AS persons,
  count(DISTINCT distinct_id) AS distinct_ids
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND timestamp >= '<start_date>'
GROUP BY variant
ORDER BY exposures DESC

If the experiment uses a custom exposure event (exposure_criteria.exposure_config.event is set in experiment-get), the variant attribution lives in a different property. Adjust both the event filter _and_ the variant projection:

-- Custom exposure event:
SELECT
  properties.`$feature/<flag-key>` AS variant,  -- note: NOT $feature_flag_response
  count() AS exposures,
  count(DISTINCT person_id) AS persons,
  count(DISTINCT distinct_id) AS distinct_ids
FROM events
WHERE event = '<custom-exposure-event>'         -- from exposure_criteria.exposure_config.event
  AND timestamp >= '<start_date>'
GROUP BY variant
ORDER BY exposures DESC

Reason: $feature_flag_called carries $feature_flag (the flag key being evaluated) and $feature_flag_response (the variant returned). Custom exposure events don't carry those — the SDK stamps $feature/<flag-key> onto subsequent events instead. Querying a custom exposure event with $feature_flag_response returns zero rows even when exposure capture is working fine.

Read off:

• Total exposures — < ~100 means "wait" territory (B0, C2); 0 means walk B-series.
• `$multiple` share — non-zero brings A1/A3/A4 onto the table; > ~0.5% is visible to the eye.
• `distinct_ids / persons` per variant — ratio noticeably > 1 (use 1.2 as a soft cue) suggests

identity fragmentation (A3).

• Visible split vs configured split — flag a real SRM only after the chi-squared check (A2);

small-sample noise is normal under ~1,000 per variant.

• Per-variant `last_seen` and exposure trajectory. Aggregate exposure counts can look healthy

while the experiment is dormant — the trajectory is where you see it. Add min(timestamp) AS first_seen, max(timestamp) AS last_seen to the snapshot SQL, and scan the daily exposures.timeseries[].exposure_counts from experiment-results-get. Two shapes to catch:

• One variant's `last_seen` is days or weeks behind the other's. The application is still

firing the flag for one variant but stopped for the other — typically because the code path serving the silent variant was removed in a refactor. Walk B-series footer.

• Total exposures flat for weeks or months on a `running` experiment (both variants stopped

accumulating). The flag-reading call is gone from the application. Confirm via feature-flags-activity-retrieve: if there are no post-launch flag edits, the flag config can't explain the plateau and the cause is application-side. Walk B-series footer.

Ignore `$feature_flag_response = false` / `None` / `null` rows. $feature_flag_called fires on every flag evaluation, including ones that didn't bucket the user into the experiment — flag returned false (user didn't match release conditions), evaluation failed, or the SDK didn't stamp the response. PostHog's experiment query filters these out via in(properties.$feature_flag_response, ['<variant1>', '<variant2>', …]). They can be larger than the real variants combined and they're not bias signals; don't pull them into the variant-balance discussion. The exception is when _every_ exposure is None/false — that's a B-series symptom, not an A-series one.

Recent flag mutations

Powers A6, E7; E5 lives on the experiment, not the flag.

If the user reports a _surprising change_ (variant ratio flipped, distribution off after an edit, flag distribution went 0/100 unexpectedly), pull recent activity _before_ diagnosing further. A8 (distinct_id strategy change) is a code-side change and does _not_ show up here — diagnose it from event-side identity signals, not the activity log.

• `feature-flags-activity-retrieve { id: <feature_flag_id> }` — recent flag edits and their diffs.

Most "why did the numbers change?" surprises trace back to a variant-distribution change visible here.

• `activity-log-list { scope: "Experiment", item_id: <experiment_id> }` — experiment-level edits as

a timeline (the response currently doesn't carry a change diff, so use it for _who/when_, not _what_).

Handing off the snapshot

If the snapshot already disproves a diagnostic, skip it; if it confirms one, lead the response with the evidence ("the data shows X → that's diagnostic Y").

Empty experiment / 0 exposures / "not enough data"

Diagnose by walking the chain: SDK call → exposure event captured → ingested → matches the configured exposure criteria → counted.

Contents

• Quick triage decision tree
• B0 — Fresh-launch check (experiment less than ~15 minutes old)
• B1 — Wrong flag-evaluation method (no exposure recorded)
• B2 — identify() timing
• B3 — $feature_flag_called deduplication per identity
• B4 — Custom exposure event missing variant property
• B5 — Required properties on $feature_flag_called
• B6 — Ad-blockers / network drops
• B7 — Test-account filter hides the data
• B8 — Metric events firing before exposure
• B9 — Eligibility check ordered after the flag check
• B10 — "Variant always undefined / false"
• B11 — Some server-side SDKs don't auto-populate $feature/<key> on subsequent events
• If none of the above: the code path may not be running

Quick triage decision tree

Ask the user (or check directly) in this order:

1. How long ago was the experiment launched? If less than ~15 minutes → see B0 first; this shape often self-resolves and isn't a real setup issue. 2. Has the code that calls the flag been deployed and is traffic flowing through it? If no → the experiment will be empty until that ships. Stop here. 3. Open the Exposures tab. Is `$feature_flag_called` showing for the flag at all (any variant)?

• Some events, but 0 attributed to the experiment → criteria mismatch (B4–B9).
• No events at all → SDK / capture issue (B1, B2, B6, B10).

4. Is `$feature_flag_called` showing for some users but not others? → likely B3 (returning-user dedup) or B11 (SDK doesn't re-emit on every call).

B0 — Fresh-launch check (experiment less than ~15 minutes old) [HIGH]

Newly-launched experiments can show 0 exposures for up to ~15 minutes even when the setup is correct. PostHog precomputes exposure data on a schedule; until the first precomputation lands, the results view falls back to a real-time query path that may briefly read nothing.

Verify directly. experiment-get already returns start_date. Compute now() - start_date — if under ~15 minutes, this is the most likely cause; no need to ask. If start_date is null the experiment isn't actually launched (a different shape — recommend launching).

Cross-check against the Step 1.5 snapshot: if exposures > 0 for _some_ variant, B0 is ruled out and you're looking at B1–B11. If exposures = 0 across the board on a fresh-launch experiment, wait and force-refresh before debugging further. Most cases in this shape resolve on their own.

Pre-computation is gated behind a 12-hour minimum runtime, so the mirage is most acute on freshly-launched experiments and shouldn't recur on experiments older than a few hours.

<!-- Source for maintainers: MIN_PRECOMPUTATION_DURATION_SECONDS in posthog/hogql_queries/experiments/experiment_query_runner.py. Verify before citing. -->

If the experiment is older than ~15 minutes and still shows 0 exposures, walk B1–B11 below.

B1 — Wrong flag-evaluation method (no exposure recorded) [MEDIUM]

Only the _single-flag evaluation_ methods record exposure. The "bulk" and "payload-only" methods don't fire $feature_flag_called — they read from the local flag cache without notifying PostHog.

The pattern across SDKs: methods that ask about one specific flag fire exposure; methods that return the whole flag bag or just a payload don't. Other SDKs (Ruby, Go, PHP, mobile) follow the same shape — when in doubt, check that SDK's docs.

Verify: ask the user which SDK method they're using to read the flag, and whether the value is read directly or pulled from a cached bulk result.

Fix: switch to the single-flag method (getFeatureFlag() / get_feature_flag() / etc.). If the user genuinely needs the bulk accessor, they must additionally fire $feature_flag_called themselves with the right properties (see B5).

B2 — identify() timing [LOW]

identify() must be called before the flag is evaluated. If identify() runs after, the exposure attaches to the anonymous distinct_id, then the person is later identified — splitting them across two distinct_ids and decoupling exposure from later metric events.

Common symptoms:

• Exposure events exist but don't match later events under the same person
• Variant-specific metric counts are far below exposures

Fix: call identify() before flag evaluation. Never re-identify() to a different distinct_id mid-session.

B3 — $feature_flag_called deduplication per identity [MEDIUM]

PostHog SDKs deduplicate $feature_flag_called to avoid flooding ingestion with identical exposure events. The _scope_ of "duplicate" varies by SDK:

• posthog-js dedupes per identity across sessions by default. Returning users who evaluated the

flag before the experiment launched will _not_ re-emit exposure on later visits — they look like they've never seen the flag. Enable advanced_feature_flags_dedup_per_session: true to reset the cache each session.

• posthog-node / posthoganalytics (Python) dedupe per distinct_id within the process

lifetime (in-memory cache: distinctIdHasSentFlagCalls / distinct_ids_feature_flags_reported). The cache resets when the process restarts. On long-lived workers, a distinct_id will only emit one exposure for the lifetime of that worker.

• Mobile SDKs (iOS / Android / React Native / Flutter) typically dedupe per session, not across

sessions — meaning B3's "returning user with stale dedup" shape is largely a web concern. Verify against the specific SDK's docs before quoting an exact policy.

Fix: match the dedup strategy to the user's complaint with a concrete change:

• Web with returning users (`posthog-js`). In the SDK init config, set

advanced_feature_flags_dedup_per_session: true. The cache resets each session, so returning users re-emit exposure once per session and the experiment captures them.

• Server-side long-lived workers (`posthog-node`, `posthoganalytics`). Two paths, pick one:

(a) restart workers more frequently so the in-memory cache flushes more often, or (b) bypass SDK dedup by firing a custom exposure event yourself (see B4) — the experiment can then use that event as its exposure criterion instead of $feature_flag_called. Option (b) is the cleaner fix when you also want the exposure to carry custom properties.

• Mobile (iOS / Android / React Native / Flutter). Mobile SDKs typically dedupe per session

by default, so B3 is rarely the cause on mobile. If a mobile setup is hitting this shape, check the SDK's docs for its specific session/dedup config — there's no single config key that's consistent across all four.

B4 — Custom exposure event missing variant property [HIGH]

If the experiment uses a custom exposure event instead of $feature_flag_called, the event must include `$feature/<flag-key>` with the variant value (e.g. $feature/new-checkout: 'control').

Without it, the experiment can't attribute exposure to a variant — events count as "exposed" but with no variant, which means they're effectively dropped from per-variant calculations.

Verify directly:

SELECT
  count() AS total,
  countIf(JSONExtractString(properties, '$feature/<flag-key>') != '') AS with_variant,
  count() - countIf(JSONExtractString(properties, '$feature/<flag-key>') != '') AS missing_variant
FROM events
WHERE event = '<custom-exposure-event>'
  AND timestamp >= '<start_date>'

If missing_variant is most of total, B4 is the cause.

Fix: set the property on the event in your tracking code, or configure the SDK so it's added automatically. (For some SDKs, only $feature_flag_called populates this automatically.)

Placebo / variant-less experiments still need the property. A "no UX impact" experiment (common for instrumentation-only or breakdown-driven analyses) requires $feature/<flag-key> on the custom exposure event just like any other experiment. PostHog uses the property for _variant attribution_, not for product behavior — without it, exposures land in the None/null variant bucket and the results page reads as empty.

B5 — Required properties on $feature_flag_called [HIGH]

The event must carry:

• $feature_flag_response — the variant value
• $feature_flag — the flag key

…on every flag retrieval, even when the variant doesn't change.

Verify directly:

SELECT
  count() AS total,
  countIf(properties.$feature_flag_response != '' AND properties.$feature_flag != '') AS well_formed
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND timestamp >= '<start_date>'

A gap between total and well_formed confirms B5.

Fix: if a custom or third-party path is firing the event, ensure both properties are set.

B6 — Ad-blockers / network drops [MEDIUM]

Common cause of partial or zero data. The SDK call goes out, but the request never reaches PostHog.

Fix: set up a reverse proxy so capture requests come from the user's own domain, which ad-blockers don't block.

B7 — Test-account filter hides the data [HIGH]

exposure_criteria.filterTestAccounts defaults to true. If the user's own traffic matches the project's test-account filter (e.g. their email domain is in the filter), their events are excluded from the experiment.

Verify directly. Pull the filter from project settings — project-get { id: "@current" } returns test_account_filters, an array of { key, type, value, operator } conditions (type is event or person; operator is the usual filter operator set: is_not, not_icontains, exact, etc.). Two ways to use it:

1. Read the live filter back to the user. Summarize the rows in plain language so they can recognize whether their own traffic matches one. Don't assume what the rows contain — they vary per project (common shapes: email-domain exclusions, localhost host filters, internal IP ranges, specific cohorts). 2. Estimate the exclusion rate. For each filter row, translate to HogQL and count events that _would be_ dropped. Example for a person-property filter:

   SELECT count() AS would_be_filtered
   FROM events
   WHERE event = '$feature_flag_called'
     AND properties.$feature_flag = '<flag-key>'
     AND timestamp >= '<start_date>'
     AND person.properties.<key> <operator> <value>  -- one row from test_account_filters

If that count is most of the exposures, B7 is the cause.

Fix: temporarily toggle filterTestAccounts off to confirm. Audit and adjust the filter conditions if needed.

B8 — Metric events firing before exposure [HIGH]

Metric events that occur before a user's first exposure are ignored. Only events after exposure are included in the calculation.

Common cause: the exposure event fires too late in the user journey. For example, if the metric event is signup_completed and the exposure event is on a checkout page that the user only reaches _after_ signup, exposures will lag the metric and the metric appears to barely register.

Verify directly:

WITH exposures AS (
  SELECT person_id, min(timestamp) AS first_exposure
  FROM events
  WHERE event = '$feature_flag_called'  -- or the custom exposure event
    AND properties.$feature_flag = '<flag-key>'
    AND properties.$feature_flag_response != '$multiple'
    AND timestamp >= '<start_date>'
  GROUP BY person_id
)
SELECT
  countIf(e.timestamp < x.first_exposure) AS before_exposure,
  countIf(e.timestamp >= x.first_exposure) AS after_exposure,
  countIf(x.first_exposure IS NULL) AS no_exposure
FROM events e
LEFT JOIN exposures x ON e.person_id = x.person_id
WHERE e.event = '<metric-event>'
  AND e.timestamp >= '<start_date>'

If before_exposure dominates, the exposure event is firing too late in the journey — confirmed B8. If no_exposure dominates, the user isn't getting bucketed at all (back to B1/B2/B10).

Fix: capture exposure at the first encounter with the experimental change, not later in the flow.

B9 — Eligibility check ordered after the flag check [MEDIUM]

Eligibility filtering should happen before you call the flag — otherwise unaffected users are pulled into the analysis and the picture gets noisy. This shows up as exposures being much higher than expected and metric rates unexpectedly low.

Fix: structure the code as: eligibility check → flag check → render. Not: flag check → eligibility → render.

B10 — "Variant always undefined / false" [MEDIUM]

Almost always one of:

• B1 (wrong evaluation method)
• B2 (identify() timing)
• posthog is not defined (SDK init order — initialize PostHog before any flag call)
• The flag is genuinely off — feature_flag.active === false, or rollout 0%, or the user is outside

release conditions

Fix: walk the user through their SDK setup. Verify in this order: (a) is PostHog initialized? (b) is the flag active and rolled out? (c) is the right variant key being requested?

B11 — Some server-side SDKs don't auto-populate $feature/<key> [MEDIUM]

Some server-side SDKs (notably Ruby; behavior varies across server SDKs) do not automatically add $feature/<flag-key> to subsequent events after the flag is read. This means metric events have no variant property, breakdowns can show "none", and the experiment under-counts.

Verify directly. Compare exposure events to a same-flag metric event under the same person, and check whether $feature/<flag-key> is set on the metric event. If exposures look fine but metric events are missing the property, this is the cause.

Fix: manually set $feature/<flag-key> on the metric events being captured server-side, or capture the metric event from the client where the JS SDK does add it automatically.

If none of the above: the code path may not be running

Two sub-cases:

Never had exposures (the experiment has shown 0 since launch). After B1–B11, check the obvious:

• Has the deploy with the flag-reading code shipped to production?
• Is real traffic flowing through that code path?
• Is the date range correct (start_date in the future, etc.)?

Ask explicitly. The "empty experiment" shape often resolves to a feature flag still on a feature branch that hasn't merged, or a page that calls the flag not being live yet.

Exposures were healthy then stopped (the experiment ran for weeks/months, then the daily exposure count plateaued and never moved again). A different shape — capture and config are both fine; the application stopped calling the flag.

_Verify directly:_

• Read exposures.timeseries[].exposure_counts from experiment-results-get. A flat tail

(e.g. 27,372 → 27,376 over 100 days = +4 new exposures total) is the signature, distinct from a fresh experiment that's still ramping or one that recently launched. Compare to the last_seen per variant from the diagnostic snapshot — both variants flat is a code-path removal; one variant flat while the other still fires is a one-sided refactor.

• Cross-check feature-flags-activity-retrieve { id: <feature_flag_id> }. If there are no

post-launch flag edits, the flag config is unchanged and the plateau cannot be explained by rollout / variant / condition changes. The cause is on the application side.

_Common causes:_

• The flag-reading call was removed in a refactor (most common).
• The page or component that hosts the flag-read was deprecated or rerouted (e.g. URL

restructuring moved the eligible traffic onto a different page that doesn't read this flag).

• A different flag is now serving the same UX (intentional migration that wasn't paired with

ending the original experiment).

_Recommend:_

• If the hypothesis is settled enough: end the experiment with the appropriate conclusion

(won / lost / inconclusive). The metric data accumulated before the plateau is the experiment's documented outcome. Don't ship the variant unless the code path is being restored — an "end + ship" on a dormant flag flips the variant distribution to a UX that isn't being served anyway.

• If you want to keep running the hypothesis: restore the flag-reading call in the

application code, then either continue (and treat the pre-/post-resumption windows separately) or reset + relaunch for a clean comparison window.

SDK-side fallback. If B1, B2, and B10 are all on the table and you can't pin one down, invoke the posthog:diagnosing-sdk-health skill — outdated SDKs are a frequent root cause of the "no exposures at all" shape (missing instrumentation, broken identify() ordering, deprecated flag methods).

Significance & interpretation traps

How to read PostHog experiment results without falling into common interpretation pitfalls.

Contents

• C1 — Peeking / early stopping
• C2 — Low-volume variance (looks broken but isn't)
• C3 — A/A test showing significance
• C4 — Multiple comparisons (no correction across variants or metrics)
• C5 — Bayesian interpretation traps
• C6 — Frequentist interpretation traps
• C7 — Bayesian vs Frequentist confusion (overlapping intervals, p-values)
• C8 — Inconclusive but trending — when is it ok to ship?
• C9 — "Significance reached" notification is not a green light to ship
• C10 — Ship-variant default does not consider any metric result
• C11 — External calculator disagrees with PostHog

C1 — Peeking / early stopping [HIGH]

Watching results live and ending the experiment the moment it looks significant inflates false positives — you're giving randomness more chances to look significant.

In Bayesian: PostHog applies a minimum-sample-size guard before analysis proceeds — a low per-variant floor plus a proportion-validity rule of np > 5 and n(1-p) > 5 for funnel/proportion metrics (legacy stats module: 100 exposures per variant via FF_DISTRIBUTION_THRESHOLD). Early swings within that band are still noise — in the early days of the experiment, significance can flip back and forth a lot.

Recommend:

• Predetermine duration _before_ launching. Use the running-time calculator on the experiment.
• For the duration calculator: shows "Pending" until at least 1 day and 100 exposures.
• Frequentist: PostHog uses α=0.05 by default → a single metric has ~5% chance of false-positive

significance even when nothing changed.

• Don't treat 0.05 as a hard cliff. It's a convention, not a meaningful threshold by itself —

results just below and just above are close to equivalent in evidence.

C2 — Low-volume variance (looks broken but isn't) [MEDIUM]

Symptom: few hundred or fewer exposures per variant; the visible split looks badly off (a roughly 2-to-1 skew at a few dozen exposures is well within normal noise).

Mechanism: With low samples per variant (rule of thumb: under a thousand), the visible split can swing widely from the configured ratio — deterministic-hash variance is large at small samples. PostHog's calculations account for this; the visible ratio is not a bug.

Funnel/proportion-specific validity gates. Beyond the per-variant exposure floor, funnel metrics also need the normal approximation to hold:

• At least 5 conversions per variant.
• n * p ≥ 5 _and_ n * (1 - p) ≥ 5, where p is the conversion rate.

If a variant has very few converters (or, symmetrically, almost everyone converted), the test will refuse to report — not a bug. The fix is the same: more exposures, or accept that the result isn't ready.

<!-- Source for maintainers:

• products/experiments/stats/frequentist/utils.py around the np / n(1-p) check;
• products/experiments/stats/bayesian/tests.py mirrors the rule and raises StatisticError when successes < 5. -->

Recommend: wait. Run longer or increase rollout. Don't read estimates before the running-time calculator threshold (≥1 day and ≥100 exposures).

C3 — A/A test showing significance [MEDIUM]

A/A tests _should_ almost never show significance. If the user reports their A/A test is showing a significant difference, work through:

1. Which stats module is the experiment on? Experiments created before January 2025 may be on the _legacy_ Bayesian module. The new module (rolled out January 2025) corrected several methodological issues that produced over-significant A/A tests in the legacy module — its A/A false-positive rate is much closer to the expected α. 2. Is it actually random chance? Even with a correct methodology, a small share of metric-variant pairs in an A/A test will look significant by chance (this is the false-positive rate, around α). With multiple metrics × multiple variant pairs, _expect_ some to flicker significant. C4 below. 3. Is it actually different exposure handling? If multiple_variant_handling = "exclude" and the A/A flag is producing $multiple users (from identity fragmentation, A3 in bias-and-skew.md), the asymmetric exclusion can produce real differences between two arms that should be identical. 4. Is the implementation correct? A large multiple-fold gap between equal-sized variants is extremely unlikely to be random — instrumentation is the more likely cause. A specific shape worth checking: data-warehouse-source metrics where per-user exposures are joined to a per-group warehouse table (ExperimentDataWarehouseNode with events_join_key: $group_<n> on the exposure side and data_warehouse_join_key on a group-keyed metric table). The LEFT JOIN duplicates each per-group row by the number of exposed users in that group, so a sum metric over-counts proportional to per-group user count. If user counts are balanced but per-group user counts aren't, the sum can swing 5–30% even on a true A/A — and Bayesian reads that as significant under the i.i.d. assumption. _Detect:_ read the generated clickhouse_sql from experiment-results-get, look for an exposures CTE joined per-user to a metric table where the metric is group-aggregated upstream. _Sanity check:_ re-aggregate the warehouse table by org/group once (deduped) and compare to the per-user sum; a large gap confirms repeated-row inflation.

Recommend: if conditions 1–3 don't explain the result, investigate instrumentation rather than assuming the methodology is wrong.

C4 — Multiple comparisons (no correction across variants or metrics) [HIGH]

PostHog does not apply multiple-comparisons correction:

• Across variants — each test variant is compared to control independently
• Across metrics — each metric is tested independently

So with many metrics or many variants, the chance of _some_ spurious significance grows. Concrete math at α=0.05 (the default): with 5 independent metrics, the chance of at least one false-positive is ~23%; with 10 metrics, ~40%. (Confidence level is configurable — see C6.)

Recommend:

• Define a small set of planned, hypothesis-driven metrics up front.
• Treat results as a pattern across planned metrics, not a single "gotcha" significant metric.
• Add guardrail metrics as secondary, not primary.
• Be especially wary of metrics added after seeing data — that's p-hacking. See mid-run-changes.md.

C5 — Bayesian interpretation traps [HIGH]

PostHog defaults to Bayesian. Common misreads:

• "96% chance to win" is about _direction_ (test is better than control), not the magnitude of

the lift. Read the credible interval alongside it.

• Don't ship the moment chance-to-win flips green — the minimum-sample guard means early flips are

within the noise band.

• Non-informative priors. PostHog uses non-informative priors (mean 0, large variance). Early swings

aren't the prior pushing things around — they're the data being sparse.

• Legacy methodology (pre-2025 experiments). Experiments created before January 2025 may use

the legacy methodology (different multivariate semantics, different significance gates). If a user is reading results from an experiment in that window and the numbers look different than expected, see PostHog's legacy-methodology docs page.

C6 — Frequentist interpretation traps [HIGH]

PostHog has Frequentist support (rolled out June 2025). Set in stats_config. Quick rules:

• 95% CI doesn't cross 0 → significant vs control. CI crosses 0 → not significant.
• PostHog uses Welch's t-test as the default — it handles unequal variance between groups, unlike

Student's t-test (which assumes equal variance).

• α = 0.05 by default → ~5% chance of false-positive on a single metric.
• Confidence level is configurable per team (and per experiment). Valid values are 0.90,

0.95, 0.99 — set via default_experiment_confidence_level on the team or confidence_level on the experiment's stats_config. If a user reports a p-value of 0.07 as "significant", they're likely on the 90% setting; check before debugging the math.

• Significance is per-metric. With many metrics, expect some to flicker in/out as the sample grows.

C7 — Bayesian vs Frequentist confusion [MEDIUM]

A frequent source of confusion:

• Overlapping confidence intervals do not imply non-significance in Bayesian. Overlapping intervals

are a _frequentist_ heuristic. In Bayesian, significance is determined by win probability, so overlapping credible intervals can still indicate a clear winner.

• p-values don't apply in Bayesian. A question about "p < 0.05" is a frequentist frame. If the

experiment is on Bayesian (default), redirect to win probability + credible interval.

• Frequentist is opt-in. Most experiments are Bayesian unless stats_config explicitly selects

Frequentist.

C8 — Inconclusive but trending — when is it ok to ship? [MEDIUM]

Shipping an inconclusive result can be defensible when all of these hold:

• A clear primary metric improvement _without_ a guardrail regression
• Strong qualitative conviction (replays, user feedback, intuition)
• The cost of being wrong is low (e.g. easy to roll back via the flag)

Do not ship if the timeseries chart shows a sustained regression — point-in-time significance can flip, but a sustained downward trend on the timeseries is a stronger signal than a snapshot reading.

Recommend the user open the experiment's _timeseries_ view (per metric) — point-in-time significance can flip, but a sustained trend is a stronger signal than a snapshot reading. The agent can also pull this directly via experiment-timeseries-results.

For the qualitative part (replays / intuition), invoke the posthog:analyzing-experiment-session-replays skill — it surfaces variant-level replay patterns and is the right tool when the call is "primary metric is up, no guardrail regression, do we ship?"

C9 — "Significance reached" notification is not a green light to ship [HIGH]

PostHog can mark a metric as significant and send a notification well before the experiment has accumulated enough data for the result to be stable. The verdict can revert as the sample grows. Treat the notification as a _prompt to review_, not an _instruction to ship_.

Before acting on a significance notification, check all of:

• Participants per variant. A minimum-sample guard runs before analysis — a low floor plus

np > 5 / n(1-p) > 5 for proportions (legacy stats module: 100 exposures per variant). That's a floor for analysis, not a sufficiency bar for shipping. Aim for the number the running-time calculator produced when the experiment was set up.

• Days running. For high-stakes ships, wait at least a full week before acting on a

significance flag — shorter windows can swing as the sample grows. This is a working norm, not a product-enforced threshold.

• Pre-planned duration. If the experiment hasn't reached its planned end date, the significance

is "current best estimate", not "settled".

• Variant balance and `$multiple %`. If A/B/skew (bias-and-skew.md) is in play, the

significance verdict is suspect regardless of how large the gap looks.

• Secondary metrics. See C10.

When a previously-significant banner reverts to not-significant, that's not a bug — it's the same analysis updated with more exposures. Explain the difference between _signal seen so far_ and _result confirmed_.

C10 — Ship-variant default does not consider any metric result [HIGH]

The End-experiment modal pre-fills the "Variant to keep" selector with the first non-control variant (feature_flag_variants[1].key) every time it opens. There is no significance check, no primary-metric direction check, and no guardrail check feeding that default. The "End experiment" button is gated by selecting a conclusion (won / lost / inconclusive / stopped early), _not_ by touching the variant selector — so a user who picks a conclusion and clicks through without re-examining the variant ships the position-default variant. The only way to end without rewriting the flag is to manually clear the variant selector before clicking; the modal does not prompt for this.

The modal also asks how to release the chosen variant, with two radio options:

• Roll out to the experiment population (default, recommended) — variant distribution flips

to 100/0 for the chosen variant; the flag's existing release conditions and per-user variant overrides are preserved. Only users already in the experiment's population see the variant.

• Roll out to all users — additionally prepends a catch-all release condition that overrides

existing release conditions and per-user overrides. Anyone hitting the flag gets the chosen variant.

The release-mode choice doesn't read metrics either; the safer "experiment population" option is the default. If the user clicks through without re-examining, they get the safer behavior on release mode but still the position-default _variant_ — those are independent risks.

<!-- Source for maintainers: FinishExperimentModal in frontend/src/scenes/experiments/ExperimentView/components.tsx. Verify before citing. -->

Recommend: before clicking "End experiment", do three things:

1. Manually review every metric — primary direction and significance, plus every secondary / guardrail metric. The position-default is not a "winner". 2. Explicitly choose the variant to keep — either re-pick from the dropdown (after reviewing metrics) or clear it to end without shipping. Don't accept the pre-fill silently. 3. Confirm the release mode matches intent — "experiment population" keeps the variant scoped to current targeting; "all users" overrides existing release conditions and per-user overrides. The default is the safer choice; flag any non-default selection back to the user explicitly.

If any guardrail is trending negative, or the primary isn't actually significant, the safe move is to keep control rather than ship the position-default. This matters most for sophisticated users who set guardrails for a reason — they are exactly the population the default will mislead.

C11 — External calculator disagrees with PostHog [MEDIUM]

A common case: conversion counts from the experiment page get pasted into an online A/B calculator, which returns a different verdict ("not significant" vs PostHog's "significant", or vice versa).

Two questions to ask before debugging stats:

1. Which methodology? PostHog is Bayesian by default. Most online calculators are Frequentist. The two answer different questions; they will not agree on borderline cases. If the user wants a Frequentist comparison, flip stats_config and re-read (see C6). 2. Are the inputs actually the same? The numbers on the experiment page are post-scope — $multiple excluded, test accounts filtered, exposure-bounded date range, per-user aggregation for trends, conversion-window applied for funnels. An online calculator gets none of that — if the user typed in raw event counts they grabbed from SQL, the calculator and PostHog are computing on different populations, and disagreement is expected.

After confirming both methodology and inputs match, if the disagreement persists, treat it as a real anomaly worth investigating with the experiment URL.

Surprises after mid-run changes (incl. lifecycle and retention quirks)

Anything that changed _after_ the experiment was launched, plus the retention-metric and long-term quirks that produce unexpected counts even without an explicit change.

Contents

• E1 — Increasing rollout (safe)
• E2 — Decreasing rollout (caution)
• E3 — Changing the variant split (anti-pattern)
• E4 — Adding/removing variants (blocked, but historical traces)
• E5 — Changing exposure criteria mid-run
• E6 — Adding metrics mid-run (p-hacking)
• E7 — "Ending" / "shipping a variant" rewrites the flag
• E8 — Reset clears results, not the flag
• E9 — Pause forces control on existing test users
• E10 — Retention metric: start event must occur after exposure
• E11 — "Matured users" filtering
• E12 — Long-term vs short-term metric divergence
• E13 — Editability locks (legacy experiments, ended experiments)
• E14 — Flag cleanup is limited after the experiment is archived
• E15 — Restarting an experiment with new variants

E1 — Increasing rollout (safe) [HIGH]

No users switch variants; new users are added cleanly. Generally the only change safe to make on a running experiment.

E2 — Decreasing rollout (caution) [MEDIUM]

Users currently in a test variant who fall outside the new rollout will switch back to the default experience (if they stay active of course). This is a visible UX disruption — the feature they had disappears.

Their data also becomes harder to interpret statistically. Their prior exposures _stay counted_ against the test variant in the analysis. The numerator and denominator already include them. Reducing rollout doesn't retroactively un-bucket; it only stops new exposures and flips re-evaluations. The metric reading after a rollback mixes "pre-rollback test behavior" with "post-rollback default behavior" for the same users — which is what makes it harder to interpret, not a loss of data.

Recommend: if the user wants to reduce rollout to _contain blast radius_ on a problem variant, rather end the experiment instead — that removes the variant cleanly and locks the result. If they genuinely want to shrink exposure while keeping the experiment alive, treat metric readings from the rollback window onward as mixed and discount them when drawing conclusions.

E3 — Changing the variant split (anti-pattern) [HIGH]

Moves bucket boundaries; users may be reassigned between variants. Creates $multiple users, who then get excluded (default) or attributed to first-seen. Either way, introduces bias.

Recommend: reset the experiment if early; end and start a new one if significant data exists.

Related shape — the flag's split at launch isn't what the user thinks. When a user reports "one variant has no traffic at all" or "the split doesn't match what I configured", the cause is sometimes not a _mid-run_ change but a _pre-launch_ edit that wasn't visible from the experiment view.

Verify directly. feature-flags-activity-retrieve { id: <feature_flag_id> } returns the full edit history with diffs. Scan results[].detail.changes[] for field == "filters" entries and read the last multivariate.variants[] before/after pair _before_ the entry where field == "active" flips false → true (the activation event). That value is the split the experiment actually launched with. If it doesn't match parameters.feature_flag_variants as the user described setting it, the launch state itself is the cause — no mid-run change is needed to explain the missing-variant data.

Fix path: same as E3 generally — reset + relaunch on a young experiment with little data; end + relaunch on one with significant accumulated data. Set the flag's variants to the intended split _before_ clicking launch on the relaunch.

E4 — Adding/removing variants (blocked, but historical traces) [HIGH]

PostHog blocks adding/removing variants on running experiments. If the user managed to do it earlier (or directly via the flag UI before the block was in place), expect $multiple exposures in the data.

Recommend: treat the post-change window as contaminated. Reset (E8) and relaunch if the contamination dominates the run, or end + start a new experiment with a fresh flag (E15) if significant clean data exists from before the change.

E5 — Changing exposure criteria mid-run [HIGH]

Edits to exposure criteria after launch can produce surprises — exposure event swap, multivariate handling change, or test-account filter toggle all change _which_ events count. Two specific cases:

• Switching multiple_variant_handling from exclude → first_seen mid-run is the **low-disruption

way to mitigate uneven-split exclusion bias** on already-collected data. No users switch variants; all data stays.

• Other exposure-criteria changes re-process historical exposures under the new criteria, which can

shift numbers without any actual change in user behavior. Communicate this to the user before they panic.

If the user is also changing how distinct_id is sent (e.g. anonymous → identified, email → user ID), that's a different shape — see bias-and-skew.md A8. Identifier migration mid-run re-buckets users; exposure-criteria edits don't.

E6 — Adding metrics mid-run (p-hacking) [MEDIUM]

Choosing what to measure _after_ seeing data biases your results. Each additional metric is another result to interpret, and with no multiple-comparisons correction (see interpretation.md), the chance of _some_ metric looking significant by chance grows.

If the user is hunting for a significant metric after the fact, that's p-hacking — not a real result.

Note: retroactive metric _addition_ is technically supported (the metric is calculated for the full experiment duration), but using it to fish for significance is a methodology problem, not a tool limitation.

E7 — "Ending" / "shipping a variant" rewrites the flag [HIGH]

Shipping a variant rewrites the linked feature flag's variant distribution: the chosen variant gets 100% of the variant distribution, every other variant goes to 0%. The flow has two release modes — pick carefully:

• Roll out to the experiment population (default, recommended). Existing release conditions on

the flag are preserved untouched. The chosen variant is served only to users who already match those conditions, and per-user variant overrides continue to apply. No catch-all release condition is added.

• Roll out to all users (explicit opt-in). In addition to the variant-distribution flip,

a catch-all release condition is _prepended_ to the flag's release groups with the literal description _"Added automatically when the experiment was ended to keep only one variant."_ This overrides existing release conditions and bypasses per-user variant overrides — anyone hitting the flag now gets the chosen variant.

Both modes flip the active variant ratio to e.g. 0/100 and mint a new flag version. The catch-all release condition is the discriminator between modes.

If the flag distribution suddenly flipped after a metric edit or end action: this is the most likely cause. Check the experiment's recent edits and any ship_variant calls. Recover by adjusting the flag's release conditions back to the experiment split, or by resetting + relaunching the experiment.

Verify directly. Call feature-flags-activity-retrieve { id: <feature_flag_id>, limit, page }. Scan results[].detail.changes[] for field == "filters":

• A multivariate.variants[] diff showing the rollout flip (typical signature: 50/50 → 0/100), and

a separate field == "version" bump → E7 is confirmed.

• Additionally, look inside after.groups[].properties[].description for the literal string

_"Added automatically when the experiment was ended to keep only one variant."_ If present, this was a "roll out to all users" ship and the new release condition overrides the flag's prior targeting and per-user overrides. If absent (release groups unchanged), this was a "roll out to the experiment population" ship — the variant distribution flipped but targeting is intact.

The MCP tool that performs this rewrite is experiment-ship-variant. It takes release_to_everyone: bool (defaults to false = "roll out to the experiment population"); the agent should confirm the release mode with the user before invoking, in addition to the variant key.

Note: activity-log-list { scope: "Experiment", item_id: <id> } will _not_ tell you this — that endpoint returns activity: "updated" with no change diff. Use the feature-flags-activity-retrieve tool.

Default to control on ambiguous ships. If the user is unsure which variant to ship — primary unclear, secondaries mixed, or they're still investigating — recommend shipping control. Accidentally rolling out control is a no-op; accidentally rolling out a test variant flips the variant distribution to a not-validated change. If the user _also_ picks "roll out to all users", the blast radius extends past the experiment's existing population — discourage this combination when the user sounds uncertain.

E8 — Reset clears results, not the flag [HIGH]

Reset returns the experiment to draft and clears start_date, end_date, conclusion, archived. Events already captured still exist but won't be applied to the experiment unless start_date is set appropriately after relaunching. The feature flag is left untouched — users continue seeing their assigned variants during the reset window.

Use case: suspected bias in the existing data, and the user wants to start a clean comparison. Reset + adjust + relaunch is the right path.

E9 — Pause forces control on existing test users [HIGH]

Pause sets the flag's active=false. The flag stops returning a variant via /decide, so users fall back to the application default — typically control. Test users effectively switch back to control during the pause window. No new exposure events fire while paused.

Implication: if the user paused and then resumed, the test variant population had a window of control-like behavior. Their data during the pause is mixed.

Recommend: when interpreting results that span a pause window, surface the pause dates from the activity log (activity-log-list { scope: "Experiment", item_id: <id> }) and explain that the metric data during that window mixes test-variant users with control-like behavior. If the pause was long relative to the run, consider reset + relaunch over interpreting the contaminated data.

E10 — Retention metric: start event must occur after exposure [HIGH]

PostHog's retention metric for experiments requires the start event to occur after the user's first exposure. This is the same design as all other metric types — the analysis question is "what is the effect of this feature _after_ a user sees it?"

`start_handling` (`FIRST_SEEN` vs `LAST_SEEN`) does _not_ relax this. It only picks _which_ post-exposure start event anchors the retention window when a user has multiple: FIRST_SEEN uses min(timestamp), LAST_SEEN uses max(timestamp) — but both are computed over events already filtered to timestamp >= first_exposure_time. Pre-exposure start events are dropped before the min/max ever runs.

<!-- Source for maintainers: _build_start_after_exposure_predicate and _build_start_event_timestamp_expr in posthog/hogql_queries/experiments/experiment_query_builder.py. The CTE INNER JOINs on start_events, so users with only pre-exposure start events are excluded entirely. -->

An alternate question — "does this feature change the standard _pre-anchored_ retention metric?", where the start event can be before exposure — isn't supported on experiments. The workaround is to track that metric separately in product analytics.

If retention undercounts unexpectedly: confirm that the start event has post-exposure occurrences for the affected users. Users whose only start events are pre-exposure are excluded entirely — they don't appear in the retention denominator.

E11 — "Matured users" filtering [HIGH]

Some metrics now support a "Only count matured users" toggle — users whose exposure was at least N days ago. Useful for retention/long-term metrics where freshly-exposed users haven't had time to convert yet.

Implication: turning this on reduces the user count in the analysis (recent users excluded) but makes per-user metric values more comparable across cohorts. If the user count drops unexpectedly, check whether this toggle is enabled.

E12 — Long-term vs short-term metric divergence [MEDIUM]

Primary (short-term) and secondary (long-term) metrics moving in different directions is normal — a checkout-flow change might lift conversion now but hurt retention later.

Recommend:

• Keep the short-term metric as primary and long-term as secondary — don't promote long-term to primary

just because it disagrees.

• Use holdouts for sustained measurement; compare outcomes over time across the holdout vs the

rolled-out cohort.

• For deeper segment analysis, click "Explore results" → filter the funnel/trend by segment, or use

session replays to see what behavior differs between variants.

E13 — Editability locks (legacy experiments, ended experiments) [HIGH]

• Legacy experiments (created before the new query runner) — metrics can no longer be edited.

A "This is a legacy experiment" notice appears in the UI. Duplicate the experiment to get it onto the new engine.

• Ended experiments — variant keys, exposure criteria, and traffic split can't be edited. If

edits are needed, clone, or reset (E8) and re-launch.

If the user is fighting an editability lock, that's a sign the experiment should be cloned or reset rather than worked around.

Legacy fingerprint in `experiment-results-get`. A common downstream symptom of the legacy-experiment case is that the metric line is rendered but the per-variant result block is empty — metrics.primary.count is non-zero, but the entry under results[] has no chance_to_win, no credible_interval, no significant, no step_counts. Exposures are fully populated; only the metric output is missing.

Verify directly (no interview needed). In experiment-get's response:

• metrics[].kind == "ExperimentFunnelsQuery" or "ExperimentTrendsQuery" (not "ExperimentMetric")

— these are the legacy metric kinds.

• filters.migrated_at is set — the experiment was migrated from the pre-new-runner schema.
• stats_config is empty / missing the method field — new-runner experiments carry

stats_config.method: bayesian (or frequentist).

When all three line up, the verdict is legacy methodology, not data corruption. Resaving the metric on the legacy experiment is not supported.

Fix path: duplicate the experiment to land it on the new runner (the new copy will carry the new metric kind and a populated stats_config); recreate the primary metric there; relaunch. Alternatively, end the existing experiment with a documented conclusion if the original hypothesis is no longer interesting — the legacy run can't be salvaged in place.

E14 — Flag cleanup is limited after the experiment is archived [HIGH]

Once an experiment is archived, the feature flag stays bound to it:

• The flag cannot be converted back to a boolean.
• The flag cannot be unlinked from the archived experiment.
• The flag cannot be deleted while the link exists.

This forces either a code change (read a different flag going forward) or a new flag for follow-up rollouts. There is no quick fix in the UI.

Recommend: before archiving, confirm the flag's future use. If the user expects to keep using the flag for general rollout after the experiment ends, ship the variant (E7) rather than archive — that leaves the flag in a usable state at the chosen rollout. If they're done with the flag too, keep both the experiment and the flag intact until the calling code has been removed.

E15 — Restarting an experiment with new variants [MEDIUM]

The "restart with different variants" pattern doesn't have a built-in flow. The clean approach is:

1. End the existing experiment (don't reset — reset reuses the same flag and prior $multiple exposures contaminate the new run). 2. Clone the experiment, or create a new one. 3. Create a new feature flag rather than reusing the previous one — this avoids inheriting cached $feature_flag_called events from users who saw the prior variants. 4. Launch the new experiment under the new flag.

Reusing the same flag with new variants on a new experiment is technically possible but tends to produce confusing exposure histories and prior-variant attribution in the metric data. Only do this if the user is explicit about wanting to keep historical bucketing comparable.

PostHog numbers don't match the user's SQL / raw count

The experiment page applies a specific scope that ad-hoc SQL almost never replicates. A common pattern: SQL is written "to verify" experiment numbers and the results don't match — most of the time, the experiment numbers are correct and the SQL is missing one or more scope filters.

Before walking this file

If the gap between exposures and downstream metric counts is very large (the metric is one or two orders of magnitude smaller than exposures), don't anchor on SQL reconciliation. That shape of divergence is most often a bucketing or identity-resolution problem, not a query-scope problem — walk bias-and-skew.md first (especially A3 / A4) and only come back here once identity is ruled out. The symptom often surfaces as "the numbers don't match", but the agent should route it to A before D.

Contents

• D1 — Scope mismatch checklist (the eight sources)
• D2 — Funnel: only first→last step counts for stats
• D3 — Breakdowns read from the _exposure_ event, not the metric event
• D4 — "Sum of revenue" = mean of per-user totals (not raw total)
• D5 — Property breakdowns silently return "none" for missing properties
• D6 — Recordings panel ≠ statistical calculation
• D7 — Conversion-window anchoring (differs by metric type)
• D8 — Cached results can lag behind ingestion
• D9 — Applying a filter doesn't change the user count
• D10 — No "current person properties" toggle on experiment metrics
• D11 — Metric definition traps (empty event filter, HogQL count(boolean))

D1 — Scope mismatch checklist (the eight sources) [HIGH]

When the user reports "PostHog says X, my SQL says Y", walk this checklist:

1. Exposure scope. The experiment counts only events that occur after the user's first exposure. Raw counts don't filter this way. 2. `$multiple` exclusion. With default handling (exclude), multi-variant users are dropped from metrics. Raw counts include them. 3. Test-account filter. Defaults to true — internal/test users excluded. Raw counts don't typically apply it. 4. Date range. The experiment is bounded by start_date / end_date; raw counts often span more. 5. Variant attribution. The experiment uses the _exposure event's_ variant property; raw counts may pull variant from a different event. 6. Conversion window (funnel metrics only). Events outside the per-user conversion window are not counted. See D7. 7. Per-user aggregation. Mean / ratio metrics aggregate per-user before averaging, so the result is not a raw event-level total. See D4. 8. Winsorization (outlier clamping) on mean metrics. Mean metrics support a percentile-clamp configuration that replaces values below the lower percentile and above the upper percentile with the percentile values themselves before averaging. When enabled, no raw SQL AVG/SUM over the underlying events will reconcile — values are post-clamp.

<!-- Source for maintainers: _build_mean_query_with_winsorization in posthog/hogql_queries/experiments/experiment_query_builder.py -->

Recommend: reproduce the experiment's scope in SQL exactly (start with experiment-get's exposure_criteria, parameters, and stats_config), or accept that ad-hoc SQL will not match by design.

Canonical scope-reproducing HogQL skeleton

Use this as the starting point when the user wants to reconcile. Fill the placeholders from experiment-get. This reproduces sources 1, 2, 4, and 5 from the checklist directly; sources 3, 6, and 7 are noted inline. Source 8 (winsorization) is not reproducible in a one-shot skeleton — if a mean metric uses the percentile-clamp config, no raw AVG/SUM reconciles by design.

WITH exposures AS (
  SELECT
    person_id,
    argMin(properties.$feature_flag_response, timestamp) AS variant,
    min(timestamp) AS first_exposure
  FROM events
  WHERE event = '$feature_flag_called'  -- or exposure_criteria.exposure_event when set
    AND properties.$feature_flag = '<flag-key>'
    AND properties.$feature_flag_response != '$multiple'  -- source 2 (drop if multiple_variant_handling='first_seen')
    AND timestamp >= '<start_date>'                       -- source 4
    AND timestamp <= coalesce('<end_date>', now())        -- source 4
  -- source 3: append the project's test-account filter here when filterTestAccounts=true
  GROUP BY person_id
  HAVING variant != ''
)
SELECT
  u.variant,
  count(DISTINCT u.person_id) AS exposed_users,
  count(e.uuid) AS metric_events,
  -- For "mean of per-user totals" (D4), wrap a per-user sum first then average:
  -- avg(per_user_total) FROM (SELECT person_id, sum(toFloat(properties.<value-prop>)) AS per_user_total ...)
  count(e.uuid) / nullIf(count(DISTINCT u.person_id), 0) AS events_per_user
FROM exposures u
LEFT JOIN events e
  ON e.person_id = u.person_id
  AND e.event = '<metric-event>'                          -- keep this in the JOIN, not WHERE,
                                                          -- so users with 0 metric events still count
  AND e.timestamp >= u.first_exposure                     -- source 1
  AND e.timestamp <= coalesce('<end_date>', now())        -- source 4
  -- source 6: for funnel metrics, also gate e.timestamp <= u.first_exposure + INTERVAL '<conversion_window>'
GROUP BY u.variant
ORDER BY u.variant

Note: keep the metric-event filter in the JOIN's ON clause, not in a top-level WHERE — moving it to WHERE would silently drop exposed users who never produced the metric event (e.event is NULL for them), breaking the denominator.

Notes:

• `multiple_variant_handling = 'first_seen'`: drop the != '$multiple' filter and keep

argMin(...) — it already picks the first variant the user saw.

• Funnel metrics (D2): only the first-step → last-step conversion counts for stats. Intermediate

steps are visualization-only. Reproduce by gating e.event on the _last_ step and joining the exposure as step_0 implicitly.

• "Sum of revenue" (D4): wrap a per-user sum(...) subquery, then avg(...) across users in the

variant — not sum(...) event-level.

• Breakdowns (D3): read the breakdown property from the exposure row in exposures, not from e.
• Test-account filter (source 3): the agent can either pull project settings and inline the

filter, or recommend the user temporarily toggle filterTestAccounts=false and re-read the experiment to confirm that's the gap.

D2 — Funnel: only first→last step counts for stats [HIGH]

For multi-step funnel metrics, statistical significance is always calculated between the first step (exposure) and the final step. Intermediate steps are shown for analysis and visualization but do not affect the significance calculation nor win probability — a user can read a significant intermediate step and incorrectly conclude the whole funnel is significant.

Implication: comparing PostHog's funnel conversion rate to a SQL query that counts intermediate conversions will not match — and that's expected.

The exposure event is automatically prepended as step_0 for funnel metrics, so a 1-step funnel is really a 2-step funnel: exposure → action. Conversion = % of exposed users who reached the action.

D3 — Breakdowns read from the exposure event, not the metric event [HIGH]

When a user adds a breakdown (e.g. "by country" or "by device type") to an experiment metric, the property is read from the exposure event, not the metric event. This is for statistical reasons — the metric event happens after exposure, but the breakdown needs to partition users at the time of exposure.

Implication: if the property only exists on the metric/conversion event (e.g. a checkout event with payment_method), breaking down the experiment by it won't work — every user will appear under "none" because the property isn't on the exposure event.

Recommend: if the user needs to break down by a property only set at conversion, they need to either:

• Set the property earlier so it's present on the exposure event (preferred)
• Use the breakdown in product analytics instead, with the appropriate filter for variant

D4 — "Sum of revenue" = mean of per-user totals (not raw total) [HIGH]

Common confusion: adding "sum of revenue" expecting the raw total of all revenue events across exposed users. PostHog instead returns the mean of per-user totals — for each exposed user, sum their revenue events, then average across users in the variant.

Worked example: user A spent $50, user B spent $10. PostHog reports ($50 + $10) / 2 = $30, not $60. The number looks much smaller than a raw SQL SUM(revenue) over the same time window because it isn't a sum at all — it's the unit on which the statistical comparison runs.

This is the correct way to do statistical comparison (per-user values are the unit of randomization), but it's a frequent source of "why is the number so much smaller than my SQL?" questions.

Recommend: explain the per-user aggregation. For a raw total for reporting, multiply the mean by the user count, or use product analytics for the descriptive total.

D5 — Property breakdowns silently return "none" for missing properties [MEDIUM]

If a user breaks down by a property that doesn't exist on the event being broken down, every value shows as "none" rather than an error. This is silent and confusing.

Verify: check that the breakdown property is actually being captured on the relevant event.

Recommend: if it's the exposure event missing the property, see D3 — set the property earlier in the journey, or capture it on $feature_flag_called directly.

D6 — Recordings panel ≠ statistical calculation [MEDIUM]

The "View recordings" panel on the experiment page applies metric events as filters for finding relevant replays — but those filters don't map exactly to the statistical calculations (e.g. funnel attribution type isn't applied, conversion windows may not be).

Implication: the "story" in recordings can't be reconciled 1:1 with the computed result. Don't debug stats discrepancies via the recordings panel.

Recommend: use recordings to _qualitatively_ understand variant differences (what users actually experienced), not to _audit_ the numbers.

D7 — Conversion-window anchoring (differs by metric type) [HIGH]

The conversion window isn't a single rule — the new query runner applies it differently per metric type:

• Mean / ratio metrics. Events count when

timestamp >= first_exposure_time AND timestamp < last_exposure_time + conversion_window. The _lower_ bound is anchored to the user's first exposure; the _upper_ bound is anchored to their _last_ exposure plus the window. Re-exposure extends the observation period; earlier conversions still count.

<!-- Source for maintainers: _conversion_window_predicate in posthog/hogql_queries/experiments/experiment_query_builder.py (mean/ratio branch). The exposures CTE defines first_exposure_time = min(timestamp), last_exposure_time = max(timestamp). -->

• Funnel metrics. The conversion window is enforced _between consecutive funnel steps_ by the

aggregate_funnel_array ClickHouse UDF — not as a single window from first exposure. Each new exposure event resets the funnel's step-0 anchor, so re-exposure _restarts_ the funnel rather than extending an existing attempt. Ordered funnels skip the SQL-level temporal filter entirely; the per-step gap check in the UDF is the only window enforcement.

<!-- Source for maintainers: funnel-udf/src/steps.rs (per-event step-0 reset and the consecutive-step gap check). experiment_query_builder.py documents the ordered-vs-unordered branch. -->

Implication for SQL reconciliation:

• Mean/ratio reconciliation: gate with

e.timestamp >= u.first_exposure_time AND e.timestamp < u.last_exposure_time + INTERVAL '<window>', not a single window from first exposure.

• Funnel reconciliation: compute step-to-step gaps, not a single window from first exposure. A user

who is re-exposed gets a fresh chance to complete the funnel — your SQL must allow this or PostHog's numbers will look larger than yours.

If the numbers shifted unexpectedly across a query-runner migration, this is the most likely cause: historical pre-migration funnel attribution did not have the per-step gap semantics.

D8 — Cached results can lag behind ingestion [HIGH]

Experiment results are cached for up to 24 hours. Force-refresh (the manual button on the page) bypasses the cache. If pre-aggregation is enabled and a precomputation insert fails, PostHog falls back to a real-time query — which can produce a small inconsistency between two consecutive views, especially on fresh data.

Recommend: if numbers look stale, force-refresh the experiment first before debugging.

D9 — Applying a filter doesn't change the user count [MEDIUM]

Symptom: a filter is added to a metric (e.g. "by device = mobile") and the exposure / user count stays the same — only the conversion side moves. The conclusion looks like "the filter isn't working."

The experiment's denominator is the set of exposed users, fixed at exposure time. A filter on a property of the metric event acts as a _gate within that fixed population_ — it changes who counts as converted, not who counts as in the experiment. The denominator correctly does not shrink.

To shrink the denominator (i.e. only count users who match the filter as part of the experiment at all), encode the eligibility upstream — either in release conditions, or by setting the property on the exposure event itself, or by using a custom exposure event that already filters.

Recommend: explain the scope difference. If the mental model comes from another A/B tool that subset-filters the population on metric properties, name the tool and explain the design choice explicitly.

D10 — No "current person properties" toggle on experiment metrics [MEDIUM]

Insights have a "Use current person properties" toggle (versus as-of-event). Experiment metrics do not expose this toggle — person properties are always evaluated as of the time the event was captured.

This is intentional: the experiment's population needs to be stable across the run. If person properties were re-resolved at query time, the population a user falls into could change over the course of the experiment as their attributes change (plan upgrades, geo moves, etc.), which would invalidate the analysis.

Recommend: for slices by "current state" attributes (e.g. "free vs paid as of today"), use one of:

• A dynamic cohort for "currently paid" users, and target the experiment to that cohort via

release conditions.

• A HogQL expression in the metric filter that joins person properties at query time, accepting

that the answer reflects the current state, not the state at exposure.

• A property captured on the exposure event (e.g. plan tier at the time of exposure), so the

slice is stable and analysable as a breakdown.

D11 — Metric definition traps (empty event filter, HogQL count(boolean)) [HIGH]

Two EventsNode-shaped metric mis-configurations recur. Both produce numbers that look like the data is broken but are actually the metric definition doing precisely what it was asked.

`event: ""` is not "all events". In an EventsNode, the event field is an _equality_ filter against the event name. An empty string matches events literally named "" — i.e. none. The metric's metric_events CTE returns no rows, the LEFT JOIN from exposures produces NULLs on the metric side, and the resulting metric collapses to a constant per user (commonly 1.0 for a mean-shaped boolean count, or 0 for a total math). "All events" as a _user-facing_ concept requires either no event filter at the metric source or a different metric kind — not event: "".

`count(boolean_expression)` counts non-null, not true. A HogQL math_hogql of the form count(properties.X = 'value') counts every event where the expression evaluates (i.e. every event where the property is set, true or false), not events where the expression is true. Use countIf(properties.X = 'value') for the "true" semantics, or sum(toInt(properties.X = 'value')) for an additive form.

Verify directly. Inspect the rendered clickhouse_sql field from experiment-results-get — the metric_events CTE shows the actual WHERE clause and the per-event value expression. If the WHERE contains equals(events.event, ''), the metric is filtering to no events. If the per-event value is a boolean expression wrapped in count(...), the math is counting evaluations not truths. Either signature is dispositive.

Validation signals from PostHog. A validation_failures entry of "baseline-mean-is-zero" on a mean metric is the system's tell that _every_ exposed user contributed 0 to the metric — almost always a total math on a never-matching event filter.

Recommend:

• Replace event: "" with the actual event to measure (or use a metric kind that genuinely means

"all events" — confirm in the metric editor, not by typing "").

• For HogQL math: pick countIf(...) or sum(toInt(...)) over count(...) of a boolean.
• Metric edits on a running experiment recompute the metric over the full duration. Flag this to

the user before recommending so the post-edit numbers don't surprise them. Force-refresh the experiment page after saving.