Product Analytics

Product Analytics

Core Principles

• Metrics over vanity — Focus on actionable metrics tied to business outcomes
• Data-driven decisions — Hypothesize, measure, learn, iterate
• User-centric measurement — Track behavior, not just pageviews
• Statistical rigor — Understand significance, avoid false positives
• Privacy-first — Respect user data, comply with GDPR/CCPA
• North Star focus — Align all teams around one key metric

---

Hard Rules (Must Follow)

These rules are mandatory. Violating them means the skill is not working correctly.

No PII in Events

Events must NEVER contain personally identifiable information.

// ❌ FORBIDDEN: PII in event properties
track('user_signed_up', {
  email: 'user@example.com',     // PII!
  name: 'John Doe',              // PII!
  phone: '+1234567890',          // PII!
  ip_address: '192.168.1.1',     // PII!
  credit_card: '4111...',        // NEVER!
});

// ✅ REQUIRED: Anonymized/hashed identifiers only
track('user_signed_up', {
  user_id: hash('user@example.com'),  // Hashed
  plan: 'pro',
  source: 'organic',
  country: 'US',                       // Broad location OK
});

// Masking utilities
const maskEmail = (email) => {
  const [name, domain] = email.split('@');
  return `${name[0]}***@${domain}`;
};

Object_Action Event Naming

All event names must follow the object_action snake_case format.

// ❌ FORBIDDEN: Inconsistent naming
track('signup');                    // No object
track('newProject');                // camelCase
track('Upload File');               // Spaces and PascalCase
track('user-created');              // kebab-case
track('BUTTON_CLICKED');            // SCREAMING_CASE

// ✅ REQUIRED: object_action snake_case
track('user_signed_up');
track('project_created');
track('file_uploaded');
track('payment_completed');
track('checkout_started');

Actionable Metrics Only

Track metrics that drive decisions, not vanity metrics.

// ❌ FORBIDDEN: Vanity metrics without context
track('page_viewed');               // No insight
track('button_clicked');            // Too generic
track('app_opened');                // Doesn't indicate value

// ✅ REQUIRED: Actionable metrics tied to outcomes
track('feature_activated', {
  feature: 'dark_mode',
  time_to_activation_hours: 2.5,
  user_segment: 'power_user',
});

track('checkout_completed', {
  order_value: 99.99,
  items_count: 3,
  payment_method: 'credit_card',
  coupon_applied: true,
});

Statistical Rigor for Experiments

A/B tests must have proper sample size and significance thresholds.

// ❌ FORBIDDEN: Drawing conclusions too early
// "After 100 users, variant B has 5% higher conversion!"
// This is not statistically significant.

// ✅ REQUIRED: Proper experiment setup
const experimentConfig = {
  name: 'new_checkout_flow',
  hypothesis: 'New flow increases conversion by 10%',

  // Statistical requirements
  significance_level: 0.05,      // 95% confidence
  power: 0.80,                   // 80% power
  minimum_detectable_effect: 0.10, // 10% lift

  // Calculated sample size
  sample_size_per_variant: 3842,

  // Guardrails
  max_duration_days: 14,
  stop_if_degradation: -0.05,    // Stop if 5% worse
};

---

Quick Reference

When to Use What

---

North Star Metric

Definition

A North Star Metric is the one metric that best captures the core value your product delivers to customers. When this metric grows sustainably, your business succeeds.

Characteristics of Good NSMs

✓ Captures product value delivery
✓ Correlates with revenue/growth
✓ Measurable and trackable
✓ Movable by product/engineering
✓ Understandable by entire org
✓ Leading (not lagging) indicator

Examples by Company

NSM Framework

North Star Metric
       ↓
┌──────┴──────┬──────────┬──────────┐
│             │          │          │
Input 1    Input 2   Input 3   Input 4
(Supporting metrics that drive NSM)

Example: Spotify
NSM: Time Spent Listening
├── Daily Active Users
├── Playlists Created
├── Songs Added to Library
└── Share/Social Actions

How to Define Your NSM

1. Identify core value proposition

• What job does your product do for users?
• When do users get "aha!" moment?

2. Find the metric that represents this value

• Transaction completed? (e.g., Nights Booked)
• Time engaged? (e.g., Time Listening)
• Content created? (e.g., Messages Sent)

3. Validate it correlates with business success

• Does NSM increase → revenue increases?
• Can product changes move this metric?

4. Define supporting input metrics

• What user behaviors drive NSM?
• Break into 3-5 key inputs

---

AARRR Framework (Pirate Metrics)

Overview

The AARRR framework tracks the customer lifecycle across five stages:

ACQUISITION → ACTIVATION → RETENTION → REFERRAL → REVENUE

Stage Definitions

1. Acquisition

When users discover your product

Key Questions:

• Where do users come from?
• Which channels have best quality users?
• What's the cost per acquisition (CPA)?

Metrics:

• Website visitors
• App installs
• Sign-ups per channel
• Cost per acquisition (CPA)
• Channel conversion rates

Example Events:

// Landing page view
track('page_viewed', {
  page: 'landing',
  utm_source: 'google',
  utm_medium: 'cpc',
  utm_campaign: 'brand_search'
});

// Sign-up started
track('signup_started', {
  source: 'homepage_cta'
});

2. Activation

When users experience core product value

Key Questions:

• What's the "aha!" moment?
• How long to first value?
• What % reach activation?

Metrics:

• Time to first action
• Activation rate (% completing key action)
• Setup completion rate
• Feature adoption rate

Example "Aha!" Moments:

Slack:     Send 2,000 messages in team
Twitter:   Follow 30 users
Dropbox:   Upload first file
LinkedIn:  Connect with 5 people

Example Events:

// Activation milestone
track('activated', {
  user_id: 'usr_123',
  activation_action: 'first_project_created',
  time_to_activation_hours: 2.5
});

3. Retention

When users keep coming back

Key Questions:

• What's Day 1/7/30 retention?
• Which cohorts retain best?
• What drives churn?

Metrics:

• Day 1/7/30 retention rate
• Weekly/Monthly active users (WAU/MAU)
• Churn rate
• Usage frequency
• Feature stickiness (DAU/MAU)

Retention Calculation:

Day X Retention = Users returning on Day X / Total users in cohort

Example:
Cohort: 1000 users signed up Jan 1
Day 7: 300 returned
Day 7 Retention = 300/1000 = 30%

Example Events:

// Daily engagement
track('session_started', {
  user_id: 'usr_123',
  session_count: 42,
  days_since_signup: 15
});

4. Referral

When users recommend your product

Key Questions:

• What's the viral coefficient (K-factor)?
• Which users refer most?
• What referral incentives work?

Metrics:

• Viral coefficient (K-factor)
• Referral rate (% users referring)
• Invites sent per user
• Invite conversion rate
• Net Promoter Score (NPS)

Viral Coefficient:

K = (% users who refer) × (avg invites per user) × (invite conversion rate)

Example:
K = 0.20 × 5 × 0.30 = 0.30

K > 1: Viral growth (each user brings >1 new user)
K < 1: Need paid acquisition

Example Events:

// Referral actions
track('invite_sent', {
  user_id: 'usr_123',
  channel: 'email',
  recipients: 3
});

track('referral_converted', {
  referrer_id: 'usr_123',
  new_user_id: 'usr_456',
  channel: 'email'
});

5. Revenue

When users generate business value

Key Questions:

• What's customer lifetime value (LTV)?
• What's LTV:CAC ratio?
• Which segments monetize best?

Metrics:

• Monthly Recurring Revenue (MRR)
• Average Revenue Per User (ARPU)
• Customer Lifetime Value (LTV)
• LTV:CAC ratio
• Conversion to paid
• Revenue churn

LTV Calculation:

LTV = ARPU × Gross Margin / Churn Rate

Example:
ARPU: $50/month
Gross Margin: 80%
Churn: 5%/month

LTV = $50 × 0.80 / 0.05 = $800

Healthy LTV:CAC ratio: 3:1 or higher

Example Events:

// Revenue events
track('subscription_started', {
  user_id: 'usr_123',
  plan: 'pro',
  mrr: 29.99,
  billing_cycle: 'monthly'
});

track('upgrade_completed', {
  user_id: 'usr_123',
  from_plan: 'basic',
  to_plan: 'pro',
  mrr_change: 20.00
});

AARRR Metrics Dashboard

## Acquisition
- Total visitors: 50,000
- Sign-ups: 2,500 (5% conversion)
- Top channels: Organic (40%), Paid (30%), Referral (20%)

## Activation
- Activated users: 1,750 (70% of sign-ups)
- Time to activation: 3.2 hours (median)
- Activation funnel drop-off: 30% at setup step 2

## Retention
- Day 1: 60%
- Day 7: 35%
- Day 30: 20%
- Churn: 5%/month

## Referral
- K-factor: 0.4
- Users referring: 15%
- Invites per user: 4.2
- Invite conversion: 25%

## Revenue
- MRR: $125,000
- ARPU: $50
- LTV: $800
- LTV:CAC: 4:1
- Conversion to paid: 25%

---

Extended Reference

Detailed material starting at ## Key Metrics & Formulas has been moved to `reference/extended.md` to keep this skill concise. Load that reference when the task requires the moved examples, command catalogs, checklists, platform details, or implementation templates.

Event Tracking & Data Modeling

Overview

Event tracking is the foundation of product analytics. Every user interaction becomes a structured data point that can be queried, analyzed, and acted upon.

2025 Best Practice: Start with your metrics, then work backward to the events you need. Don't track everything—track what matters.

---

Event-Based Analytics

What is Event-Based Analytics?

Event-based analytics tracks discrete user actions (events) rather than just pageviews. Each event captures:

• What happened (event name)
• Who did it (user ID)
• When it happened (timestamp)
• Where it happened (page, feature)
• How it happened (properties, context)

Event vs. Pageview Tracking

## Pageview-Only Tracking (Old Approach)
❌ User visited /pricing page → pageview count
   • Can't tell if they compared plans
   • Can't tell which plan they viewed
   • Can't correlate with conversion

## Event-Based Tracking (Modern Approach)
✓ User viewed pricing page → pricing_page_viewed
✓ User toggled plan comparison → plans_compared
✓ User clicked "Start Trial" → trial_started
   • Full user journey captured
   • Behavior patterns clear
   • Conversion attribution accurate

---

Event Taxonomy

Event Naming Standards

Pattern: object_action

Objects: user, project, file, payment, message, etc.
Actions: created, updated, deleted, viewed, clicked, shared, etc.

Examples:
✓ user_signed_up
✓ project_created
✓ file_uploaded
✓ payment_completed
✓ message_sent
✓ search_performed

Naming Conventions

Format: snake_case (recommended)
  user_signed_up, checkout_completed

Alternative: camelCase
  userSignedUp, checkoutCompleted

Alternative: Period-separated
  user.signed_up, checkout.completed

AVOID:
  ✗ Mixed case: User_SignedUp
  ✗ Spaces: "User Signed Up"
  ✗ Too vague: "click", "action"
  ✗ Too technical: "post_api_users_create_200"

Event Categories

## User Lifecycle Events
user_signed_up          → Registration complete
user_activated          → Completed first key action
user_invited_teammate   → Sent invitation
user_upgraded           → Changed to paid plan
user_downgraded         → Reduced plan
user_churned            → Canceled or became inactive

## Feature Interaction Events
feature_viewed          → User saw the feature
feature_enabled         → User turned on feature
feature_used            → User interacted with feature
feature_completed       → User finished feature workflow

## Content Events
content_created         → Post, file, project created
content_viewed          → Item viewed
content_edited          → Item updated
content_deleted         → Item removed
content_shared          → Shared with others
content_published       → Made public

## Commerce Events
product_viewed          → Product page viewed
product_added_to_cart   → Added to cart
cart_viewed             → Cart page opened
checkout_started        → Began checkout
payment_attempted       → Tried to pay
payment_completed       → Payment successful
payment_failed          → Payment error
order_shipped           → Fulfillment started
order_delivered         → Fulfillment complete

## Engagement Events
session_started         → User logged in
session_ended           → User logged out
page_viewed             → Page navigation
search_performed        → Search query
filter_applied          → Applied filter
notification_received   → Push/email received
notification_clicked    → Opened notification

## Social Events
user_followed           → Followed another user
message_sent            → Direct message
comment_posted          → Commented on content
like_added              → Liked content
share_completed         → Shared to social

## Error Events
error_occurred          → Application error
form_validation_failed  → Invalid input
api_request_failed      → Backend error
timeout_occurred        → Request timeout

---

Event Properties

Property Schema

// Event structure
{
  event: "checkout_completed",
  timestamp: "2025-12-16T10:30:00.123Z",
  user_id: "usr_7b3f8e2a",
  session_id: "ses_4c9d1a5e",

  // Event-specific properties
  properties: {
    order_id: "ord_123456",
    total_amount: 149.99,
    currency: "USD",
    item_count: 3,
    payment_method: "credit_card",
    shipping_method: "express",
    coupon_code: "SAVE20",
    discount_amount: 30.00,
    tax_amount: 12.00,
    items: [
      { sku: "PROD-001", quantity: 2, price: 49.99 },
      { sku: "PROD-002", quantity: 1, price: 50.01 }
    ]
  },

  // Global context
  context: {
    app_version: "2.4.1",
    platform: "web",
    os: "macOS",
    browser: "Chrome",
    screen_size: "1920x1080",
    locale: "en-US",
    timezone: "America/New_York",
    utm_source: "google",
    utm_medium: "cpc",
    utm_campaign: "holiday_sale"
  }
}

Property Naming

snake_case preferred:
  total_amount, user_id, created_at

Data types:
  Strings:    "pro", "credit_card", "monthly"
  Numbers:    99.99, 5, 1000
  Booleans:   true, false
  Timestamps: "2025-12-16T10:30:00.123Z" (ISO 8601)
  Arrays:     ["tag1", "tag2"]
  Objects:    { plan: "pro", price: 29.99 }

AVOID:
  ✗ Inconsistent naming: totalAmount vs total_amount
  ✗ String numbers: "99.99" (use 99.99)
  ✗ Ambiguous booleans: "yes", "1", "Y" (use true/false)
  ✗ Non-standard dates: "12/16/2025" (use ISO 8601)

Required vs. Optional Properties

// Required for ALL events
{
  event: string,          // Event name
  timestamp: string,      // ISO 8601
  user_id: string,        // User identifier (or anonymous_id)
}

// Recommended for context
{
  session_id: string,     // Session tracking
  app_version: string,    // Version tracking
  platform: string,       // "web", "ios", "android"
}

// Event-specific (varies by event)
{
  properties: {
    // Whatever makes sense for this event
  }
}

---

Tracking Plan

What is a Tracking Plan?

A tracking plan is a living document that defines:

1. What events to track (event catalog) 2. What properties each event has (schema) 3. Where events are triggered (implementation location) 4. Why we track them (business purpose)

Benefits

✓ Single source of truth for analytics
✓ Consistent naming across teams
✓ Clear implementation specs for engineers
✓ Data quality and governance
✓ Easier debugging and validation

Tracking Plan Structure

## Event: checkout_completed

**Category:** Commerce
**Description:** Triggered when user successfully completes payment

**When to trigger:**
- After payment processor confirms success
- Before order confirmation page loads

**Properties:**

| Property | Type | Required | Description | Example |
|----------|------|----------|-------------|---------|
| order_id | string | Yes | Unique order identifier | "ord_123456" |
| total_amount | number | Yes | Order total in dollars | 149.99 |
| currency | string | Yes | ISO currency code | "USD" |
| item_count | number | Yes | Number of items | 3 |
| payment_method | string | Yes | Payment type | "credit_card" |
| discount_amount | number | No | Discount applied | 30.00 |
| coupon_code | string | No | Coupon used | "SAVE20" |

**Implementation:**
- Platform: Web, iOS, Android
- Triggered by: `PaymentService.processPayment()` success callback
- Code location: `src/services/payment.ts`

**Related events:**
- checkout_started (precedes)
- order_shipped (follows)

**Metrics using this event:**
- Revenue metrics
- Conversion rate
- Average order value

---

Event Implementation

Client-Side vs. Server-Side Tracking

## Client-Side (Browser/App)
✓ Rich user context (device, browser, screen size)
✓ User interactions (clicks, scrolls, form fills)
✓ Real-time behavior tracking
✗ Can be blocked by ad blockers
✗ Unreliable for critical business events
✗ Privacy concerns (cookies, fingerprinting)

## Server-Side (Backend)
✓ Reliable (no ad blockers)
✓ Secure (no client manipulation)
✓ Critical business events (payments, conversions)
✓ Privacy-friendly (no cookies)
✗ Less user context
✗ Can't track client-side interactions directly

## Best Practice: Hybrid Approach
- Client-side: UI interactions, engagement, behavior
- Server-side: Transactions, conversions, critical actions
- Validate critical events on both sides

Implementation Examples

JavaScript (Web)

// Using Amplitude
amplitude.track('checkout_completed', {
  order_id: 'ord_123456',
  total_amount: 149.99,
  currency: 'USD',
  item_count: 3,
  payment_method: 'credit_card'
});

// Using Mixpanel
mixpanel.track('checkout_completed', {
  order_id: 'ord_123456',
  total_amount: 149.99,
  currency: 'USD',
  item_count: 3,
  payment_method: 'credit_card'
});

// Using Segment (works with many destinations)
analytics.track('checkout_completed', {
  order_id: 'ord_123456',
  total_amount: 149.99,
  currency: 'USD',
  item_count: 3,
  payment_method: 'credit_card'
});

// Custom implementation
function track(event, properties = {}) {
  const payload = {
    event,
    timestamp: new Date().toISOString(),
    user_id: getCurrentUserId(),
    session_id: getSessionId(),
    properties,
    context: {
      app_version: APP_VERSION,
      platform: 'web',
      url: window.location.href,
      referrer: document.referrer,
      user_agent: navigator.userAgent,
      screen_size: `${window.screen.width}x${window.screen.height}`
    }
  };

  // Send to analytics service
  fetch('/api/analytics/events', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload)
  });
}

Python (Server-Side)

from amplitude import Amplitude

# Initialize client
client = Amplitude(api_key='your_api_key')

# Track event
client.track({
    'event_type': 'checkout_completed',
    'user_id': 'usr_123',
    'event_properties': {
        'order_id': 'ord_123456',
        'total_amount': 149.99,
        'currency': 'USD',
        'item_count': 3,
        'payment_method': 'credit_card'
    },
    'time': int(time.time() * 1000)  # Unix timestamp in ms
})

# Using Mixpanel
from mixpanel import Mixpanel

mp = Mixpanel('your_token')
mp.track('usr_123', 'checkout_completed', {
    'order_id': 'ord_123456',
    'total_amount': 149.99,
    'currency': 'USD',
    'item_count': 3,
    'payment_method': 'credit_card'
})

React (Component Tracking)

import { useEffect } from 'react';
import { track } from '@/lib/analytics';

function CheckoutPage() {
  // Track page view
  useEffect(() => {
    track('checkout_page_viewed', {
      step: 'payment'
    });
  }, []);

  // Track button click
  const handleSubmit = async (formData) => {
    track('checkout_submitted', {
      payment_method: formData.paymentMethod
    });

    try {
      const result = await processPayment(formData);

      // Track success
      track('checkout_completed', {
        order_id: result.orderId,
        total_amount: result.total,
        currency: 'USD',
        item_count: cart.items.length,
        payment_method: formData.paymentMethod
      });

    } catch (error) {
      // Track failure
      track('checkout_failed', {
        error_message: error.message,
        payment_method: formData.paymentMethod
      });
    }
  };

  return <CheckoutForm onSubmit={handleSubmit} />;
}

---

Data Modeling

Event Schema Design

-- Events table (fact table)
CREATE TABLE events (
  id BIGSERIAL PRIMARY KEY,
  event_name VARCHAR(255) NOT NULL,
  user_id VARCHAR(255),
  anonymous_id VARCHAR(255),
  session_id VARCHAR(255),
  timestamp TIMESTAMPTZ NOT NULL,
  properties JSONB,
  context JSONB,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for common queries
CREATE INDEX idx_events_user_id ON events(user_id);
CREATE INDEX idx_events_timestamp ON events(timestamp DESC);
CREATE INDEX idx_events_name_timestamp ON events(event_name, timestamp DESC);
CREATE INDEX idx_events_properties ON events USING GIN(properties);

User Properties (Dimension Table)

-- Users table (dimension table)
CREATE TABLE users (
  user_id VARCHAR(255) PRIMARY KEY,
  email VARCHAR(255),
  created_at TIMESTAMPTZ,
  first_name VARCHAR(255),
  last_name VARCHAR(255),
  plan VARCHAR(50),
  mrr DECIMAL(10,2),
  ltv DECIMAL(10,2),

  -- Computed properties
  first_seen_at TIMESTAMPTZ,
  last_seen_at TIMESTAMPTZ,
  total_sessions INT,
  total_events INT,

  -- Segmentation
  cohort_month VARCHAR(7),  -- '2025-12'
  acquisition_channel VARCHAR(100),

  updated_at TIMESTAMPTZ DEFAULT NOW()
);

Common Queries

-- Daily active users
SELECT
  DATE(timestamp) as date,
  COUNT(DISTINCT user_id) as dau
FROM events
WHERE timestamp >= NOW() - INTERVAL '30 days'
GROUP BY DATE(timestamp)
ORDER BY date;

-- Event funnel
WITH funnel AS (
  SELECT
    user_id,
    MAX(CASE WHEN event_name = 'page_viewed' THEN 1 ELSE 0 END) as viewed,
    MAX(CASE WHEN event_name = 'signup_started' THEN 1 ELSE 0 END) as started,
    MAX(CASE WHEN event_name = 'signup_completed' THEN 1 ELSE 0 END) as completed
  FROM events
  WHERE timestamp >= '2025-12-01'
  GROUP BY user_id
)
SELECT
  SUM(viewed) as step1_viewed,
  SUM(started) as step2_started,
  SUM(completed) as step3_completed,
  ROUND(100.0 * SUM(started) / NULLIF(SUM(viewed), 0), 2) as view_to_start_rate,
  ROUND(100.0 * SUM(completed) / NULLIF(SUM(started), 0), 2) as start_to_complete_rate
FROM funnel;

-- Top events by user
SELECT
  event_name,
  COUNT(*) as event_count,
  COUNT(DISTINCT user_id) as unique_users,
  ROUND(COUNT(*) * 1.0 / COUNT(DISTINCT user_id), 2) as avg_per_user
FROM events
WHERE timestamp >= NOW() - INTERVAL '7 days'
GROUP BY event_name
ORDER BY event_count DESC
LIMIT 20;

---

Data Quality

Validation & Testing

// Event validation schema (using Zod)
import { z } from 'zod';

const CheckoutCompletedSchema = z.object({
  event: z.literal('checkout_completed'),
  timestamp: z.string().datetime(),
  user_id: z.string().min(1),
  properties: z.object({
    order_id: z.string().min(1),
    total_amount: z.number().positive(),
    currency: z.enum(['USD', 'EUR', 'GBP']),
    item_count: z.number().int().positive(),
    payment_method: z.enum(['credit_card', 'paypal', 'apple_pay'])
  })
});

// Use in tracking
function trackCheckoutCompleted(data) {
  try {
    const validated = CheckoutCompletedSchema.parse(data);
    sendToAnalytics(validated);
  } catch (error) {
    console.error('Invalid event data:', error);
    // Send to error monitoring
  }
}

Common Data Issues

## Issue 1: Duplicate Events
Problem: Same event sent multiple times
Cause: Retry logic, double-clicks, network issues
Solution: Use idempotency keys, debounce client events

## Issue 2: Missing Properties
Problem: Events sent without required properties
Cause: Code bugs, schema changes, null values
Solution: Schema validation, required fields, tests

## Issue 3: Inconsistent Naming
Problem: "user_signup" vs "userSignup" vs "User Signed Up"
Cause: Multiple developers, no standards
Solution: Tracking plan, linting, code review

## Issue 4: Wrong Data Types
Problem: "99.99" (string) instead of 99.99 (number)
Cause: Type coercion, API inconsistencies
Solution: TypeScript, validation, testing

## Issue 5: PII in Events
Problem: Email, password, credit card in properties
Cause: Developer error, overly broad logging
Solution: PII scanning, masking, training

---

Privacy & Compliance

GDPR & CCPA Compliance

## User Rights
1. Right to access: Provide all data for user_id
2. Right to deletion: Delete all events for user_id
3. Right to portability: Export user data
4. Right to opt-out: Stop tracking

## Implementation
- Hash or pseudonymize user_id
- Don't track PII in event properties
- Implement data deletion endpoints
- Provide opt-out mechanisms
- Document retention policies

PII Masking

// PII detection and masking
const maskEmail = (email) => {
  const [name, domain] = email.split('@');
  return `${name[0]}***@${domain}`;
};

const maskPhone = (phone) => `****${phone.slice(-4)}`;

const maskCreditCard = (card) => `****${card.slice(-4)}`;

// Sanitize event properties
function sanitizeProperties(properties) {
  const sanitized = { ...properties };

  // Remove known PII fields
  delete sanitized.email;
  delete sanitized.password;
  delete sanitized.credit_card;
  delete sanitized.ssn;

  // Mask if needed
  if (properties.email) {
    sanitized.email_domain = properties.email.split('@')[1];
  }

  if (properties.credit_card) {
    sanitized.card_last_4 = properties.credit_card.slice(-4);
    sanitized.card_type = detectCardType(properties.credit_card);
  }

  return sanitized;
}

---

2025 Trends

1. AI-Powered Analytics

- Automated anomaly detection
- Predictive churn modeling
- Natural language queries ("Show me users who churned last month")
- Auto-generated insights

2. Privacy-First Tracking

- Server-side tracking dominance
- Cookieless tracking
- Differential privacy
- On-device analytics (iOS, Android)

3. Real-Time Analytics

- Stream processing (Kafka, Flink)
- Sub-second dashboards
- Real-time personalization
- Instant A/B test results

4. Warehouse-Native Analytics

- Analytics tools run on your data warehouse
- No data copying (Snowflake, BigQuery, Databricks)
- SQL-based analysis
- Unified data model

---

Tools Comparison

---

Checklist

## Tracking Plan
- [ ] Event naming convention documented
- [ ] All events cataloged with descriptions
- [ ] Property schemas defined
- [ ] Implementation locations specified
- [ ] Privacy review completed

## Implementation
- [ ] Analytics library integrated
- [ ] Critical events tracked server-side
- [ ] Event validation in place
- [ ] Error handling for failed sends
- [ ] Testing coverage for tracking code

## Data Quality
- [ ] Schema validation active
- [ ] Duplicate detection implemented
- [ ] PII masking in place
- [ ] Data retention policy defined
- [ ] Regular data quality audits

## Privacy
- [ ] GDPR/CCPA compliance reviewed
- [ ] User opt-out mechanism
- [ ] Data deletion process
- [ ] Privacy policy updated
- [ ] Team trained on PII handling

A/B Testing & Experimentation

Overview

A/B testing (split testing) is the practice of comparing two or more variants to determine which performs better. This guide covers statistical best practices, common pitfalls, and 2025 trends in experimentation.

2025 Reality: 80% of A/B tests fail to produce a statistically significant winner, yet countless hours are spent acting on inconclusive results.

---

Fundamentals

What is A/B Testing?

A/B Testing = Controlled experiment comparing variants

Setup:
• Control (A): Current experience
• Variant (B): Proposed change
• Random assignment: Users → A or B
• Measure: Which performs better?

Example:
Control:  Blue "Sign Up" button → 5% conversion
Variant:  Green "Get Started" button → 6% conversion
Result:   Green button wins (+20% lift)

When to A/B Test

✓ Test before building expensive features
✓ Optimize conversion funnels
✓ Validate design changes
✓ Test pricing changes
✓ Optimize email/push campaigns
✓ Test copy, CTAs, layouts

✗ Don't test without sufficient traffic
✗ Don't test if you can't implement winner
✗ Don't test if outcome doesn't matter
✗ Don't test radical product changes (use beta)

---

Statistical Significance

P-Value Explained

The p-value is the probability that the observed difference occurred by chance.

p-value < 0.05 (5%)
= Less than 5% chance result is random
= 95% confidence in the result
= Statistically significant

Standard thresholds:
• p ≤ 0.01: Highly significant (99% confidence)
• p ≤ 0.05: Significant (95% confidence)
• p > 0.05: Not significant (insufficient evidence)

Type I and Type II Errors

## Type I Error (False Positive)
Definition: Rejecting null hypothesis when it's true
Example: Concluding B is better when it's not
Controlled by: Significance level (α = 0.05)

Impact:
• Ship a "winning" variant that doesn't actually work
• Waste engineering time
• Potentially harm metrics

## Type II Error (False Negative)
Definition: Failing to reject null hypothesis when it's false
Example: Missing a real improvement
Controlled by: Statistical power (1 - β)

Impact:
• Miss opportunity to improve product
• Leave better variant unreleased
• Slow growth

## Optimal Balance
• α (significance) = 0.05 (5% false positive rate)
• Power = 0.80 (80% chance detecting real effect)
• This gives β = 0.20 (20% false negative rate)

Statistical Power

Power is the probability of detecting a real effect if one exists.

Power = 1 - β (Type II error rate)

Standard: 80% power
= 80% chance of detecting real 5% lift
= Need sufficient sample size

Factors affecting power:
• Sample size (more = higher power)
• Effect size (larger effect = easier to detect)
• Significance level (lower α = lower power)
• Baseline conversion rate

Sample Size Calculation

Minimum sample size depends on:
• Baseline conversion rate
• Minimum detectable effect (MDE)
• Significance level (α)
• Statistical power (1 - β)

Formula (simplified):
n = 16 × σ² / (MDE²)

Where σ² = p(1-p) for binomial outcomes

Example:
Baseline: 10% conversion
MDE: 2% (absolute), 20% (relative)
α = 0.05, Power = 0.80

Required: ~2,000 users per variant

Online Sample Size Calculators

• Optimizely: https://www.optimizely.com/sample-size-calculator/
• VWO: https://vwo.com/tools/ab-test-significance-calculator/
• Evan's Awesome A/B Tools: https://www.evanmiller.org/ab-testing/

---

Running Experiments

Experiment Design

## 1. Hypothesis
Format: "If [change], then [impact], because [reasoning]"

Example:
"If we change the CTA from 'Sign Up' to 'Get Started',
then conversion will increase by 10%,
because 'Get Started' is more action-oriented and less committal"

## 2. Primary Metric
The ONE metric you're trying to move

Examples:
• Conversion rate (sign-ups / visitors)
• Click-through rate (CTR)
• Revenue per user
• Retention rate

## 3. Secondary Metrics
Supporting metrics to watch for side effects

Examples:
• Time on page
• Bounce rate
• Downstream conversions
• Revenue impact

## 4. Guardrail Metrics
Metrics that should NOT decrease

Examples:
• Overall revenue
• User satisfaction (NPS)
• Load time
• Error rate

Test Setup

// Example: Feature flag with A/B test
import { getVariant } from '@/lib/experiments';

function SignUpButton() {
  const variant = getVariant('signup-button-test', {
    control: 0.5,    // 50% traffic
    variant: 0.5     // 50% traffic
  });

  const buttonText = variant === 'variant'
    ? 'Get Started'    // Variant B
    : 'Sign Up';       // Control A

  const buttonColor = variant === 'variant'
    ? 'green'
    : 'blue';

  const handleClick = () => {
    // Track conversion
    track('signup_clicked', {
      variant: variant,
      button_text: buttonText,
      button_color: buttonColor
    });

    // Proceed with sign-up
    navigate('/signup');
  };

  return (
    <button
      onClick={handleClick}
      style={{ backgroundColor: buttonColor }}
    >
      {buttonText}
    </button>
  );
}

Random Assignment

## User-Level Randomization (Recommended)
• Assign each user consistently to A or B
• Same user always sees same variant
• Prevents confusion from variant switching

## Session-Level Randomization
• Each session gets random assignment
• Same user may see A then B
• Good for content testing, bad for UX changes

## Implementation (Hash-Based)
function getVariant(userId, experimentId) {
  const hash = md5(userId + experimentId);
  const bucket = parseInt(hash.substring(0, 8), 16) % 100;

  if (bucket < 50) return 'control';
  return 'variant';
}

Test Duration

## Minimum Duration Guidelines
• Run for at least 1-2 weeks
• Capture weekly patterns (weekday vs weekend)
• Account for seasonality
• Don't stop early just because you see significance

## When to Stop
✓ Reached required sample size
✓ Ran for minimum duration
✓ Result is statistically significant
✓ OR decided test is inconclusive

## Common Mistakes
❌ Peeking and stopping early
❌ Running indefinitely waiting for significance
❌ Not accounting for weekly cycles

---

Frequentist vs. Bayesian

Frequentist Approach (Traditional)

## Philosophy
• Probability = long-run frequency
• Null hypothesis testing (p-values)
• Binary outcome: significant or not

## Process
1. Set α = 0.05 before test
2. Collect data
3. Calculate p-value
4. If p < 0.05 → reject null, B wins
5. If p ≥ 0.05 → fail to reject, inconclusive

## Pros
• Industry standard
• Well-understood
• Objective (no priors)

## Cons
• Binary outcome (significant or not)
• Misinterpreted p-values
• "Peeking" problem (checking early invalidates test)

Bayesian Approach (Modern)

## Philosophy
• Probability = degree of belief
• Updates beliefs with evidence
• Continuous probability estimates

## Process
1. Start with prior belief (e.g., 50/50)
2. Collect data
3. Update posterior probability
4. "B is 85% likely to be better than A"

## Pros
• Intuitive probabilities
• Can peek anytime (no invalidation)
• Incorporates prior knowledge
• Handles small samples better

## Cons
• Requires setting priors
• More complex to implement
• Less familiar to stakeholders

## 2025 Trend
Experimentation leaders are 270% more likely to grow when using
Bayesian or CUPED methods vs. basic frequentist A/B tests.

---

Common Pitfalls

1. Peeking Problem

Problem: Checking results repeatedly and stopping when significant

Why it's bad:
• Increases false positive rate (Type I error)
• Standard α = 0.05 no longer valid
• Random fluctuations look like significance early on

Solution:
• Wait until reaching required sample size
• Use sequential testing (if you must peek)
• Use Bayesian methods (peeking is valid)

2. Multiple Comparisons

Problem: Testing many variants or metrics without correction

Example:
• Test 10 button colors
• Even if none work, ~40% chance of false positive

Why:
• With α = 0.05, each test has 5% false positive rate
• More tests = higher overall false positive rate

Solution:
• Bonferroni correction: α_adjusted = α / n
• Focus on primary metric
• Pre-register hypotheses

3. Sample Ratio Mismatch (SRM)

Problem: Unequal split when expecting 50/50

Example:
• Expected: 50% control, 50% variant
• Observed: 48% control, 52% variant
• Chi-square test p < 0.05 → SRM detected!

Causes:
• Bug in randomization
• Variant has different load time (users leave)
• Bot traffic hitting only one variant
• Bucketing logic error

Solution:
• Check split before analyzing results
• Investigate and fix randomization
• Don't trust results if SRM exists

4. Novelty Effect

Problem: Users react to change, not improvement

Example:
• Change button from blue to red
• Week 1: Conversions up 10% (users notice change)
• Week 4: Conversions return to baseline

Solution:
• Run tests for 2-4 weeks
• Check if effect persists
• Segment new vs. returning users

5. Ignoring Statistical Power

Problem: Not enough traffic, test never reaches significance

Example:
• Need 10,000 users per variant
• Only have 1,000 users/month
• Test runs for years...

Solution:
• Calculate required sample size upfront
• Don't test if traffic insufficient
• Consider larger changes (bigger effect size)
• Combine multiple small changes

---

Advanced Techniques

CUPED (Controlled-Experiment Using Pre-Experiment Data)

## What is CUPED?
Variance reduction technique using pre-experiment data

## How it Works
1. Measure metric before experiment (pre-period)
2. Use correlation between pre/post to reduce variance
3. Detect smaller effects with same sample size

## Benefit
• 2-3× variance reduction
• Faster experiment results
• Detect smaller lifts

## When to Use
• High correlation between pre/post metrics
• Want to detect small effects (<5%)
• Limited traffic

## 2025 Adoption
Used by Netflix, Microsoft, Booking.com
Becoming standard at experimentation-mature companies

Multi-Armed Bandits

## What is MAB?
Adaptive algorithm that shifts traffic to better variants

## How it Works
• Start: Equal traffic to all variants
• During: More traffic to better-performing variants
• End: Most traffic to winner

## Pros
• Minimize opportunity cost
• Good for content testing (headlines, images)
• Explores and exploits simultaneously

## Cons
• Not pure A/B test (less scientific)
• Can be slow to converge
• Doesn't give significance test

## When to Use
• High traffic, fast feedback
• Content optimization
• OK with "good enough" not "statistically proven"

Holdout Groups

## Purpose
Validate that shipped experiments actually worked

## Setup
• After A/B test, ship winning variant
• Keep 5-10% of users in control (holdout)
• Monitor long-term impact

## Example
• A/B test shows 5% conversion lift
• Ship to 90% of users
• Keep 10% in control
• After 6 months: Is lift still there?

## Why Important
• Novelty effects wear off
• Interaction effects with other changes
• Validates experiment program ROI

---

Funnel Experiments

Multi-Step Funnels

## Challenge
Testing change in Step 2 affects Step 3, 4, 5...

Example Funnel:
Landing Page → Sign-Up → Onboarding → Activation → Payment

Test: Change onboarding (Step 3)

Metrics to measure:
• Step 3 completion (direct impact)
• Step 4 activation (downstream)
• Step 5 payment (ultimate goal)
• Overall funnel conversion

## Analysis
• Primary: Overall conversion (Landing → Payment)
• Secondary: Each step individually
• Check for trade-offs (better Step 3, worse Step 5?)

---

Experiment Analysis

Statistical Test Selection

## Binary Outcomes (Conversion, Click)
Use: Two-proportion z-test

Example:
Control: 1000 users, 50 conversions (5.0%)
Variant: 1000 users, 60 conversions (6.0%)

H0: p_variant = p_control
HA: p_variant ≠ p_control

z-score = (6.0% - 5.0%) / SE
p-value = 0.032

Result: p < 0.05 → Significant!

## Continuous Outcomes (Revenue, Time)
Use: Two-sample t-test

Example:
Control: avg revenue = $50, σ = $20, n = 500
Variant: avg revenue = $55, σ = $22, n = 500

t-statistic = (55 - 50) / SE
p-value = 0.003

Result: p < 0.05 → Significant!

Interpreting Results

## Scenario 1: Clear Winner
• p < 0.05 (statistically significant)
• Practical significance (lift > 5%)
• No SRM issues
• Ran for sufficient duration

Action: Ship the winner

## Scenario 2: Inconclusive
• p > 0.05 (not significant)
• Small sample, short duration

Action: Run longer or accept inconclusive

## Scenario 3: Significant but Small
• p < 0.05 (significant)
• But lift is tiny (0.1%)

Action: Evaluate if worth engineering effort

## Scenario 4: Mixed Results
• Primary metric: No change
• Secondary metric: Big improvement

Action: Investigate, might have misidentified primary metric

## Scenario 5: Negative Result
• p < 0.05 (significant)
• But variant performed WORSE

Action: Don't ship, investigate why hypothesis failed

Effect Size

## Absolute vs. Relative Lift

Absolute Lift:
= Variant rate - Control rate
= 6% - 5% = 1 percentage point

Relative Lift:
= (Variant - Control) / Control × 100%
= (6% - 5%) / 5% × 100% = 20%

Report both:
"Variant increased conversion from 5% to 6%
(absolute lift: +1pp, relative lift: +20%)"

## Practical Significance
Statistical significance ≠ Practical significance

Example:
• Statistically significant: p = 0.001
• But only +0.1% conversion increase
• Engineering effort: 2 weeks
• Worth it? Probably not.

Consider:
• Implementation cost
• Maintenance burden
• Opportunity cost (what else could you build?)

---

Experimentation Culture

Organizational Maturity

## Level 1: Ad-Hoc Testing
• No formal process
• Tests run occasionally
• Inconsistent methodology

## Level 2: Systematic Testing
• Regular A/B tests
• Documentation
• Consistent statistical methods

## Level 3: Experimentation Platform
• Centralized tooling
• Self-service for teams
• Automated analysis

## Level 4: Advanced Methods
• CUPED, Bayesian methods
• Holdout groups
• Causal inference

## 2025 Survey Finding
Teams with shared metrics and reporting frameworks were
significantly more likely to launch impactful experiments faster.

Best Practices

1. Document everything
   • Hypothesis
   • Expected impact
   • Sample size calculation
   • Results and learnings

2. Learn from failures
   • 80% of tests fail
   • Failed tests teach what DOESN'T work
   • Share learnings across teams

3. Align on thresholds
   • Marketing: 90% confidence OK?
   • Product: 95% required?
   • Standardize to avoid friction

4. Velocity matters
   • More experiments = more learning
   • Don't overthink small tests
   • Fail fast, iterate quickly

5. Build institutional knowledge
   • Track all experiments
   • Analyze patterns (what works?)
   • Create playbooks

---

Tools

Experimentation Platforms

Statistical Analysis

# Python: Two-proportion z-test
from scipy import stats

# Control: 50/1000 = 5%
# Variant: 60/1000 = 6%

control_conversions = 50
control_total = 1000
variant_conversions = 60
variant_total = 1000

# Two-proportion z-test
z_stat, p_value = stats.proportions_ztest(
    [variant_conversions, control_conversions],
    [variant_total, control_total]
)

print(f"z-statistic: {z_stat:.3f}")
print(f"p-value: {p_value:.3f}")

if p_value < 0.05:
    print("Statistically significant!")
else:
    print("Not significant")

# R: Two-proportion test
control <- c(50, 950)  # 50 conversions, 950 non-conversions
variant <- c(60, 940)  # 60 conversions, 940 non-conversions

result <- prop.test(c(60, 50), c(1000, 1000))
print(result)

---

2025 Trends

1. Bayesian Methods Adoption

• Easier to interpret ("85% chance B is better")
• Can peek without invalidating results
• Better for small samples
• Tools: Statsig, GrowthBook, VWO

2. CUPED/Variance Reduction

• 2-3× faster experiments
• Standard at Netflix, Booking.com, Microsoft
• Requires pre-period data
• Advanced technique becoming mainstream

3. Causal Inference

• Beyond correlation → causation
• Synthetic controls
• Difference-in-differences
• Academic methods entering industry

4. AI-Powered Experimentation

• Auto-generate test ideas
• Predict experiment outcomes
• Anomaly detection
• Automated interpretation

---

Checklist

## Before Experiment
- [ ] Hypothesis clearly stated
- [ ] Primary metric defined
- [ ] Secondary/guardrail metrics identified
- [ ] Sample size calculated
- [ ] Minimum duration determined
- [ ] Random assignment validated
- [ ] Implementation QA'd

## During Experiment
- [ ] Monitor for SRM (sample ratio mismatch)
- [ ] Check for technical issues
- [ ] Don't peek and stop early
- [ ] Track experiment in central log

## After Experiment
- [ ] Wait for required sample size
- [ ] Check statistical significance (p-value)
- [ ] Evaluate practical significance (effect size)
- [ ] Review secondary metrics
- [ ] Document results and learnings
- [ ] Make ship/no-ship decision
- [ ] Share results with team

product-analytics Extended Reference

This file preserves detailed material moved out of SKILL.md for progressive disclosure. Load it only when the current task needs the specific examples, commands, templates, or checklists below.

Moved content starts at: ## Key Metrics & Formulas.

Key Metrics & Formulas

Engagement Metrics

Daily Active Users (DAU)
= Unique users performing key action per day

Monthly Active Users (MAU)
= Unique users performing key action per month

Stickiness = DAU / MAU × 100%
• 20%+ = Good (users engage 6+ days/month)
• 10-20% = Average
• <10% = Low engagement

Session Duration
= Average time between session start and end

Session Frequency
= Average sessions per user per time period

Retention Metrics

Retention Rate (Classic)
= Users active in Week N / Users in original cohort

Retention Rate (Bracket)
= Users active in Week N / Users active in Week 0

Churn Rate
= (Users at start - Users at end) / Users at start

Quick Ratio (Growth Health)
= (New MRR + Expansion MRR) / (Churned MRR + Contraction MRR)
• >4 = Excellent growth
• 2-4 = Good
• <1 = Shrinking

Conversion Metrics

Conversion Rate
= (Conversions / Total visitors) × 100%

Funnel Conversion
= (Users completing final step / Users entering funnel) × 100%

Time to Convert
= Median time from first touch to conversion

Revenue Metrics

Monthly Recurring Revenue (MRR)
= Sum of all monthly subscription values

Annual Recurring Revenue (ARR)
= MRR × 12

Average Revenue Per User (ARPU)
= Total revenue / Number of users

Customer Lifetime Value (LTV)
= ARPU × Average customer lifetime (months)
OR
= ARPU × Gross Margin % / Monthly Churn Rate

Customer Acquisition Cost (CAC)
= Total sales & marketing spend / New customers acquired

LTV:CAC Ratio
= LTV / CAC
• >3:1 = Healthy
• 1:1 = Unsustainable

Payback Period
= CAC / (ARPU × Gross Margin %)
• <12 months = Good
• 12-18 months = Acceptable
• >18 months = Concerning

---

Event Tracking Best Practices

Event Naming Convention

Object + Action pattern (recommended)

✓ user_signed_up
✓ project_created
✓ file_uploaded
✓ payment_completed

✗ signup (unclear)
✗ new_project (inconsistent)
✗ Upload File (inconsistent case)

Event Properties Structure

// Standard event structure
{
  event: "checkout_completed",        // Event name
  timestamp: "2025-12-16T10:30:00Z",  // When
  user_id: "usr_123",                 // Who
  session_id: "ses_abc",              // Session context
  properties: {                       // Event-specific data
    order_id: "ord_789",
    total_amount: 99.99,
    currency: "USD",
    item_count: 3,
    payment_method: "credit_card",
    coupon_used: true,
    discount_amount: 10.00
  },
  context: {                          // Global context
    app_version: "2.4.1",
    platform: "web",
    user_agent: "...",
    ip: "192.168.1.1",
    locale: "en-US"
  }
}

Critical Events to Track

## User Lifecycle
- user_signed_up
- user_activated (first key action)
- user_onboarded (completed setup)
- user_upgraded (plan change)
- user_churned (canceled/inactive)

## Feature Usage
- feature_viewed
- feature_used
- feature_completed

## Commerce
- product_viewed
- product_added_to_cart
- checkout_started
- payment_completed
- order_fulfilled

## Engagement
- session_started
- session_ended
- page_viewed
- search_performed
- content_shared

## Errors
- error_occurred
- payment_failed
- api_error

Privacy & Compliance

// ✓ GOOD: No PII in events
track('user_signed_up', {
  user_id: hashUserId('user@example.com'),  // Hashed
  plan: 'pro',
  source: 'organic'
});

// ✗ BAD: Contains PII
track('user_signed_up', {
  email: 'user@example.com',  // PII!
  password: '...',            // Never log!
  credit_card: '...'          // Never log!
});

// Masking strategies
const maskEmail = (email) => {
  const [name, domain] = email.split('@');
  return `${name[0]}***@${domain}`;
};

const maskCard = (card) => `****${card.slice(-4)}`;

---

See Also

• reference/event-tracking.md — Event tracking and data modeling guide
• reference/metrics-framework.md — North Star, AARRR, key metrics deep dive
• reference/experimentation.md — A/B testing and statistical best practices
• reference/retention.md — Cohort analysis and retention strategies
• templates/tracking-plan.md — Event tracking plan template

Metrics Framework: North Star, AARRR & Key Metrics

Overview

This guide covers the complete metrics framework for product analytics, including how to select, define, and track the right metrics for your product.

---

North Star Metric Framework

What is a North Star Metric?

The North Star Metric (NSM) is the single metric that best captures the core value your product delivers to customers. When your NSM grows sustainably, your business grows.

Created by Amplitude's Sean Ellis and popularized in the growth hacking community, the NSM provides:

• Single point of alignment across teams
• Clear measure of product value delivery
• Leading indicator of business success

Characteristics of a Good NSM

✓ Expresses value delivered to customers
✓ Represents vision and strategy
✓ Predicts long-term success
✓ Measurable and actionable
✓ Not a vanity metric
✓ Hard to game or manipulate

NSM vs. Other Metrics

## North Star Metric
• Spotify: Time Spent Listening
  → Captures engagement with core value (music)

## NOT North Star (Supporting Metrics)
• Sign-ups → Acquisition metric
• Revenue → Lagging indicator
• App downloads → Vanity metric
• Page views → Activity, not value

The NSM bridges user value and business outcomes.

Real-World Examples

How to Choose Your NSM

## Step 1: Define Core Value Proposition
What is the "aha!" moment when users get value?

Examples:
• Slack: "Wow, my team is communicating so much faster!"
• Dropbox: "My files are accessible everywhere!"
• Airbnb: "I found the perfect place to stay!"

## Step 2: Find the Metric That Captures It
What behavior represents this value delivery?

Questions:
• When does the user get value?
• What action demonstrates they got value?
• Can we measure it consistently?

## Step 3: Validate Business Correlation
Does NSM growth → revenue growth?

Test:
• Plot NSM vs. revenue over time
• Look for correlation
• Validate causation (not just correlation)

## Step 4: Define Input Metrics
What drives the NSM?

Break down into 3-5 inputs:
• User actions that lead to NSM
• Metrics teams can directly influence
• Measurable and actionable

NSM Tree Structure

           North Star Metric
                  ↓
         Time Spent Listening
                  ↓
    ┌─────────────┴─────────────┬─────────────┐
    ↓                           ↓             ↓
Daily Active    Playlists    Songs Added   Social
  Users          Created      to Library     Shares
    ↓              ↓              ↓            ↓
┌───┴───┐      ┌───┴───┐    ┌────┴────┐   ┌──┴──┐
New    Return  Create  Follow Discovery Rec  Share Collab
Users  Users   Personal List  Mode     Algos Posts Playlists

Each level = actionable for different teams
Bottom level = direct product/eng impact

Common Mistakes

❌ Choosing Revenue as NSM
• Revenue is a lagging indicator
• Doesn't capture user value directly
• Use as validation, not NSM

❌ Picking Multiple NSMs
• Defeats the purpose of "North Star"
• Creates conflicting priorities
• Choose ONE, support with inputs

❌ Selecting a Vanity Metric
• Sign-ups look good but don't show value
• Downloads don't equal engagement
• Focus on value delivery, not top-of-funnel

❌ Gaming the Metric
• If NSM = page views, add unnecessary clicks
• If NSM = sessions, force re-logins
• Choose metrics that align incentives

❌ Ignoring Input Metrics
• NSM without inputs = just a dashboard
• Teams need actionable metrics
• Break down into influenceable components

---

AARRR Framework Deep Dive

Framework Overview

AARRR (Pirate Metrics) was created by Dave McClure to track the customer lifecycle:

ACQUISITION → ACTIVATION → RETENTION → REFERRAL → REVENUE

Each stage has specific metrics, goals, and optimization strategies.

---

1. Acquisition

Definition: How users discover and arrive at your product.

Key Metrics

• Traffic sources (organic, paid, referral, social, direct)
• Visitors per channel
• Cost Per Acquisition (CPA) per channel
• Sign-up conversion rate per channel
• Channel ROI = Revenue / Cost

Acquisition Channels

## Organic
• SEO (Google, Bing)
• Content marketing
• Brand search

## Paid
• Google Ads (search, display)
• Facebook/Instagram Ads
• LinkedIn Ads
• Sponsored content

## Referral
• Word of mouth
• Referral programs
• Affiliate marketing

## Social
• Organic social posts
• Influencer marketing
• Community engagement

## Direct
• Direct traffic (bookmarks, typed URL)
• Email marketing
• Retargeting

Optimization Strategies

1. Measure channel quality, not just volume
   → Users from Channel A may have 2x retention vs. Channel B
   → Optimize for LTV, not just CPA

2. Track full funnel per channel
   → Organic: Visit → Sign-up → Activation → Retention
   → Paid: Click → Land → Sign-up → Activation → Retention

3. Calculate channel LTV:CAC
   → LTV:CAC > 3:1 = Good channel
   → < 1:1 = Unsustainable

4. Attribution modeling
   → Last-click: Credit final touchpoint
   → First-click: Credit initial discovery
   → Linear: Equal credit all touchpoints
   → Data-driven: ML-based attribution

Acquisition Formulas

Cost Per Acquisition (CPA)
= Total Marketing Spend / Number of Customers Acquired

Channel Conversion Rate
= (Sign-ups from Channel / Visitors from Channel) × 100%

Channel ROI
= (Revenue from Channel - Cost) / Cost × 100%

Example:
Spent $10,000 on Google Ads
Acquired 200 customers
Average LTV = $300

CPA = $10,000 / 200 = $50
LTV:CAC = $300 / $50 = 6:1 (Excellent!)

---

2. Activation

Definition: When users experience the core value of your product ("aha!" moment).

Key Metrics

• Activation rate (% users reaching "aha!" moment)
• Time to activation (median time to first value)
• Setup completion rate
• Feature adoption rate
• Tutorial completion rate

Defining Activation

Activation is NOT:
❌ Sign-up completion
❌ Email verification
❌ Profile creation

Activation IS:
✓ First meaningful action
✓ Core value delivered
✓ User gets "aha!" moment

Examples:
• Slack: Team sends 2,000 messages
• Twitter: Follow 30 users
• Dropbox: Upload first file and access from another device
• LinkedIn: Connect with 5 people
• Asana: Create first project and assign tasks
• Figma: Invite teammate and collaborate on design

Activation Funnel

Step 1: Sign-up completed
        ↓ 90% proceed
Step 2: Email verified
        ↓ 70% proceed
Step 3: Profile completed
        ↓ 60% proceed
Step 4: First action (ACTIVATION)
        ↓
Overall activation rate: 90% × 70% × 60% = 38%

Optimization focus: Biggest drop-off = Step 2 → Step 3

Optimization Strategies

1. Shorten time to value
   → Remove unnecessary onboarding steps
   → Pre-populate data where possible
   → Progressive disclosure (show features as needed)

2. Personalize onboarding
   → Role-based flows (developer, marketer, designer)
   → Use case-based setup
   → Show relevant examples

3. Use in-app guidance
   → Interactive tutorials
   → Tooltips and walkthroughs
   → Progress indicators

4. Celebrate activation
   → Confetti animation
   → Achievement unlocked
   → Email: "You did it!"

5. Measure activation cohorts
   → Users activated in < 1 hour: 80% retain
   → Users activated in 1-24 hours: 50% retain
   → Users activated in 24+ hours: 20% retain
   → Focus on fast activation

Activation Formulas

Activation Rate
= (Users who completed activation action / Total sign-ups) × 100%

Time to Activation
= Median time between sign-up and activation event

Activation Funnel Conversion
= (Users completing final step / Users starting funnel) × 100%

Example:
1,000 sign-ups this month
600 reached activation (first project created)
Median time: 3.2 hours

Activation rate = 600 / 1,000 = 60%
Time to activation = 3.2 hours

---

3. Retention

Definition: When users come back and continue using the product.

Key Metrics

• Day 1/7/30 retention rate
• Weekly/Monthly Active Users (WAU/MAU)
• Churn rate (% users who stop using)
• Cohort retention curves
• Feature stickiness (DAU/MAU ratio)
• Usage frequency (sessions per user)

Retention Curves

## Good Retention Curve
100% ┐
     │╲
 80% │ ╲___________  ← Flattens (users retain)
     │
 60% │
     └──────────────────────
      D1  D7  D30  D60  D90

## Bad Retention Curve
100% ┐
     │╲
 80% │ ╲
     │  ╲
 20% │   ╲___________  ← Keeps dropping
     └──────────────────────
      D1  D7  D30  D60  D90

Goal: Curve flattens = found core engaged users

Retention Benchmarks by Industry

## Consumer Apps (B2C)
Day 1:  60-80%
Day 7:  30-50%
Day 30: 15-30%

## SaaS (B2B)
Day 1:  70-90%
Day 7:  50-70%
Day 30: 40-60%

## Social Networks
Day 1:  50-70%
Day 7:  40-60%
Day 30: 30-50%

## Gaming
Day 1:  40-60%
Day 7:  20-40%
Day 30: 10-20%

Note: Varies widely by product type and market

Optimization Strategies

1. Identify retention drivers
   → Which features do retained users use?
   → Correlation analysis: feature usage vs. retention
   → Encourage high-retention behaviors

2. Re-engagement campaigns
   → Email: "You haven't logged in for 7 days"
   → Push notifications: "Your team posted an update"
   → In-app messages: "New features available"

3. Habit formation
   → Daily streaks (Duolingo)
   → Email digests (daily/weekly)
   → Reminder notifications

4. Prevent churn
   → Identify at-risk users (low engagement)
   → Proactive outreach
   → Offer help, incentives

5. Improve product stickiness
   → Add integrations (harder to leave)
   → Data network effects (more data = more value)
   → Social features (friends keep you engaged)

Retention Formulas

Day N Retention
= (Users active on Day N / Users in cohort) × 100%

Churn Rate
= (Users at start - Users at end) / Users at start × 100%

Stickiness (DAU/MAU Ratio)
= (DAU / MAU) × 100%
• 20%+ = Very sticky (users engage 6+ days/month)
• 10-20% = Moderate
• <10% = Low engagement

Example:
Cohort: 1,000 users signed up Jan 1
Day 7: 350 returned
Day 30: 200 returned

Day 7 retention = 350 / 1,000 = 35%
Day 30 retention = 200 / 1,000 = 20%

---

4. Referral

Definition: When users recommend your product to others.

Key Metrics

• Viral coefficient (K-factor)
• Referral rate (% users who refer)
• Invites sent per user
• Invite acceptance rate
• Net Promoter Score (NPS)
• Viral cycle time

Viral Coefficient (K-Factor)

K = (% users inviting) × (avg invites per user) × (invite conversion rate)

K > 1: Viral growth (each user brings >1 new user)
K = 1: Replacement growth
K < 1: Need paid acquisition

Example:
20% of users send invites
Average 5 invites per user
30% invitation acceptance

K = 0.20 × 5 × 0.30 = 0.30

Interpretation: Each user brings 0.30 new users
→ Need paid acquisition to grow

Viral Cycle Time

Viral Cycle Time = Time from sign-up to sending first invite

Shorter = Faster growth

Example:
K = 0.5, cycle time = 7 days
100 users → +50 in week 1 → +25 in week 2 → +13 in week 3

K = 0.5, cycle time = 1 day
100 users → +50 in day 1 → +25 in day 2 → +13 in day 3

Same K, but daily cycle = 7× faster growth

Referral Mechanisms

## Inherent Virality (Built into product)
• Zoom: Invite to meeting → receiver must install
• Google Docs: Share document → receiver must sign up
• WhatsApp: Message friend → network effect

## Incentivized Referral (Rewards)
• Dropbox: Refer friend → both get extra storage
• Airbnb: Refer host/guest → both get credits
• Uber: Give $10 off → get $10 off

## Social Sharing (Content distribution)
• Canva: Share design → branded footer
• Loom: Share video → "Record your own with Loom"
• Medium: Article → "Read more on Medium"

## Word of Mouth (Organic)
• Slack: Teams talk about productivity
• Notion: Users share templates
• Figma: Designers showcase work

Optimization Strategies

1. Make sharing valuable for sharer
   → Collaboration requires inviting others
   → Sharing creates accountability (fitness app)
   → Social validation (post achievements)

2. Make sharing easy
   → One-click invite
   → Import contacts
   → Shareable links

3. Incentivize appropriately
   → Two-sided incentives (both get reward)
   → Reward upon conversion, not just invite
   → Avoid spam (rate limits)

4. Measure and optimize
   → A/B test invite copy
   → Test reward amounts
   → Optimize timing (when to prompt invite)

---

5. Revenue

Definition: When users generate business value (paying customers).

Key Metrics

• Monthly Recurring Revenue (MRR)
• Annual Recurring Revenue (ARR)
• Average Revenue Per User (ARPU)
• Customer Lifetime Value (LTV)
• Customer Acquisition Cost (CAC)
• LTV:CAC ratio
• Conversion to paid rate
• Revenue churn
• Expansion revenue

Revenue Formulas

Monthly Recurring Revenue (MRR)
= Sum of all monthly subscription values

Annual Recurring Revenue (ARR)
= MRR × 12

Average Revenue Per User (ARPU)
= Total revenue / Number of paying users

Customer Lifetime Value (LTV)
= ARPU × Average lifetime (months)

OR

LTV = ARPU × Gross Margin % / Monthly Churn Rate

Customer Acquisition Cost (CAC)
= Total sales & marketing spend / New customers

LTV:CAC Ratio
= LTV / CAC
• >3:1 = Healthy (for every $1 spent, get $3+ back)
• 1:1 = Unsustainable (not profitable)

Payback Period
= CAC / (ARPU × Gross Margin %)
• <12 months = Good
• 12-18 months = Acceptable
• >18 months = Risky

Revenue Optimization

## Increase Conversion Rate
• Optimize pricing page
• Add social proof (testimonials, logos)
• Offer free trial (reduce friction)
• Transparent pricing (no "Contact sales")

## Increase ARPU
• Upsell to higher tiers
• Cross-sell additional products
• Usage-based pricing (more usage = more revenue)
• Annual vs. monthly (discount for commitment)

## Reduce Churn
• Improve product value
• Customer success programs
• Proactive support
• Prevent cancellations (retention offers)

## Expansion Revenue
• Seat expansion (more users)
• Feature upsells
• Add-ons and integrations
• Enterprise contracts

SaaS Metrics

## Monthly Metrics
New MRR:        +$10,000  (new customers)
Expansion MRR:  +$3,000   (upsells/upgrades)
Churned MRR:    -$2,000   (cancellations)
Contraction MRR: -$500    (downgrades)

Net New MRR = +$10,000 + $3,000 - $2,000 - $500 = +$10,500

## Growth Metrics
MRR Growth Rate = (Net New MRR / Starting MRR) × 100%
Quick Ratio = (New + Expansion) / (Churned + Contraction)
• >4 = Excellent
• 2-4 = Good
• <1 = Shrinking

---

Additional Key Metrics

Engagement Metrics

Session Duration
= Average time between session start and end

Pages Per Session
= Average page views per visit

Bounce Rate
= (Single-page sessions / Total sessions) × 100%

Feature Adoption
= (Users using feature / Total active users) × 100%

Conversion Metrics

Conversion Rate
= (Conversions / Visitors) × 100%

Qualified Lead Rate
= (Qualified leads / Total leads) × 100%

Sales Cycle Length
= Average days from first touch to closed deal

---

Metric Selection Guide

Choosing the Right Metrics

## For Startups (Pre-PMF)
Focus: Retention + Activation
• Are users coming back?
• Are they getting value?
→ Ignore revenue/growth until retention stabilizes

## For Growth Stage
Focus: AARRR + NSM
• North Star Metric
• AARRR funnel optimization
• Channel efficiency (LTV:CAC)

## For Enterprise
Focus: Expansion + Net Revenue Retention
• Account expansion (seats, features)
• NRR (Net Revenue Retention) > 100%
• Customer health scores

Metrics Anti-Patterns

❌ Vanity Metrics
• Total users (includes churned)
• Total downloads (not active users)
• Page views (no business impact)

✓ Actionable Metrics
• Active users (engaged recently)
• Retained users (came back)
• Converted users (paid)

❌ Too Many Metrics
• Tracking 50+ KPIs
• No clear priorities
• Analysis paralysis

✓ Focused Metrics
• 1 North Star
• 3-5 supporting inputs
• AARRR framework

❌ Lagging Indicators Only
• Revenue (happens after value delivery)
• Churn (user already lost)

✓ Mix of Leading + Lagging
• Leading: Feature usage, engagement
• Lagging: Revenue, churn

---

Checklist

## North Star Metric
- [ ] NSM defined and documented
- [ ] Validated correlation with revenue
- [ ] Input metrics identified (3-5)
- [ ] Dashboards tracking NSM + inputs
- [ ] Team aligned on NSM

## AARRR Framework
- [ ] Acquisition channels tracked
- [ ] Activation event defined
- [ ] Retention curves analyzed
- [ ] Referral mechanism in place
- [ ] Revenue metrics monitored

## Metric Hygiene
- [ ] Definitions documented
- [ ] Calculation methods clear
- [ ] Consistent across teams
- [ ] Regular reporting cadence
- [ ] Avoid vanity metrics

Cohort Analysis & Retention

Overview

Retention is the single most important metric for product success. Cohort analysis reveals which users stick around and why, enabling you to build more engaging products.

Key Insight: A 5% increase in retention can lead to 25-95% increase in profits. Retaining users is far cheaper than acquiring new ones.

---

What is Cohort Analysis?

Definition

Cohort analysis groups users by shared characteristics (usually sign-up date) and tracks their behavior over time.

Cohort = Group of users sharing a common characteristic

Common cohorts:
• Time-based: Users who signed up in January
• Acquisition: Users from Google Ads
• Behavioral: Users who completed onboarding
• Demographic: Users from California

Why Cohort Analysis Matters

✓ Reveals retention patterns over time
✓ Identifies product-market fit (PMF)
✓ Measures impact of product changes
✓ Compares quality across acquisition channels
✓ Validates growth strategies
✓ Predicts future revenue

Example:
Total active users = 10,000 (looks good!)
But: 9,000 are new this month, 1,000 from prior months
→ Terrible retention, unsustainable growth

---

Types of Cohort Analysis

1. Acquisition Cohorts (Time-Based)

Most common: Group by sign-up date

Cohort: Jan 2025 sign-ups
Track: How many return Day 1, 7, 30, 90?

Example Table:
| Cohort    | Size  | D1   | D7   | D30  | D90  |
|-----------|-------|------|------|------|------|
| Jan 2025  | 1000  | 60%  | 35%  | 20%  | 15%  |
| Dec 2024  | 950   | 58%  | 32%  | 18%  | 14%  |
| Nov 2024  | 900   | 55%  | 30%  | 16%  | 12%  |

Insight: Retention improving month-over-month! ✓

2. Behavioral Cohorts

Group by actions taken

Examples:
• Users who completed onboarding
• Users who invited teammates
• Users who used Feature X
• Users who made first purchase

Purpose: Identify retention drivers

Example:
Cohort A: Completed onboarding → 80% D30 retention
Cohort B: Skipped onboarding → 20% D30 retention

Insight: Onboarding is critical! Focus here.

3. Acquisition Channel Cohorts

Group by traffic source

Cohorts:
• Organic search
• Paid ads
• Referrals
• Social media
• Direct

Purpose: Measure channel quality

Example:
| Channel  | D30 Retention | CPA  | LTV  | LTV:CAC |
|----------|---------------|------|------|---------|
| Organic  | 40%           | $10  | $200 | 20:1    |
| Paid     | 25%           | $50  | $100 | 2:1     |
| Referral | 55%           | $5   | $300 | 60:1    |

Insight: Referrals = highest quality users

---

Retention Calculation Methods

Method 1: Classic Retention (Unbounded)

Definition: % of cohort active on Day N (regardless of gaps)

Day N Retention = Users active on Day N / Cohort size

Example:
Cohort: 1,000 users signed up Jan 1
Day 7: 350 users were active (anytime on Day 7)

Day 7 Retention = 350 / 1,000 = 35%

Use case: Standard retention metric

Method 2: Bracket Retention (Range)

Definition: % of cohort active during period (e.g., Week 1 = Day 1-7)

Week N Retention = Users active in Week N / Cohort size

Example:
Cohort: 1,000 users signed up Jan 1
Week 1 (Day 1-7): 600 users active at least once

Week 1 Retention = 600 / 1,000 = 60%

Use case: More forgiving, accounts for usage patterns

Method 3: N-Day Return Retention

Definition: % of cohort active on EXACTLY Day N (not before)

Day N Return = Users active Day N (not Day N-1) / Cohort size

Use case: Measure re-engagement campaigns

Method 4: Rolling Retention

Definition: % of cohort active on Day N or any day after

Rolling Day N = Users active Day N or later / Cohort size

Use case: Best for measuring long-term engagement

---

Retention Cohort Table

Standard Format

| Cohort     | Size | D0   | D1   | D7   | D14  | D30  | D60  | D90  |
|------------|------|------|------|------|------|------|------|------|
| 2025-01-01 | 1000 | 100% | 60%  | 35%  | 28%  | 20%  | 17%  | 15%  |
| 2025-01-02 | 1050 | 100% | 62%  | 37%  | 30%  | 22%  | 18%  | 16%  |
| 2025-01-03 | 980  | 100% | 58%  | 33%  | 26%  | 19%  | 16%  | 14%  |
| 2025-01-04 | 1100 | 100% | 64%  | 40%  | 32%  | 24%  | 20%  | 18%  |

Observations:
• Jan 4 cohort retains best (product improvement?)
• Typical pattern: sharp drop D0→D1, then gradual decline
• D90 retention stabilizing ~15-18% (core engaged users)

Heatmap Visualization

        D1   D7   D14  D30  D60  D90
Jan-01  🟢   🟡   🟡   🟠   🟠   🔴
Jan-02  🟢   🟢   🟡   🟡   🟠   🟠
Jan-03  🟢   🟡   🟠   🟠   🔴   🔴
Jan-04  🟢   🟢   🟢   🟡   🟡   🟠

🟢 = High retention (>50%)
🟡 = Medium (30-50%)
🟠 = Low (15-30%)
🔴 = Very low (<15%)

---

Retention Curves

The Ideal Curve

100% ┐
     │╲
 80% │ ╲
     │  ╲_________ ← Flattens = found core users
 60% │
     │
 40% │
     │
 20% │
     │
  0% └─────────────────────────────────
     D0  D1  D7   D30   D60   D90  D180

Good retention curve:
• Sharp initial drop (casual users churn)
• Flattens (core engaged users remain)
• Plateau = product-market fit

Bad Retention Curves

## Continuous Decline (No PMF)
100% ┐
     │╲
     │ ╲
     │  ╲
     │   ╲
     │    ╲______
  0% └─────────────────
     D0  D1  D7  D30

Problem: Never flattens, users keep churning
Diagnosis: No product-market fit
Action: Rethink product, find core value

## Smile Curve (Re-engagement)
100% ┐
     │╲
 50% │ ╲___/─── ← Rises back up
     │
  0% └─────────────────
     D0  D1  D7  D30

Pattern: Drop then rise
Diagnosis: Effective re-engagement (email, push)
OR: Subscription renewal cycle
Action: Optimize re-engagement campaigns

---

Retention Benchmarks

By Product Type

## Consumer Social (Facebook, Instagram, TikTok)
D1:  60-80%
D7:  40-60%
D30: 30-50%

## SaaS B2B (Slack, Asana, Notion)
D1:  70-90%
D7:  50-70%
D30: 40-60%

## E-commerce (Amazon, Shopify stores)
D1:  20-40%
D7:  10-25%
D30: 5-15%

## Gaming (Mobile games)
D1:  40-60%
D7:  20-40%
D30: 10-20%

## Fintech (Banking, Investment apps)
D1:  60-80%
D7:  50-70%
D30: 40-60%

Note: Wide variance based on product category and quality

"Good" Retention Thresholds

## General Guidelines
D1 retention:
• >70% = Excellent
• 50-70% = Good
• 30-50% = Average
• <30% = Poor

D30 retention:
• >40% = Excellent (strong PMF)
• 25-40% = Good
• 15-25% = Average
• <15% = Poor (PMF questionable)

Retention curve shape:
• Flattens by D30-60 = Good (found core users)
• Continuous decline = Bad (no engaged user base)

---

Analyzing Retention

Cohort Comparison

## Question: Did new feature improve retention?

Analysis:
| Cohort       | Feature | D30 Retention |
|--------------|---------|---------------|
| Pre-launch   | No      | 18%           |
| Post-launch  | Yes     | 25%           |

Result: +7pp improvement!

## Question: Which channel has best retention?

Analysis:
| Channel      | D30 Retention |
|--------------|---------------|
| Organic      | 35%           |
| Paid ads     | 20%           |
| Referrals    | 50%           |

Insight: Referrals >> Organic >> Paid
Action: Invest in referral program

Segmentation

Break cohorts into segments:

By user properties:
• Geography: US vs EU vs Asia
• Plan: Free vs Pro vs Enterprise
• Company size: SMB vs Mid-market vs Enterprise

By behavior:
• Activated vs Not activated
• Used Feature X vs Didn't use
• Invited teammates vs Solo user

Example:
| Segment              | D30 Retention |
|----------------------|---------------|
| Invited teammates    | 60%           |
| Didn't invite        | 15%           |

Insight: Collaboration = retention driver
Action: Encourage invites during onboarding

---

Improving Retention

Identify Retention Drivers

-- SQL: Find behaviors correlated with retention
WITH retained_users AS (
  SELECT DISTINCT user_id
  FROM events
  WHERE event_name = 'session_started'
    AND user_id IN (
      SELECT user_id FROM users
      WHERE created_at >= '2025-01-01'
    )
    AND timestamp >= (
      SELECT created_at + INTERVAL '30 days'
      FROM users u WHERE u.user_id = events.user_id
    )
),
feature_usage AS (
  SELECT
    user_id,
    MAX(CASE WHEN event_name = 'invited_teammate' THEN 1 ELSE 0 END) as invited,
    MAX(CASE WHEN event_name = 'created_project' THEN 1 ELSE 0 END) as created_project,
    MAX(CASE WHEN event_name = 'integrated_slack' THEN 1 ELSE 0 END) as integrated
  FROM events
  WHERE event_name IN ('invited_teammate', 'created_project', 'integrated_slack')
  GROUP BY user_id
)
SELECT
  'invited_teammate' as feature,
  SUM(CASE WHEN r.user_id IS NOT NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as retention_rate
FROM feature_usage f
LEFT JOIN retained_users r ON f.user_id = r.user_id
WHERE f.invited = 1

UNION ALL

SELECT
  'created_project',
  SUM(CASE WHEN r.user_id IS NOT NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*)
FROM feature_usage f
LEFT JOIN retained_users r ON f.user_id = r.user_id
WHERE f.created_project = 1

UNION ALL

SELECT
  'integrated_slack',
  SUM(CASE WHEN r.user_id IS NOT NULL THEN 1 ELSE 0 END) * 100.0 / COUNT(*)
FROM feature_usage f
LEFT JOIN retained_users r ON f.user_id = r.user_id
WHERE f.integrated = 1;

-- Results might show:
-- invited_teammate: 65% retention
-- created_project: 45% retention
-- integrated_slack: 75% retention
-- → Focus on Slack integration!

Activation Impact on Retention

## Hypothesis: Faster activation = Better retention

Analysis:
| Time to Activate | Cohort Size | D30 Retention |
|------------------|-------------|---------------|
| < 1 hour         | 2000        | 50%           |
| 1-24 hours       | 3000        | 35%           |
| 24+ hours        | 1500        | 15%           |
| Never activated  | 1000        | 3%            |

Insight: Speed to activation critical
Action: Streamline onboarding, reduce time to first value

Re-engagement Strategies

## Email Campaigns
Trigger: User inactive for 7 days
Subject: "We miss you! Here's what's new"
Content: Product updates, tips, social proof

Trigger: User inactive for 30 days
Subject: "Your team is waiting for you"
Content: Team activity, missed messages

## Push Notifications
Trigger: Friend posts update
Message: "Alice shared a new project"

Trigger: Weekly digest
Message: "5 new features this week"

## In-App Messaging
Trigger: Login after 14 days
Message: "Welcome back! Check out these new features"

## Retargeting Ads
Target: Churned users (inactive 60+ days)
Message: "Come back and see what's changed"
Offer: Discount or incentive

Habit Formation

## Daily Streaks
Example: Duolingo
• Visual streak counter
• Reminder notifications
• Streak freeze (don't break it!)

Impact: Creates daily habit loop

## Email Digests
Frequency: Daily or weekly
Content: Personalized activity summary
Goal: Pull users back regularly

## Scheduled Notifications
Example: Fitness app
• Morning: "Time for your workout"
• Evening: "Log your meals"

Builds routine around product

---

Churn Analysis

What is Churn?

Churn = When users stop using your product

Churn Rate = (Users lost / Total users at start) × 100%

Example:
Start of month: 1,000 users
End of month: 950 users
Churned: 50 users

Churn rate = 50 / 1,000 = 5%

Churn Calculation Methods

## User Churn
Churn Rate = Churned users / Total users

Example: 50 / 1,000 = 5% monthly churn

## Revenue Churn (SaaS)
MRR Churn Rate = Churned MRR / Starting MRR

Example:
Start: $100,000 MRR
Churned: $5,000 MRR
MRR Churn = $5,000 / $100,000 = 5%

## Net Revenue Retention (NRR)
NRR = (Starting MRR + Expansion - Churn) / Starting MRR

Example:
Starting MRR: $100,000
Expansion (upsells): +$10,000
Churned MRR: -$5,000

NRR = ($100,000 + $10,000 - $5,000) / $100,000 = 105%

NRR > 100% = Growing from existing customers! ✓

Predicting Churn

## At-Risk User Signals
• Login frequency dropped 50%
• Haven't used core feature in 14 days
• Support ticket: "How do I cancel?"
• Email open rate declined
• Team size decreased (removed seats)

## Churn Prediction Model
Use ML to score users 0-100 (likelihood to churn)

Features:
• Days since last login
• Feature usage frequency
• Support ticket count
• Payment failed attempts
• Engagement trend (increasing/decreasing)

Output: Churn risk score
→ Proactive outreach to high-risk users

Preventing Churn

## Early Intervention
Trigger: User hasn't logged in for 7 days
Action: Email: "Need help getting started?"

Trigger: Usage declining
Action: In-app message: "Having trouble? Talk to us"

## Customer Success
High-value accounts:
• Dedicated customer success manager
• Quarterly business reviews
• Proactive check-ins

## Cancellation Flow
When user clicks "Cancel":
• Survey: "Why are you leaving?"
• Offer alternative: Downgrade vs cancel?
• Retention offer: 50% off for 3 months
• Make it easy to pause (not cancel)

## Win-Back Campaigns
Target: Churned users
Message: "We've improved based on your feedback"
Incentive: Free month, discount, new features

---

Cohort Analysis in Practice

SQL Queries

-- Retention cohort table
WITH cohorts AS (
  SELECT
    DATE_TRUNC('week', created_at) as cohort_week,
    user_id
  FROM users
  WHERE created_at >= '2025-01-01'
),
activity AS (
  SELECT
    DATE_TRUNC('week', timestamp) as activity_week,
    user_id
  FROM events
  WHERE event_name = 'session_started'
    AND timestamp >= '2025-01-01'
)
SELECT
  c.cohort_week,
  COUNT(DISTINCT c.user_id) as cohort_size,
  a.activity_week,
  FLOOR((EXTRACT(EPOCH FROM a.activity_week - c.cohort_week) / 604800)) as weeks_since_signup,
  COUNT(DISTINCT a.user_id) as active_users,
  ROUND(100.0 * COUNT(DISTINCT a.user_id) / COUNT(DISTINCT c.user_id), 2) as retention_pct
FROM cohorts c
LEFT JOIN activity a ON c.user_id = a.user_id
GROUP BY c.cohort_week, a.activity_week
ORDER BY c.cohort_week, weeks_since_signup;

Python Analysis

import pandas as pd
import numpy as np

# Load data
users = pd.read_sql("SELECT user_id, created_at FROM users", conn)
events = pd.read_sql("SELECT user_id, timestamp FROM events WHERE event_name = 'session_started'", conn)

# Create cohorts
users['cohort'] = users['created_at'].dt.to_period('M')

# Calculate retention
retention = []
for cohort in users['cohort'].unique():
    cohort_users = users[users['cohort'] == cohort]['user_id']
    cohort_size = len(cohort_users)

    for period in range(0, 12):  # 12 months
        target_month = cohort + period
        active_users = events[
            (events['user_id'].isin(cohort_users)) &
            (events['timestamp'].dt.to_period('M') == target_month)
        ]['user_id'].nunique()

        retention.append({
            'cohort': cohort,
            'period': period,
            'active_users': active_users,
            'cohort_size': cohort_size,
            'retention_pct': 100 * active_users / cohort_size
        })

retention_df = pd.DataFrame(retention)

# Pivot for heatmap
retention_pivot = retention_df.pivot(
    index='cohort',
    columns='period',
    values='retention_pct'
)

print(retention_pivot)

---

2025 Trends

1. Predictive Cohort Analytics

• ML-powered churn prediction
• Automated cohort discovery
• Real-time retention scoring
• AI-generated insights

2. Product-Led Growth (PLG) Focus

• Self-serve onboarding
• In-product activation
• Usage-based pricing
• Bottom-up adoption (user → team → enterprise)

Retention is THE metric for PLG companies

3. Privacy-First Cohort Analysis

• Cookieless tracking
• Aggregated cohorts (privacy preserving)
• Differential privacy
• First-party data focus

4. Real-Time Cohort Monitoring

• Sub-second cohort updates
• Live retention dashboards
• Instant alerts on retention drops
• Stream processing (Kafka, Flink)

---

Tools

Cohort Analysis Platforms

---

Checklist

## Setup
- [ ] Define what "active" means (core action)
- [ ] Choose retention metric (Day 1, 7, 30)
- [ ] Set up event tracking
- [ ] Build cohort table/dashboard
- [ ] Establish baseline retention

## Analysis
- [ ] Monitor retention trends over time
- [ ] Compare cohorts (time, channel, behavior)
- [ ] Identify retention drivers (behavioral analysis)
- [ ] Segment high vs low retention users
- [ ] Calculate churn rate

## Action
- [ ] Optimize onboarding (activation)
- [ ] Build re-engagement campaigns
- [ ] Test retention improvements (A/B test)
- [ ] Implement churn prevention
- [ ] Measure impact of changes on retention

Event Tracking Plan Template

Overview

Product: [Product Name] Last Updated: [Date] Owner: [Product Manager / Team] Version: [1.0]

---

Purpose

This tracking plan documents all analytics events tracked in [Product Name]. It serves as the single source of truth for:

• Event definitions and naming conventions
• Property schemas and data types
• Implementation locations
• Business purpose and usage

---

Naming Conventions

Event Naming

Pattern: object_action (snake_case)

Examples:
✓ user_signed_up
✓ project_created
✓ file_uploaded
✓ payment_completed

Format Rules:
• snake_case (lowercase, underscores)
• Past tense for completed actions
• Present tense for ongoing states
• Descriptive and specific

Property Naming

Format: snake_case

Data Types:
• Strings: "value"
• Numbers: 123, 45.67
• Booleans: true, false
• Timestamps: ISO 8601 ("2025-12-16T10:30:00Z")
• Arrays: ["item1", "item2"]
• Objects: { key: "value" }

---

Global Properties

These properties are included in every event:

*Either user_id OR anonymous_id required

Context Properties (Recommended)

UTM Parameters (Marketing)

---

Event Catalog

User Lifecycle Events

user_signed_up

Description: Triggered when user completes registration

When to trigger:

• After successful account creation
• Before redirect to onboarding

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: AuthService.createAccount() success callback
• Location: src/services/auth.ts (line 145)

Related Events:

• signup_started (precedes)
• user_activated (follows)

Used in Metrics:

• Acquisition metrics
• Sign-up conversion rate
• Channel performance

Example:

{
  "event": "user_signed_up",
  "timestamp": "2025-12-16T10:30:00.123Z",
  "user_id": "usr_7b3f8e2a",
  "properties": {
    "signup_method": "google",
    "plan": "trial",
    "trial_days": 14
  }
}

---

user_activated

Description: User completed first key action (activation milestone)

When to trigger:

• When user completes activation action
• Definition: Created first project OR invited first teammate

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: First occurrence of activation action
• Location: src/lib/analytics/activation.ts

Related Events:

• user_signed_up (precedes)
• project_created OR teammate_invited (triggers this)

Used in Metrics:

• Activation rate
• Time to activation
• Onboarding conversion

---

Feature Interaction Events

project_created

Description: User created a new project

When to trigger:

• After project successfully saved to database
• Before showing success message

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: ProjectService.create() success
• Location: src/services/projects.ts

Related Events:

• project_updated
• project_deleted
• user_activated (may trigger)

Used in Metrics:

• Feature adoption
• Activation rate
• User engagement

---

file_uploaded

Description: User uploaded a file to project

When to trigger:

• After file upload completes
• Before showing in UI

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: FileService.upload() complete
• Location: src/services/files.ts

---

Commerce Events

checkout_completed

Description: User successfully completed payment

When to trigger:

• After payment processor confirms success
• Before order confirmation page

Properties:

Implementation:

• Platform: Web (server-side)
• Trigger: PaymentService.processPayment() webhook
• Location: src/api/webhooks/stripe.ts

Related Events:

• checkout_started (precedes)
• subscription_started (follows)

Used in Metrics:

• Revenue (MRR, ARR)
• Conversion to paid
• Average order value

---

Engagement Events

session_started

Description: User started a new session

When to trigger:

• On page load (web)
• On app launch (mobile)
• After 30 minutes of inactivity

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: Session initialization
• Location: src/lib/analytics/session.ts

Used in Metrics:

• Daily/Monthly Active Users
• Session frequency
• Retention

---

page_viewed

Description: User viewed a page

When to trigger:

• On page load (SPA route change)
• Only for significant pages (not every scroll)

Properties:

Implementation:

• Platform: Web
• Trigger: React Router navigation
• Location: src/App.tsx (route change listener)

---

Error Events

error_occurred

Description: Application error occurred

When to trigger:

• On caught exceptions
• On API errors (4xx, 5xx)
• On validation failures

Properties:

Implementation:

• Platform: Web, iOS, Android
• Trigger: Error boundary, catch blocks
• Location: src/lib/analytics/errors.ts

Used in Metrics:

• Error rate
• Reliability metrics
• User friction points

---

Implementation Checklist

Before Launch

• [ ] Event naming conventions documented
• [ ] All events reviewed by engineering
• [ ] Privacy review completed (no PII)
• [ ] Analytics SDK integrated
• [ ] Event validation implemented

After Launch

• [ ] Events firing correctly (QA testing)
• [ ] Dashboards created
• [ ] No duplicate events
• [ ] Sample ratio correct (50/50 splits)
• [ ] Data quality monitoring active

Ongoing

• [ ] Monthly tracking plan review
• [ ] Update for new features
• [ ] Deprecate unused events
• [ ] Validate data quality
• [ ] Document schema changes

---

Privacy & Compliance

PII Policy

Never track:

• Passwords
• Credit card numbers
• Social Security Numbers
• Full email addresses (hash or mask)
• Phone numbers (mask)
• Private keys, API tokens

Masking Examples:

// Email: user@example.com → u***r@example.com
const maskEmail = (email) => {
  const [name, domain] = email.split('@');
  return `${name[0]}***${name[name.length - 1]}@${domain}`;
};

// Phone: 555-123-4567 → ****4567
const maskPhone = (phone) => `****${phone.slice(-4)}`;

// Credit card: 4242424242424242 → ****4242
const maskCard = (card) => `****${card.slice(-4)}`;

GDPR/CCPA Compliance

• [ ] User consent obtained before tracking
• [ ] Opt-out mechanism available
• [ ] Data deletion process documented
• [ ] Data retention policy defined (365 days)
• [ ] User data export capability

---

Testing

Event Validation

// Example: Zod schema for event validation
import { z } from 'zod';

const CheckoutCompletedSchema = z.object({
  event: z.literal('checkout_completed'),
  timestamp: z.string().datetime(),
  user_id: z.string().min(1),
  properties: z.object({
    order_id: z.string().min(1),
    total_amount: z.number().positive(),
    currency: z.enum(['USD', 'EUR', 'GBP']),
    item_count: z.number().int().positive(),
    payment_method: z.enum(['credit_card', 'paypal', 'apple_pay']),
    plan: z.string().min(1),
    billing_cycle: z.enum(['monthly', 'annual']),
    coupon_code: z.string().optional(),
    discount_amount: z.number().nonnegative().optional(),
    tax_amount: z.number().nonnegative().optional()
  })
});

// Use in code
function trackCheckoutCompleted(data) {
  const validated = CheckoutCompletedSchema.parse(data);
  sendToAnalytics(validated);
}

QA Checklist

For each event:

• [ ] Event fires at correct time
• [ ] All required properties present
• [ ] Data types correct
• [ ] No PII in properties
• [ ] Consistent naming (snake_case)
• [ ] Appears in analytics tool
• [ ] Associated with correct user_id

---

Analytics Tools

Primary Platform

• Tool: [Amplitude / Mixpanel / PostHog]
• Environment: Production
• API Key: [Stored in env vars]

Data Warehouse

• Destination: [Snowflake / BigQuery]
• Pipeline: [Segment / Fivetran]
• Sync Frequency: Real-time

Dashboards

• Product Metrics: [Link to dashboard]
• Growth Metrics: [Link to dashboard]
• Revenue Metrics: [Link to dashboard]

---

Changelog

Version 1.0 (2025-12-16)

• Initial tracking plan
• Defined 15 core events
• Established naming conventions
• Privacy policy documented

Future Versions

• Document changes here
• Track schema updates
• Note deprecated events

---

Contact

Questions or Updates:

• Product Manager: [name@company.com]
• Engineering Lead: [name@company.com]
• Analytics Team: [analytics@company.com]

Slack Channel: #product-analytics