Predictive Analytics

Predictive Analytics

When to Use

• Define the prediction target, unit of analysis, horizon, and success criteria before modeling
• Audit leakage, label timing, and train/validation design for tabular or time-ordered data
• Engineer and select features for churn, propensity, fraud, demand, or risk scoring use cases
• Choose model families and baselines (linear, tree ensembles, gradient boosting) matched to data size and interpretability needs
• Run validation: holdout, cross-validation, or time-based splits with metrics aligned to the decision
• Tune calibration, thresholds, and cost-sensitive operating points for classification and scores
• Explain models at practitioner level (importance, partial dependence, SHAP-style intuition—not full XAI research)
• Plan conceptual post-deployment monitoring: drift signals, retrain triggers, and limitation language for stakeholders

When NOT to Use

• MLOps platform build—feature stores, model registry, K8s serving, CI for training pipelines → ml-infrastructure-engineer-safeguards, ml-ops-engineer (if installed)
• Deep learning research, LLM fine-tuning, or novel architecture search → ml-research-engineer-safeguards, ai-engineer
• Actuarial reserving, loss development, rate indication, or regulatory pricing math → actuarial-analyst, advanced-short-term-actuarial-mathematics, actuary
• Dashboard-only KPI layout, chart design, or viz specs without modeling → data-visualization, bi-analyst
• Warehouse/dbt dimensional modeling, mart SQL, or pipeline idempotency only → data-warehouse-engineer, analytics-data-engineer
• A/B test design, power, randomization, and experiment readouts → ab-testing-engineer
• Markets/finance factor research, backtests, and alpha hygiene → quantitative-researcher
• Cloud cost forecasting and FinOps allocation → finops-analyst

Related skills

Core Workflows

1. Frame the problem and label

1. State the business decision the score or forecast will drive 2. Define target (binary, multiclass, continuous), horizon, and grain (user, account, SKU, region) 3. Document label window and feature cutoff (what is knowable at scoring time) 4. Check base rate, class imbalance, and whether rules/heuristics already suffice 5. List constraints: latency, interpretability, fairness review, regulatory context

See `references/problem_framing_and_data_prep.md`.

2. Prepare data and features

1. Profile missingness, cardinality, outliers, and temporal coverage 2. Build point-in-time feature tables; never join future outcomes into training rows 3. Encode categoricals, scale numerics, and handle high-cardinality keys deliberately 4. Document feature definitions, refresh cadence, and upstream dependencies 5. Split data with the right strategy (random, grouped, or time-based)

See `references/problem_framing_and_data_prep.md`.

3. Model, validate, and compare

1. Fit baselines first (majority class, linear/logistic, naive forecast) 2. Iterate tree ensembles or regularized models; avoid unnecessary complexity 3. Select metrics tied to the decision (PR-AUC for rare events, MAPE vs WAPE for demand, etc.) 4. Use cross-validation or rolling-origin evaluation for time-ordered problems 5. Compare models on holdout data never used for tuning

See `references/modeling_and_validation.md`.

4. Classification, propensity, and scores

1. Calibrate probabilities when ranks are not enough for thresholds or expected value 2. Set thresholds using cost matrices, capacity, or top-decile lift—not default 0.5 3. Report lift, capture, and stability by segment and time 4. Separate model quality from policy (who gets contacted, approved, or reviewed)

See `references/classification_and_propensity.md`.

5. Forecasting and time series (when applicable)

1. Choose granularity (daily/weekly) and horizon aligned to planning cycles 2. Use naive and seasonal baselines; add exogenous features only when available at forecast time 3. Validate with rolling or blocked time splits; report interval or quantile forecasts when decisions need range 4. Document seasonality, promotions, and structural breaks that limit extrapolation

See `references/forecasting_and_time_series.md`.

6. Deploy, monitor, and communicate

1. Specify scoring cadence, input schema, and fallback when features are missing 2. Define monitoring: label delay, score distribution drift, feature drift, and business outcome tracking 3. Set retrain triggers (calendar, performance decay, population shift)—conceptual triggers only here 4. Deliver a model card or memo: intended use, limitations, metrics, and what would falsify trust

See `references/deployment_monitoring_and_communication.md`.

When to load references

Classification and Propensity

Table of contents

1. Propensity vs classification 2. Scores, ranks, and probabilities 3. Calibration 4. Threshold selection 5. Cost-sensitive decisions 6. Capacity and workflow constraints 7. Segmentation and stability 8. Common use-case patterns

Propensity vs classification

Propensity models often feed ranking (who to contact first). Classification adds an operating point (approve/deny, flag/review).

Scores, ranks, and probabilities

• Raw score (logit, margin): Good for ranking if monotonic; not calibrated.
• Probability: Needed for expected value, budgeting, and some compliance reviews.
• Decile lift: Compares top decile event rate to population rate—easy for stakeholders.

Report which output production will consume.

Calibration

Uncalibrated models can rank well but misstate risk.

Evaluate with reliability diagrams and Brier score on a calibration holdout—not the same fold used for threshold tuning if possible.

Threshold selection

Do not default to 0.5. Choose thresholds by:

1. Cost matrix (FP vs FN costs in business units) 2. Capacity (max reviews, calls, or approvals per day) 3. Target precision (fraud: “at 95% precision, what recall?”) 4. Top-k policy (contact best 5% of list)

Document the expected confusion matrix at the chosen point and sensitivity to ±0.05 score shift.

Cost-sensitive decisions

Example framing:

Expected value = P(positive) * benefit_true_positive
               - P(false positive) * cost_false_positive
               - action_cost

When costs are unknown, present threshold curves: precision/recall, lift, and workload vs cutoff.

For fraud and compliance queues, pair with anti-false-positive-decision-making when alert volume and evidence bars matter.

Capacity and workflow constraints

Separate model evaluation from operating policy in documentation.

Segmentation and stability

Report metrics by:

• Acquisition cohort, tenure band, product, geography
• Time (train period vs recent holdout month)
• Score decile stability (do top deciles stay top?)

Investigate simpson’s paradox slices where global metrics hide failure modes.

Common use-case patterns

For experiment-based validation of treatments, route to ab-testing-engineer—do not confuse model lift with campaign lift.

Deployment, Monitoring, and Communication

Table of contents

1. Deployment modes (conceptual) 2. Scoring contract 3. Monitoring signals 4. Drift types 5. Retrain triggers 6. Explainability for stakeholders 7. Model card outline 8. Uncertainty and limitations language 9. Governance handoffs

Deployment modes (conceptual)

Platform implementation (containers, feature store, autoscaling) → ml-infrastructure-engineer-safeguards. This skill stops at requirements and acceptance criteria.

Scoring contract

Document for engineering partners:

• Input schema: feature names, types, max null rates
• Output: score, probability, decile, optional reason codes
• Version: model ID, training data snapshot date
• Fallback: default score or rules when features missing
• Latency budget and throughput
• Idempotency for batch replays

Monitoring signals

Track at minimum:

Set review cadence (daily for fraud, weekly for churn, etc.).

Drift types

Population Stability Index (PSI) is a common heuristic—document thresholds and false alarm tolerance.

Retrain triggers

Define triggers in advance:

• Calendar (quarterly retrain) for stable domains
• Performance decay (PR-AUC drop vs benchmark on recent labeled window)
• Drift breach (PSI or KS on key features/scores)
• Material business change (new product line, pricing overhaul)
• Regulatory / policy change requiring review

Include rollback plan: keep prior model version until new model passes shadow or champion/challenger checks (conceptual).

Explainability for stakeholders

Practitioner toolkit (not full XAI research):

Caveats to state:

• Importance ≠ causation
• Correlated features split importance arbitrarily
• Explanations trained on historical bias reflect historical decisions

Fairness and adverse action processes are organizational—flag review with legal/compliance; do not provide legal conclusions.

Model card outline

1. Intended use and non-goals 2. Training data (dates, population, exclusions) 3. Target definition and horizon 4. Features (categories, PII, refresh) 5. Metrics on holdout and key slices 6. Calibration and threshold policy 7. Limitations and known failure segments 8. Monitoring plan and owners 9. Version history and approval sign-off

Uncertainty and limitations language

Prefer:

• “On holdout data from Q1–Q3, top decile had 3.2× lift vs random.”
• “Precision at our review capacity (2k/day) was 42%; expect ±5% if base rate shifts.”
• “Forecasts are median scenarios; intervals covered 80% of holdout weeks.”

Avoid:

• “The model proves causation.”
• “Accuracy is 99%” without prevalence context.
• Point forecasts without horizon or segment caveats.

Governance handoffs

Forecasting and Time Series

Table of contents

1. When this applies 2. Problem setup 3. Baselines 4. Feature patterns 5. Model choices (tabular focus) 6. Validation for time order 7. Intervals and scenarios 8. Demand forecasting pitfalls 9. Handoff to stakeholders

When this applies

Use this reference when the target is future values over time (demand, volume, revenue, tickets) rather than a one-time entity label. For cross-sectional tabular classification, use classification_and_propensity.md.

Markets-style factor research and backtests → quantitative-researcher.

Problem setup

Define explicitly:

Align granularity with planning cadence (ops buys weekly; finance plans monthly).

Baselines

Required before ML:

Report MASE (vs seasonal naive) when seasonality is strong—scale-free comparison.

Feature patterns

Lag and window features (all computed with cutoff at origin time):

• Lags: y_{t-1}, y_{t-7}, y_{t-52}
• Rolling means / std / min / max over windows
• Calendar: day-of-week, month, holiday flags
• Promotions: lead/lag flags only if known in advance
• Price and stock: use planned price; document stockout censoring

Avoid global normalization that uses future series statistics.

Model choices (tabular focus)

Deep sequence models are out of default scope unless paired with data-scientist / research skills.

Validation for time order

Never shuffle time randomly for final evaluation.

Report error by horizon step (h=1 often easier than h=4).

Intervals and scenarios

Stakeholders often need ranges, not only point forecasts:

• Quantile regression or conformal-style intervals (workflow level)
• Scenario overlays: promo on/off, supply shock assumptions
• Clearly label intervals are not guarantees—coverage on holdout only

Demand forecasting pitfalls

Handoff to stakeholders

Deliver:

1. Forecast table with horizon, grain, point, optional intervals 2. Accuracy history on holdout by horizon and key segments 3. Assumptions (promo calendar, price list, capacity) 4. Known failure modes (new markets, sparse history)

Visualization of forecasts → data-visualization; warehouse pipelines for history → analytics-data-engineer.

Modeling and Validation

Table of contents

1. Modeling philosophy 2. Baseline ladder 3. Model families 4. Feature selection 5. Hyperparameters and tuning 6. Validation design 7. Metrics by problem type 8. Diagnostics and sanity checks

Modeling philosophy

• Simple first: If baselines are strong, complexity needs a measured lift.
• Match interpretability to stakeholder needs and regulatory review.
• Separate offline model selection from policy (thresholds, queues, budgets).
• Document random seeds, data snapshots, and library versions for reproducibility.

Baseline ladder

Always report at least one baseline before advanced models:

If uplift over baseline is small, question whether ML is worth operational cost.

Model families

Avoid deep learning for typical business tabular problems unless data-scientist or research skills justify it.

Feature selection

Principles:

1. Domain-first: Keep features with causal or operational story when possible. 2. Remove redundant highly correlated columns after documenting which to keep. 3. Use embedded selection (L1, tree importance) on training folds only—not holdout. 4. Prefer stability across CV folds over single-run importance rankings. 5. Cap dimensionality when rows are limited (rules of thumb: tens of features per thousand positives for rare events).

Do not iterate feature sets using holdout performance.

Hyperparameters and tuning

• Tune on inner CV or a dedicated validation fold.
• Track search budget; avoid exhaustive grids on large data without need.
• Record best params and variance across folds—a lucky fold is not production-ready.
• For boosting: watch early stopping on validation; document n_estimators effective count.

Validation design

Final holdout is touched once for the delivery report unless deploying iterative research under governance.

Metrics by problem type

Binary classification (imbalanced)

Avoid accuracy as the sole metric when prevalence < 10%.

Multiclass

Macro vs weighted F1 depending on whether rare classes matter equally; report per-class recall for operational classes.

Regression

Forecasting

Report accuracy by horizon bucket (1-step vs 4-step ahead). See forecasting_and_time_series.md.

Diagnostics and sanity checks

• Calibration plot (reliability curve) for classifiers used with thresholds
• Residual plots for regression; heteroscedasticity and tail errors
• Learning curves (train vs val) for over/underfit
• Slice analysis: time cohorts, regions, product lines, acquisition channel
• Score stability: rank correlation month-over-month on holdout cohorts

Flag any slice where the model is worse than baseline before release.

Predictive Analytics — Scope

Table of contents

1. Role definition 2. In scope 3. Out of scope 4. Typical use cases 5. Deliverables 6. Quality bar

Role definition

Predictive Analytics covers applied supervised learning for business decisions: defining what to predict, building tabular features, training and validating models, and communicating scores or forecasts with explicit uncertainty and limitations. The focus is practitioner-grade workflow—not research novelty, not platform engineering, and not actuarial statutory methods.

In scope

Out of scope

Typical use cases

Deliverables

1. Problem brief — decision, target, horizon, grain, exclusions 2. Data dictionary — features, cutoff rules, refresh cadence 3. Validation report — splits, metrics, baselines, segment slices 4. Model artifact spec — inputs, outputs, versioning, fallback behavior 5. Threshold / policy memo — costs, capacity, expected confusion matrix at operating point 6. Model card — intended use, limitations, monitoring plan, owner

Quality bar

Work is ready to hand off when:

• [ ] Baseline beats are documented; complexity is justified
• [ ] Leakage and label timing are explicitly ruled out or flagged
• [ ] Holdout or rolling evaluation was not used for hyperparameter tuning
• [ ] Metrics match the business decision (not accuracy alone on rare events)
• [ ] Segments (time, region, product) show stability or known gaps
• [ ] Limitations and retrain triggers are stated in plain language

Problem Framing and Data Prep

Table of contents

1. Decision-first framing 2. Target and horizon 3. Unit of analysis and grain 4. Leakage checklist 5. Label construction 6. Feature cutoff rules 7. Data quality gates 8. Split strategies

Decision-first framing

Start from the action, not the algorithm:

• Who receives the prediction, and what do they do differently?
• What is the cost of false positives vs false negatives (or forecast error direction)?
• Is a score rank enough, or are calibrated probabilities / dollar forecasts required?
• Can a simple rule or segmentation deliver 80% of value?

Document answers before selecting features or models.

Target and horizon

Align horizon with when the business can act and when features are frozen.

Unit of analysis and grain

Pick one primary grain and stick to it:

• User — consumer apps, subscriptions
• Account / customer — B2B, householding rules documented
• SKU-location-day — retail demand
• Transaction — fraud (often with user/account aggregates as features)

If labels exist at a different grain (e.g., account churn from user events), define aggregation rules explicitly and avoid duplicate rows that inflate confidence.

Leakage checklist

Leakage makes offline metrics lie. Audit for:

• [ ] Features computed after the prediction moment (including “current status” fields)
• [ ] Target information embedded in features (e.g., cancellation_reason before cancel)
• [ ] Random train/test split on time-series or repeated measures per entity
• [ ] Future aggregates (30-day spend including days after scoring date)
• [ ] Test set used for feature selection, imputation stats, or target encoding fit globally
• [ ] Duplicate entities across train and test without grouped splits
• [ ] Survivorship (only customers who stayed long enough to get a label)

When in doubt, rebuild features with a strict as-of timestamp per row.

Label construction

Document class balance and whether negatives are “true negatives” or “not yet positive.”

Feature cutoff rules

For each feature, record:

1. Source table and refresh lag 2. As-of rule (e.g., events with event_time < score_time) 3. Aggregation window (7d, 30d, lifetime-with-decay) 4. Default when missing (and whether missingness is informative)

Prefer interpretable aggregates over opaque embeddings unless ml-research-engineer-safeguards scope applies.

Data quality gates

Before modeling:

• Coverage by time, segment, and key dimensions
• Cardinality and rare level handling plan
• Outlier policy (cap, winsorize, flag)—document, do not silently drop
• Consistency between training snapshot and production schema
• Join fan-out checks (one-to-many explosions)

Split strategies

Never tune on the final holdout. For time series, see forecasting_and_time_series.md.