Ai Ml Data Science

Data Science Engineering Suite - Quick Reference

This skill turns raw data and questions into validated, documented models ready for production:

• EDA workflows: Structured exploration with drift detection
• Feature engineering: Reproducible feature pipelines with leakage prevention and train/serve parity
• Model selection: Baselines first; strong tabular defaults; escalate complexity only when justified
• Evaluation & reporting: Slice analysis, uncertainty, model cards, production metrics
• SQL transformation: SQLMesh for staging/intermediate/marts layers
• MLOps: CI/CD, CT (continuous training), CM (continuous monitoring)
• Production patterns: Data contracts, lineage, feedback loops, streaming features

Modern emphasis (2026): Feature stores, automated retraining, drift monitoring (Evidently), train-serve parity, and agentic ML loops (plan -> execute -> evaluate -> improve). Tools: LightGBM, CatBoost, scikit-learn, PyTorch, Polars (lazy eval for larger-than-RAM datasets), lakeFS for data versioning.

---

Quick Reference

---

Data Lake & Lakehouse

For comprehensive data lake/lakehouse patterns (beyond SQLMesh transformation), see [data-lake-platform](../data-lake-platform/SKILL.md):

• Table formats: Apache Iceberg, Delta Lake, Apache Hudi
• Query engines: ClickHouse, DuckDB, Apache Doris, StarRocks
• Alternative transformation: dbt (alternative to SQLMesh)
• Ingestion: dlt, Airbyte (connectors)
• Streaming: Apache Kafka patterns
• Orchestration: Dagster, Airflow

This skill focuses on ML feature engineering and modeling. Use data-lake-platform for general-purpose data infrastructure.

---

Related Skills

For adjacent topics, reference:

• [ai-mlops](../ai-mlops/SKILL.md) - APIs, batch jobs, monitoring, drift, data ingestion (dlt)
• [ai-llm](../ai-llm/SKILL.md) - LLM prompting, fine-tuning, evaluation
• [ai-rag](../ai-rag/SKILL.md) - RAG pipelines, chunking, retrieval
• [ai-llm-inference](../ai-llm-inference/SKILL.md) - LLM inference optimization, quantization
• [ai-ml-timeseries](../ai-ml-timeseries/SKILL.md) - Time series forecasting, backtesting
• [qa-testing-strategy](../qa-testing-strategy/SKILL.md) - Test-driven development, coverage
• [data-sql-optimization](../data-sql-optimization/SKILL.md) - SQL optimization, index patterns (complements SQLMesh)
• [data-lake-platform](../data-lake-platform/SKILL.md) - Data lake/lakehouse infrastructure (ClickHouse, Iceberg, Kafka)

---

Decision Tree: Choosing Data Science Approach

User needs ML for: [Problem Type]
  - Tabular data?
    - Small-medium (<1M rows)? -> LightGBM (fast, efficient)
    - Large and complex (>1M rows)? -> LightGBM first, then NN if needed
    - High-dim sparse (text, counts)? -> Linear models, then shallow NN

  - Time series?
    - Seasonality? -> LightGBM, then see ai-ml-timeseries
    - Long-term dependencies? -> Transformers (see ai-ml-timeseries)

  - Text or mixed modalities?
    - LLMs/Transformers -> See ai-llm

  - SQL transformations?
    - SQLMesh (staging/intermediate/marts layers)

Rule of thumb: For tabular data, tree-based gradient boosting is a strong baseline, but must be validated against alternatives and constraints.

---

Core Concepts (Vendor-Agnostic)

• Problem framing: define success metrics, baselines, and decision thresholds before modeling.
• Leakage prevention: ensure all features are available at prediction time; split by time/group when appropriate.
• Uncertainty: report confidence intervals and stability (fold variance, bootstrap) rather than single-point metrics.
• Reproducibility: version code/data/features, fix seeds, and record the environment.
• Operational handoff: define monitoring, retraining triggers, and rollback criteria with MLOps.

Implementation Practices (Tooling Examples)

• Track experiments and artifacts (run id, commit hash, data version).
• Add data validation gates in pipelines (schema + distribution + freshness).
• Prefer reproducible, testable feature code (shared transforms, point-in-time correctness).
• Use datasheets/model cards and eval reports as deployment prerequisites (Datasheets for Datasets: https://arxiv.org/abs/1803.09010; Model Cards: https://arxiv.org/abs/1810.03993).

Do / Avoid

Do

• Do start with baselines and a simple model to expose leakage and data issues early.
• Do run slice analysis and document failure modes before recommending deployment.
• Do keep an immutable eval set; refresh training data without contaminating evaluation.

Avoid

• Avoid random splits for temporal or user-correlated data.
• Avoid "metric gaming" (optimizing the number without validating business impact).
• Avoid training on labels created after the prediction timestamp (silent future leakage).

Core Patterns (Overview)

Pattern 1: End-to-End DS Project Lifecycle

Use when: Starting or restructuring any DS/ML project.

Stages:

1. Problem framing - Business objective, success metrics, baseline 2. Data & feasibility - Sources, coverage, granularity, label quality 3. EDA & data quality - Schema, missingness, outliers, leakage checks 4. Feature engineering - Per data type with feature store integration 5. Modelling - Baselines first, then LightGBM, then complexity as needed 6. Evaluation - Offline metrics, slice analysis, error analysis 7. Reporting - Model evaluation report + model card 8. MLOps - CI/CD, CT (continuous training), CM (continuous monitoring)

Detailed guide: EDA Best Practices

---

Pattern 2: Feature Engineering

Use when: Designing features before modelling or during model improvement.

By data type:

• Numeric: Standardize, handle outliers, transform skew, scale
• Categorical: One-hot/ordinal (low cardinality), target/frequency/hashing (high cardinality)
• Feature Store Integration: Store encoders, mappings, statistics centrally
• Text: Cleaning, TF-IDF, embeddings, simple stats
• Time: Calendar features, recency, rolling/lag features

Key Modern Practice: Use feature stores (Feast, Tecton, Databricks) for versioning, sharing, and train-serve parity.

Detailed guide: Feature Engineering Patterns

---

Pattern 3: Data Contracts & Lineage

Use when: Building production ML systems with data quality requirements.

Components:

• Contracts: Schema + ranges/nullability + freshness SLAs
• Lineage: Track source -> feature store -> train -> serve
• Feature store hygiene: Materialization cadence, backfill/replay, encoder versioning
• Schema evolution: Backward/forward-compatible migrations with shadow runs

Detailed guide: Data Contracts & Lineage

---

Pattern 4: Model Selection & Training

Use when: Picking model families and starting experiments.

Decision guide (modern benchmarks):

• Tabular: Start with a strong baseline (linear/logistic, then gradient boosting) and iterate based on error analysis
• Baselines: Always implement simple baselines first (majority class, mean, naive forecast)
• Train/val/test splits: Time-based (forecasting), group-based (user/item leakage), or random (IID)
• Hyperparameter tuning: Start manual, then Bayesian optimization (Optuna, Ray Tune)
• Overfitting control: Regularization, early stopping, cross-validation

Detailed guide: Modelling Patterns

---

Pattern 5: Evaluation & Reporting

Use when: Finalizing a model candidate or handing over to production.

Key components:

• Metric selection: Primary (ROC-AUC, PR-AUC, RMSE) + guardrails (calibration, fairness)
• Threshold selection: ROC/PR curves, cost-sensitive, F1 maximization
• Slice analysis: Performance by geography, user segments, product categories
• Error analysis: Collect high-error examples, cluster by error type, identify systematic failures
• Uncertainty: Confidence intervals (bootstrap where appropriate), variance across folds, and stability checks
• Evaluation report: 8-section report (objective, data, features, models, metrics, slices, risks, recommendation)
• Model card: Documentation for stakeholders (intended use, data, performance, ethics, operations)

Detailed guide: Evaluation Patterns

---

Pattern 6: Reproducibility & MLOps

Use when: Ensuring experiments are reproducible and production-ready.

Modern MLOps (CI/CD/CT/CM):

• CI (Continuous Integration): Automated testing, data validation, code quality
• CD (Continuous Delivery): Environment-specific promotion (dev -> staging -> prod), canary deployment
• CT (Continuous Training): Drift-triggered and scheduled retraining
• CM (Continuous Monitoring): Real-time data drift, performance, system health

Versioning:

• Code (git commit), data (DVC, LakeFS), features (feature store), models (MLflow Registry)
• Seeds (reproducibility), hyperparameters (experiment tracker)

Detailed guide: Reproducibility Checklist

---

Pattern 7: Feature Freshness & Streaming

Use when: Managing real-time features and streaming pipelines.

Components:

• Freshness contracts: Define freshness SLAs per feature, monitor lag, alert on breaches
• Batch + stream parity: Same feature logic across batch/stream, idempotent upserts
• Schema evolution: Version schemas, add forward/backward-compatible parsers, backfill with rollback
• Data quality gates: PII/format checks, range checks, distribution drift (KL, KS, PSI)

Detailed guide: Feature Freshness & Streaming

---

Pattern 8: Production Feedback Loops

Use when: Capturing production signals and implementing continuous improvement.

Components:

• Signal capture: Log predictions + user edits/acceptance/abandonment (scrub PII)
• Labeling: Route failures/edge cases to human review, create balanced sets
• Dataset refresh: Periodic refresh (weekly/monthly) with lineage, protect eval set
• Online eval: Shadow/canary new models, track solve rate, calibration, cost, latency

Detailed guide: Production Feedback Loops

---

Resources (Detailed Guides)

For comprehensive operational patterns and checklists, see:

• EDA Best Practices - Structured workflow for exploratory data analysis
• Feature Engineering Patterns - Operational patterns by data type
• Data Contracts & Lineage - Data quality, versioning, feature store ops
• Modelling Patterns - Model selection, hyperparameter tuning, train/test splits
• Evaluation Patterns - Metrics, slice analysis, evaluation reports, model cards
• Reproducibility Checklist - Experiment tracking, MLOps (CI/CD/CT/CM)
• Feature Freshness & Streaming - Real-time features, schema evolution
• Production Feedback Loops - Online learning, labeling, canary deployment
• Class Imbalance Patterns - Resampling, cost-sensitive learning, threshold tuning, evaluation for skewed datasets
• Hyperparameter Optimization - Bayesian optimization, early stopping, search strategies, budget allocation
• Interpretability & Explainability - SHAP, LIME, feature importance, model cards for regulated domains

---

Templates

Use these as copy-paste starting points:

Project & Workflow Templates

• Standard DS project template: assets/project/template-standard.md
• Quick DS experiment template: assets/project/template-quick.md

Feature Engineering & EDA

• Feature engineering template: assets/features/template-feature-engineering.md
• EDA checklist & notebook template: assets/eda/template-eda.md

Evaluation & Reporting

• Model evaluation report: assets/evaluation/template-evaluation-report.md
• Model card: assets/evaluation/template-model-card.md
• ML experiment review: assets/review/experiment-review-template.md

SQL Transformation (SQLMesh)

For SQL-based data transformation and feature engineering:

• SQLMesh project setup: ../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-project.md
• SQLMesh model types: ../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-model.md (FULL, INCREMENTAL, VIEW)
• Incremental models: ../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-incremental.md
• DAG and dependencies: ../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-dag.md
• Testing and data quality: ../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-testing.md

Use SQLMesh when:

• Building SQL-based feature pipelines
• Managing incremental data transformations
• Creating staging/intermediate/marts layers
• Testing SQL logic with unit tests and audits

For data ingestion (loading raw data), use:

• ai-mlops skill (dlt templates for REST APIs, databases, warehouses)

Navigation

Resources

• references/reproducibility-checklist.md
• references/evaluation-patterns.md
• references/feature-engineering-patterns.md
• references/modelling-patterns.md
• references/feature-freshness-streaming.md
• references/eda-best-practices.md
• references/data-contracts-lineage.md
• references/production-feedback-loops.md
• references/class-imbalance-patterns.md
• references/hyperparameter-optimization.md
• references/interpretability-explainability.md

Templates

• assets/project/template-standard.md
• assets/project/template-quick.md
• assets/features/template-feature-engineering.md
• assets/eda/template-eda.md
• assets/evaluation/template-evaluation-report.md
• assets/evaluation/template-model-card.md
• assets/review/experiment-review-template.md
• template-sqlmesh-project.md
• template-sqlmesh-model.md
• template-sqlmesh-incremental.md
• template-sqlmesh-dag.md
• template-sqlmesh-testing.md

Data

• data/sources.json - Curated external references

---

External Resources

See data/sources.json for curated foundational and implementation references:

• Core ML/DL: scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, JAX
• Data processing: pandas, NumPy, Polars, DuckDB, Spark, Dask
• SQL transformation: SQLMesh, dbt (staging/marts/incremental patterns)
• Feature stores: Feast, Tecton, Databricks Feature Store (centralized feature management)
• Data validation: Pydantic, Great Expectations, Pandera, Evidently (quality + drift)
• Visualization: Matplotlib, Seaborn, Plotly, Streamlit, Dash
• MLOps: MLflow, W&B, DVC, Neptune (experiment tracking + model registry)
• Hyperparameter tuning: Optuna, Ray Tune, Hyperopt
• Model serving: BentoML, FastAPI, TorchServe, Seldon, Ray Serve
• Orchestration: Kubeflow, Metaflow, Prefect, Airflow, ZenML
• Cloud platforms: AWS SageMaker, Google Vertex AI, Azure ML, Databricks, Snowflake

Use this skill to execute data science projects end-to-end: concrete checklists, patterns, and templates, not theory.

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

EDA Template

A reusable structure for conducting data exploration consistently across projects.

---

1. Dataset Summary

• Shape
• Columns
• Dtypes
• Unique key check
• Memory usage

---

2. Missingness Analysis

---

3. Outlier Analysis

• Outlier definition: <method>
• Detected outliers: <summary>
• Treatment plan: <method>

---

4. Distribution Analysis

Numeric

• Histograms
• Boxplots
• Quantiles

Categorical

• Category counts
• Rare category scan

---

5. Target Variable Analysis

• Distribution
• Class imbalance
• Skewness
• Baseline performance

---

6. Leakage Detection

• Time-based checks
• ID leakage patterns
• Text leakage risks
• Upstream data sanity checks

---

7. Feature Relationships

• Correlation heatmap
• Pairplot
• Target vs feature plots
• Interaction hypotheses

---

8. EDA Deliverables

• Summary of findings
• Risks
• Proposed next steps

Model Evaluation Report

A standardized, production-ready evaluation report.

---

1. Executive Summary

<Short description of model behavior and results.>

---

2. Problem Definition

• Business context
• Success metrics
• Constraints

---

3. Data Description

Known Data Risks:

• <risk 1>
• <risk 2>

---

4. Feature Engineering Summary

• Key numeric features
• Categorical encodings
• Text features
• Datetime features
• Leakage prevention measures

---

5. Model Experiments

---

6. Slice Analysis

<Include tables/plots of performance by slice.>

---

7. Error Analysis

• Systematic error patterns
• Representative failure cases

---

8. Final Recommendation

• <Deploy / Iterate / Reject>
• Justification

---

9. Risks & Mitigations

• <risk> -> <mitigation>

---

10. Appendix

• Hyperparameters
• Seeds
• Environment info

Model Card

A concise, production-grade model card for handoff and governance.

---

Model Overview

• Model name
• Version
• Date
• Owners
• Intended use
• Out-of-scope uses

---

Data Summary

• Sources
• Date ranges
• Known biases
• Data limitations

---

Performance Summary

Slice performance:

• <slice>: <metric>
• <slice>: <metric>

---

Safety & Ethical Considerations

• Sensitive attributes
• Bias risks
• Failure cases

---

Operational Details

• Input schema
• Output schema
• Expected latency
• Training environment
• Hardware requirements

---

Maintenance Plan

• Monitoring strategy
• Retraining cadence
• Ownership and support

Feature Engineering Template

Use this document to define, track, and validate all engineered features.

---

1. Overview

Target Variable: <describe>

Feature Set Version: <vX.Y>

---

2. Raw -> Engineered Feature Mapping

---

3. Numeric Features

Scaling:

• <method>

Outlier Handling:

• <method>

Transformations:

• log(x)
• sqrt(x)
• binning

---

4. Categorical Features

Encoding Types:

• One-hot
• Frequency
• Target (with CV)
• Hashing

Rules:

• Handle rare categories
• Map unseen categories

---

5. Text Features

Preprocessing:

• Lowercase
• Strip HTML
• Remove punctuation

Representations:

• TF-IDF
• Pretrained embeddings

---

6. Datetime Features

• Day of week
• Hour of day
• Weekend flag
• Holiday flag

Leakage Checks:

• No use of future information

---

7. Final Feature List

---

8. Validation Checklist

• [ ] Deterministic transformations
• [ ] Leakage reviewed
• [ ] Train/serve parity ensured
• [ ] Versioned in registry

Quick DS Workflow Template

A lightweight structure for rapid iterations, proofs of concept, or hackathon-speed modeling.

---

Objective

<Define the problem in 2-3 sentences.>

Data

• Datasets used
• Rows/columns
• Brief quality issues

EDA Notes

• Top trends
• Missingness summary
• Leakage risks

Features

• Key engineered features
• Encoders used
• Transformations

Models Tried

• Baseline: <metric>
• Candidate models: <models>

Best Model

• Model: <name>
• Metric: <value>
• Notes: <limitations>

Next Steps

• Add features
• Add slices
• Improve validation
• Try alternative models

Standard Data Science Project Template

This template structures a complete DS/ML project with clear deliverables and reproducible steps.

---

1. Project Overview

Objective: <Describe the business problem, target outcome, and decision impact.>

Success Criteria:

• Primary metric: <metric>
• Guardrail metrics: <metric(s)>

Constraints:

• Latency: <ms>
• Compute: <limits>
• Data availability: <summary>

---

2. Data Summary

Datasets Used:

Data Risks:

• <risk 1>
• <risk 2>

---

3. EDA Summary

• Top findings
• Missingness overview
• Outlier patterns
• Leakage risks
• Target distribution notes

---

4. Feature Engineering Plan

Numeric Features:

• Scaling: <method>
• Outlier handling: <method>

Categorical Features:

• Encoding method: <one-hot/target/frequency>

Text Features:

• Representation: <tfidf/embeddings>

Datetime Features:

• Extracted: <list>
• Timezone: <details>

---

5. Modelling Plan

Baseline Models:

• <baseline>

Candidate Models:

• <models>

Validation Strategy:

• Split type: <temporal/group/random>
• Test set size: <size>

---

6. Evaluation Plan

Primary Metric: <metric> Guardrails:

• <list>

Slice Evaluation:

• <dimensions>

---

7. Deliverables

• EDA notebook
• Feature engineering pipeline
• Training pipeline
• Evaluation report
• Model card
• Production handoff package

---

8. Risks & Mitigations

---

9. Project Timeline

ML Experiment Review Template

Purpose: Ensure reproducibility, validate methodology, document decisions for future reference.

---

Template Contract

Goals

• Validate methodology, prevent leakage, and document decisions.
• Make results reproducible and interpretable for stakeholders.

Inputs

• Problem statement + success criteria.
• Dataset version(s), split strategy, and feature definitions.
• Experiment config (code commit, environment, seeds).

Decisions

• Baseline and final model selection, metric thresholds, and deployment recommendation.
• Follow-up actions for failure modes and slices.

Risks

• Leakage, metric gaming, overfitting narratives, and non-reproducible runs.
• Miscommunication of uncertainty and limitations.

Metrics

• Primary/secondary metrics with confidence intervals.
• Slice performance and calibration (where applicable).

1. Experiment Metadata

experiment_id: ""
created: "YYYY-MM-DD"
author: ""
hypothesis: ""
status: "planning | running | completed | abandoned"
repository: ""
commit_hash: ""

---

2. Problem Definition

Business Context

• Problem statement: _______________
• Success criteria: _______________
• Stakeholder: _______________
• Timeline: _______________

ML Framing

• Task type: [ ] Classification [ ] Regression [ ] Ranking [ ] Clustering [ ] Other
• Target variable: _______________
• Prediction horizon: _______________
• Baseline to beat: _______________

---

3. Data Review

Dataset Summary

Leakage Check (CRITICAL)

Data Quality

Data Contract

• [ ] Schema documented
• [ ] Expected distributions documented
• [ ] Freshness requirements defined
• [ ] Data source reliability assessed

---

4. Feature Engineering

Feature Summary

Feature Validation

Feature Importance (Top 10)

---

5. Modeling

Baseline

• Model: _______________
• Metric: _______________
• Score: _______________

Models Evaluated

Final Model

• Selected model: _______________
• Key hyperparameters: _______________
• Selection rationale: _______________

Hyperparameter Tuning

• Method: [ ] Grid [ ] Random [ ] Bayesian [ ] Manual
• Search space: _______________
• Best parameters: _______________
• Tuning logged in: _______________

---

6. Evaluation

Cross-Validation

Test Set Results

Statistical Significance

• Test vs Baseline: p-value = ___
• Confidence level: ___%
• Effect size: ___

Sliced Analysis

Calibration (if probabilities)

• [ ] Calibration plot reviewed
• [ ] Brier score: ___
• [ ] Calibration method applied: _______________

---

7. Uncertainty Communication

Point Estimate

• Metric: _______________
• Value: _______________

Confidence Interval

• 95% CI: [___, ___]
• Method: [ ] Bootstrap [ ] Analytical [ ] Other

Known Limitations

1. _______________ 2. _______________ 3. _______________

Failure Modes

1. _______________ 2. _______________

---

8. Reproducibility

Environment

• Python version: ___
• Key packages: _______________
• Requirements file: _______________
• Random seed: ___

Artifacts

• [ ] Code committed: [commit hash]
• [ ] Data version: _______________
• [ ] Model artifact saved: _______________
• [ ] Metrics logged: _______________

Re-run Instructions

# Commands to reproduce

---

9. Decision

Recommendation

• [ ] Deploy: Results meet threshold, proceed to production
• [ ] Iterate: Promising but needs improvement on _______________
• [ ] Abandon: Does not beat baseline meaningfully
• [ ] Pivot: Reframe problem as _______________

Justification

_______________

Next Steps

1. _______________ 2. _______________ 3. _______________

---

10. Anti-Patterns Check

---

11. Sign-Off

Class Imbalance Patterns

Operational guide for handling imbalanced datasets in classification tasks. Covers sampling strategies, loss reweighting, threshold tuning, and evaluation metrics that actually reflect minority-class performance.

Freshness anchor: January 2026 — imbalanced-learn 0.12+, scikit-learn 1.5+, LightGBM 4.x

---

Decision Tree: Choosing an Imbalance Strategy

START
│
├─ Imbalance ratio < 5:1?
│   ├─ YES → Class weights usually sufficient
│   │         └─ Try `class_weight='balanced'` first
│   └─ NO → Continue
│
├─ Imbalance ratio 5:1 – 50:1?
│   ├─ Dataset > 50k rows?
│   │   ├─ YES → Undersampling + ensemble (EasyEnsemble, BalancedRF)
│   │   └─ NO  → SMOTE or ADASYN oversampling
│   └─ Tree-based model?
│       ├─ YES → `scale_pos_weight` or `is_unbalance` first
│       └─ NO  → Sampling + class weights combined
│
├─ Imbalance ratio > 50:1?
│   ├─ Anomaly detection framing viable?
│   │   ├─ YES → Switch to One-Class SVM / Isolation Forest
│   │   └─ NO  → Hybrid sampling + cost-sensitive learning
│   └─ Sufficient minority samples (>500)?
│       ├─ YES → SMOTE + Tomek links cleanup
│       └─ NO  → Data collection > algorithmic tricks
│
└─ Always: tune decision threshold via PR curve, not default 0.5

---

Quick Reference: Sampling Methods

---

Operational Patterns

Pattern 1: Class Weights (Simplest First)

• Use when: Imbalance ratio < 10:1, tree-based or linear model
• Implementation:

# scikit-learn
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight='balanced', n_estimators=300)

# LightGBM — two options
params_a = {'is_unbalance': True}  # auto-computes weight
params_b = {'scale_pos_weight': neg_count / pos_count}  # manual

# XGBoost
params_xgb = {'scale_pos_weight': neg_count / pos_count}

• Validation: compare PR-AUC with and without weights
• Gotcha: class_weight='balanced' uses n_samples / (n_classes * class_counts) — verify the math matches your expectations

Pattern 2: SMOTE Oversampling

• Use when: Dataset 1k–50k rows, continuous features, ratio 5:1–100:1
• Implementation:

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

pipeline = ImbPipeline([
    ('smote', SMOTE(
        sampling_strategy=0.5,  # target minority:majority ratio
        k_neighbors=5,          # lower for very small minorities
        random_state=42
    )),
    ('clf', RandomForestClassifier(n_estimators=300))
])

# CRITICAL: SMOTE inside CV, never before split
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipeline, X, y, cv=5, scoring='average_precision')

• Key rule: NEVER apply SMOTE before train/test split — causes data leakage
• k_neighbors tuning: if minority class < 20 samples, set k_neighbors=3 or lower

Pattern 3: Undersampling with Ensembles

• Use when: Large dataset (>50k rows), need fast training
• Implementation:

from imblearn.ensemble import BalancedRandomForestClassifier
from imblearn.ensemble import EasyEnsembleClassifier

# Option A: Balanced Random Forest
brf = BalancedRandomForestClassifier(
    n_estimators=300,
    sampling_strategy='all',
    replacement=False,
    random_state=42
)

# Option B: EasyEnsemble (AdaBoost on balanced subsets)
ee = EasyEnsembleClassifier(
    n_estimators=20,
    random_state=42
)

• Advantage: retains all minority samples, subsamples majority per tree
• Gotcha: BalancedRF can be slower than regular RF due to resampling overhead

Pattern 4: Threshold Tuning via PR Curve

• Use when: Always — default 0.5 threshold is almost never optimal for imbalanced data
• Implementation:

from sklearn.metrics import precision_recall_curve
import numpy as np

y_proba = clf.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

# F1-optimal threshold
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-8)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]

# F-beta for recall-heavy use cases (e.g., fraud)
beta = 2
fbeta = (1 + beta**2) * (precision * recall) / (beta**2 * precision + recall + 1e-8)
best_threshold_fbeta = thresholds[np.argmax(fbeta)]

• Business alignment: choose beta based on cost of FN vs FP
• beta=2 — missing positives is 4x worse than false alarms (fraud, medical)
• beta=0.5 — false alarms are 4x worse than misses (spam, content moderation)

Pattern 5: Cost-Sensitive Learning

• Use when: Business has explicit cost matrix (cost of FN != cost of FP)
• Implementation:

# Custom sample weights reflecting business cost
sample_weights = np.where(y_train == 1, cost_fn, cost_fp)
clf.fit(X_train, y_train, sample_weight=sample_weights)

# For LightGBM: per-instance weighting
train_data = lgb.Dataset(X_train, label=y_train, weight=sample_weights)

• Cost matrix example:

• Weight ratio: cost_fn / cost_fp = 50 → use as scale_pos_weight

Pattern 6: Hybrid Sampling

• Use when: Ratio > 20:1, need clean decision boundaries
• Implementation:

from imblearn.combine import SMOTETomek, SMOTEENN

# SMOTE + Tomek (moderate cleanup)
smt = SMOTETomek(
    smote=SMOTE(sampling_strategy=0.5, k_neighbors=5),
    random_state=42
)

# SMOTE + ENN (aggressive cleanup — better boundaries, fewer samples)
smenn = SMOTEENN(
    smote=SMOTE(sampling_strategy=0.5, k_neighbors=5),
    random_state=42
)

---

Evaluation Metrics for Imbalanced Data

Metrics Decision Table

Metric Implementation

from sklearn.metrics import (
    average_precision_score,  # PR-AUC
    f1_score,
    fbeta_score,
    matthews_corrcoef,        # MCC
    classification_report
)

y_proba = clf.predict_proba(X_test)[:, 1]
y_pred = (y_proba >= best_threshold).astype(int)

metrics = {
    'PR-AUC': average_precision_score(y_test, y_proba),
    'F1': f1_score(y_test, y_pred),
    'F2': fbeta_score(y_test, y_pred, beta=2),
    'MCC': matthews_corrcoef(y_test, y_pred),
}

---

Anti-Patterns

---

Validation Checklist

• [ ] Imbalance ratio measured and documented
• [ ] Baseline established with no correction (to measure improvement)
• [ ] Sampling applied ONLY inside cross-validation folds
• [ ] Test set left untouched (original distribution)
• [ ] PR-AUC or F-beta used as primary metric (NOT accuracy or ROC-AUC)
• [ ] Decision threshold tuned on validation set, evaluated on test set
• [ ] Confusion matrix reviewed at chosen threshold
• [ ] Business cost alignment verified (FN cost vs FP cost)
• [ ] Stratified splits used (StratifiedKFold)
• [ ] Results stable across multiple random seeds

---

Cross-References

• ai-ml-data-science/references/hyperparameter-optimization.md — tuning scale_pos_weight and sampling params
• ai-ml-data-science/references/interpretability-explainability.md — explaining minority-class predictions
• ai-mlops/references/automated-retraining-patterns.md — monitoring class distribution drift
• ai-mlops/references/experiment-tracking-patterns.md — logging imbalance metrics per run

Data Contracts, Lineage & Feature Store Operations

Operational patterns for managing data contracts, lineage tracking, and feature store hygiene in production ML systems.

---

Overview

Data contracts and lineage tracking are critical for production ML systems. They ensure data quality, enable debugging, and maintain train-serve consistency. This guide covers modern best practices for feature store operations and data governance.

---

1. Data Contracts

1.1 Contract Components

A robust data contract defines:

• Schema: Column names, types, nullability constraints
• Ranges & Constraints: Min/max values, allowed categorical values, regex patterns
• Freshness SLAs: Maximum acceptable data lag
• Versioning: Contract version number and compatibility rules

1.2 Contract Enforcement

When to check contracts:

• At data ingestion (source -> feature store)
• Before training (feature store -> training pipeline)
• At serving time (feature store -> production model)

Enforcement strategy:

• Fail fast: Block pipeline on critical contract violations
• Warn: Log non-critical violations but continue
• Degrade gracefully: Use fallback values for optional fields

1.3 Schema Evolution

Backward-compatible changes (safe):

• Adding optional fields
• Relaxing constraints (e.g., widening ranges)
• Adding new enum values

Breaking changes (require coordination):

• Removing fields
• Changing data types
• Renaming columns
• Tightening constraints

Migration strategy: 1. Version the contract (v1 -> v2) 2. Run shadow mode (dual write to v1 and v2) 3. Validate v2 data quality matches v1 4. Gradual cutover with rollback plan 5. Deprecate v1 after validation period

Checklist: Schema Evolution

• [ ] Contract version incremented
• [ ] Backward/forward compatibility assessed
• [ ] Shadow run completed successfully
• [ ] Rollback artifacts preserved
• [ ] Deprecation timeline communicated

---

2. Data Lineage Tracking

2.1 What to Track

Essential lineage metadata:

• Source system and extraction timestamp
• Feature store write timestamp and version
• Training run ID and model version
• Feature transformation code version (git commit)
• Data quality metrics at each stage

Why it matters:

• Debug data quality issues
• Audit compliance (GDPR, SOC2)
• Root cause analysis for model degradation
• Reproduce experiments

2.2 Lineage Implementation Patterns

Storage:

• Structured logs (JSON lines)
• Metadata stores (MLflow, DVC, Feast)
• Graph databases (Neo4j for complex lineage)

Tagging convention:

lineage_metadata = {
    "source": "postgres://db/table",
    "extraction_ts": "2024-11-22T10:00:00Z",
    "feature_store_version": "v2.1",
    "git_commit": "a1b2c3d4",
    "run_id": "train-20241122-001",
    "model_version": "v1.5.2",
    "feature_set_hash": "sha256:abcd1234..."
}

Checklist: Lineage Implemented

• [ ] Source -> feature store -> train -> serve path tracked
• [ ] Run IDs and model versions logged
• [ ] Git commits captured for reproducibility
• [ ] Lineage queryable (e.g., "which training runs used this data version?")
• [ ] Retention policy defined (how long to keep lineage)

---

3. Feature Store Hygiene

3.1 Materialization Cadence

Document and enforce:

• Batch update frequency (hourly, daily, weekly)
• Streaming update latency targets
• Backfill procedures for historical data

Monitoring:

• Freshness lag (source timestamp -> feature store timestamp)
• Materialization job success rate
• Data volume anomalies

3.2 Backfill & Replay

Requirements for safe backfill:

• Idempotent writes (duplicate runs produce same result)
• Timestamp-based partitioning
• Preserved input data snapshots
• Validation against production data

Replay scenarios:

• Bug fix in feature transformation logic
• Schema migration
• Historical model training
• Audit requirements

Checklist: Backfill Ready

• [ ] Backfill procedure documented and tested
• [ ] Idempotency verified (run twice -> same output)
• [ ] Validation metrics defined (replay vs original)
• [ ] Impact assessment for downstream consumers

3.3 Encoder & Mapping Versioning

What to version:

• Categorical encoders (target, frequency, hash)
• Normalization parameters (mean, std, min, max)
• Embedding models and weights
• Lookup tables and dictionaries

Storage strategy:

• Store alongside model artifacts
• Use feature store's versioning system
• Tag with training run ID
• Keep rollback versions

Checklist: Encoders Versioned

• [ ] All transformations serialized
• [ ] Encoder versions match training config
• [ ] Unseen category handling defined
• [ ] Serving uses same encoder version as training

---

4. Train-Serve Parity

4.1 Common Parity Issues

Sources of divergence:

• Different feature computation logic (Python vs SQL)
• Timezone mismatches
• Rounding/precision differences
• Async updates (training uses stale data)

Detection:

• Shadow mode: run both pipelines, compare features
• Synthetic tests with known inputs
• Production monitoring: track distribution drift

4.2 Ensuring Parity

Best practices:

• Single source of truth: shared feature transformation code
• Use feature store for both training and serving
• Integration tests: compare training vs serving features
• Monitor drift between training and production feature distributions

Checklist: Parity Validated

• [ ] Shared transformation code between train/serve
• [ ] Feature store used for both pipelines
• [ ] Integration tests pass (feature equality within tolerance)
• [ ] Drift monitoring active (KL divergence, PSI)
• [ ] Serving feature distributions match training

---

5. Monitoring & Alerts

5.1 Key Metrics

Data quality:

• Null rate per feature
• Out-of-range values
• Distribution shift (KL divergence, KS test, PSI)

Freshness:

• Data lag (source -> feature store)
• Staleness alerts (SLA violations)

Operational:

• Materialization job failures
• Query latency (p50, p99)
• Storage costs

5.2 Alert Strategy

Critical (page on-call):

• Contract violation blocking production
• Freshness SLA breach > 2x threshold
• Materialization job failures

Warning (Slack/email):

• Non-critical contract violations
• Minor distribution drift
• Performance degradation

Checklist: Monitoring Active

• [ ] Freshness SLAs defined and monitored
• [ ] Distribution drift alerts configured
• [ ] Contract violation alerts active
• [ ] Runbooks documented for common alerts
• [ ] False positive rate < 5%

---

6. Governance & Compliance

6.1 PII Handling

Requirements:

• PII identified and tagged in metadata
• Access controls enforced (RBAC)
• Audit logs for PII access
• Hard delete capability (GDPR right to erasure)

6.2 Data Residency

Multi-region strategy:

• Segment feature stores by region
• Enforce data sovereignty rules
• Replicate non-sensitive features
• Document cross-border data flows

Checklist: Governance Ready

• [ ] PII fields identified and tagged
• [ ] Access controls implemented
• [ ] Hard delete procedure tested
• [ ] Data residency requirements documented
• [ ] Audit trail enabled

---

Related Resources

• Feature Engineering Patterns - Feature transformation techniques
• Reproducibility Checklist - Experiment tracking and versioning
• Feature Freshness & Streaming - Real-time feature updates
• Production Feedback Loops - Online learning and model updates

EDA Best Practices

This guide provides a structured, repeatable workflow for exploratory data analysis with explicit checks, patterns, and decision rules. It is designed for fast onboarding and consistent DS project execution.

---

1. Initial Scan Checklist

Perform immediately after loading the dataset.

• [ ] Print shape (rows, columns)
• [ ] Inspect dtypes and nullable fields
• [ ] Identify primary keys or unique identifier candidates
• [ ] Check for duplicate rows and duplicate keys
• [ ] Evaluate memory usage
• [ ] Validate expected ranges for numeric columns
• [ ] Confirm presence/absence of target variable

Pattern: Schema Validation df.info() df.describe(include='all') df.isna().sum()

---

2. Data Quality Assessment

2.1 Missingness

• Identify missingness patterns by:
• Row
• Column
• Groups (user, product, geography)
• Evaluate mechanisms:
• MCAR (random)
• MAR (depends on other features)
• MNAR (depends on itself; dangerous)

Checklist - Missingness Strategy

• [ ] Strategy per field documented
• [ ] No target leakage introduced by imputation
• [ ] Imputation pipelines reproducible

---

2.2 Outliers

Detection methods (choose at least one):

• Z-score
• IQR
• Winsorization scan
• Domain-rule scans (e.g., speed < 0 impossible)

Checklist - Outlier Review

• [ ] Extreme values inspected manually
• [ ] Outlier handling strategy defined (cap/remove/flag)
• [ ] Illegal values corrected or removed

---

3. Distribution Analysis

Perform both univariate and bivariate analysis.

Numeric

• Histograms
• Boxplots
• Quantile tables
• Skewness/kurtosis review

Categorical

• Frequency distributions
• Top-N categories report
• Rare category detection (<1% threshold)

Checklist - Distribution Health

• [ ] Long tails annotated
• [ ] Rare categories flagged
• [ ] Highly skewed features documented for potential transforms

---

4. Target Variable Analysis

Classification targets:

• Class imbalance
• Rare event frequency
• Conditional distributions

Regression targets:

• Scale and skew
• Outliers
• Zero-inflation

Checklist - Target Evaluation

• [ ] Imbalance noted
• [ ] Appropriate metric selection influenced (e.g., PR-AUC for imbalance)
• [ ] Target leakage checks started

---

5. Leakage Detection

Leakage is the leading cause of unrealistic performance.

High-Risk Leakage Types:

• Timestamps after event date
• IDs encoding target
• Aggregates computed using full window
• Target visible in free text
• Future features used in temporal splits

Checklist - Leakage Review

• [ ] Time-based checks performed
• [ ] ID/cardinality checks performed
• [ ] No future window features in train set
• [ ] Free text screened for target bleed

---

6. EDA Deliverables

A complete EDA must include:

• Profile report (summary tables + visualizations)
• Data dictionary draft
• Issue register (severity, owner, fix plan)
• List of known risks
• Candidate hypotheses

Evaluation Patterns

Operational guidance for evaluating ML models: metric selection, slice analysis, error analysis, evaluation reports, and model cards.

---

1. Metric & Threshold Selection

1.1 Primary Metrics

Classification:

• ROC-AUC: Area under ROC curve (good for balanced classes)
• PR-AUC: Precision-recall AUC (better for imbalanced classes)
• F1 / F-beta: Harmonic mean of precision and recall (beta > 1 favors recall, beta < 1 favors precision)
• Accuracy: Only for balanced data
• Log loss: Penalizes confident wrong predictions

Regression:

• MAE: Mean Absolute Error (robust to outliers)
• RMSE: Root Mean Squared Error (penalizes large errors)
• MAPE / sMAPE: Mean Absolute Percentage Error (scale-independent)
• R^2: Coefficient of determination (guardrail only, not for optimization)

Ranking:

• NDCG: Normalized Discounted Cumulative Gain
• MAP: Mean Average Precision
• Recall@K: Fraction of relevant items in top K

1.2 Guardrail Metrics

Beyond primary metric:

• Calibration: Do predicted probabilities match actual rates? (Brier score, calibration plots)
• Fairness: Performance parity across demographic groups
• Business constraints: Profit, cost, latency, model size
• Operational: Inference time, memory usage, explainability

1.3 Threshold Selection

Methods: 1. ROC / PR curves: Visualize trade-offs, pick operating point 2. F1 maximization: Find threshold that maximizes F1 score 3. Cost-sensitive: Assign costs to FP and FN, minimize expected cost 4. Business rule: E.g., "flag top 10% riskiest transactions"

Context-specific examples:

• Fraud detection: High recall (catch fraudsters), tolerate FP
• Spam filtering: High precision (don't block legitimate emails)
• Medical diagnosis: Balance based on cost of false negatives vs false positives

Checklist: Metrics & Thresholds

• [ ] Primary metric chosen and justified
• [ ] Guardrail metrics defined (calibration, fairness, constraints)
• [ ] Threshold selection documented (with trade-offs)
• [ ] Metric definitions and calculations reproducible
• [ ] Per-segment thresholds validated (if applicable)

---

2. Slice & Error Analysis

2.1 Slice Definition

Why slice:

• Overall metrics hide subgroup performance issues
• Fairness: ensure equitable performance
• Business impact: different segments have different value

Common slices:

• Geography: US, EU, APAC, country-level
• User segments: New vs returning, free vs paid, power users
• Product categories: Electronics, clothing, books
• Time periods: Weekday vs weekend, seasonality, recency
• Data characteristics: High vs low confidence, common vs rare events
• Demographics: Age groups, language (for fairness analysis)

2.2 Slice Metrics

Analysis: 1. Compute primary metric per slice 2. Identify slices with largest performance gaps vs overall 3. Rank slices by:

• Absolute performance (worst performers)
• Degradation from overall (largest gaps)
• Business impact (volume x performance gap)

Visualization:

• Bar charts: metric by slice
• Heatmaps: 2D slices (e.g., geography x product)
• Time series: metric over time per slice

2.3 Error Analysis

Steps:

1. Collect high-error examples

• Top 100 highest-loss predictions
• Misclassified examples (FP and FN separately)
• Low-confidence correct predictions

2. Cluster errors

• Manual review: identify patterns
• Automated: cluster by features, error magnitude
• Tag by error type (e.g., "confuses cats and dogs", "struggles with low light")

3. Identify systematic failures

• Missing features (e.g., "time of day not captured")
• Data quality issues (e.g., "corrupted images")
• Modeling gaps (e.g., "rare classes underrepresented")

4. Propose fixes

• New features
• Data augmentation
• Re-labeling
• Model architecture changes

Checklist: Slice & Error Analysis

• [ ] Key slices identified and evaluated
• [ ] Weak slices documented with hypotheses
• [ ] Example-level errors reviewed qualitatively (top 50-100)
• [ ] Systematic failure modes identified
• [ ] Candidate feature/model changes proposed
• [ ] Business impact of slice performance gaps quantified

---

3. Evaluation Report

3.1 Purpose

When to create:

• Finalizing a model candidate
• Handing over to production team
• Compliance or audit requirements
• Decision to deploy, iterate, or hold

3.2 Report Structure

Section 1: Objective & Context

• Business problem and success criteria
• Scope and out-of-scope
• Baseline performance

Section 2: Data Description & Limitations

• Data sources and time period
• Sample size (train, val, test)
• Known biases or coverage gaps
• PII handling and privacy considerations

Section 3: Feature Engineering Summary

• Key features and transformations
• Feature importance (top 10-20)
• Feature store integration (if applicable)

Section 4: Modeling Approaches Tried

• Baseline models
• Candidate models (with hyperparameters)
• Why final model was chosen

Section 5: Metrics (Primary & Guardrails)

• Primary metric on test set
• Guardrail metrics (calibration, fairness, latency)
• Statistical significance vs baseline

Section 6: Slice & Error Analysis

• Performance by key slices
• Worst-performing slices with hypotheses
• Example errors and failure modes

Section 7: Risks & Mitigations

• Known limitations
• Failure modes and edge cases
• Mitigation strategies (fallbacks, monitoring)

Section 8: Recommendation

• Deploy: Model ready for production
• Iterate: Needs improvement before deploy
• Hold: Not viable, explore alternatives

Checklist: Evaluation Report Complete

• [ ] Report includes enough detail for reviewer to follow decisions
• [ ] Limitations and risks explicitly listed
• [ ] Comparisons vs baselines included
• [ ] Monitoring and retraining expectations noted
• [ ] Recommendation justified with evidence

---

4. Model Card

4.1 Purpose

When to create:

• Documenting model for stakeholders
• Compliance requirements (EU AI Act, etc.)
• Internal governance and model registry
• External-facing models (API products)

4.2 Model Card Structure (Short Form)

Section 1: Model Overview

• What it does (task, inputs, outputs)
• For whom (intended users, use cases)
• Version and release date

Section 2: Intended Use and Non-Intended Use

• Intended: Approved use cases with examples
• Non-intended: Explicitly out-of-scope uses
• Misuse risks: Potential harmful applications

Section 3: Training Data

• Data sources and time period
• Sampling strategy
• Known biases (geographic, demographic, temporal)
• PII handling

Section 4: Performance Summary

• Primary metric on test set
• Performance by key slices
• Calibration and fairness metrics

Section 5: Ethical/Fairness Considerations

• Demographic parity or equalized odds
• Potential discriminatory impacts
• Mitigation strategies

Section 6: Operational Notes

• Input schema and expected format
• Output schema and interpretation
• Expected latency (p50, p95, p99)
• Dependencies (feature store, external APIs)

Section 7: Owners and Contacts

• Model owner / team
• Maintenance plan (retraining cadence)
• Contact for questions / issues

4.3 Model Card (Long Form)

For compliance-critical applications, extend with:

• Model architecture: Detailed description
• Hyperparameters: Full configuration
• Metrics: Complete evaluation results
• Sensitivity analysis: Impact of data distribution shifts
• Uncertainty quantification: Confidence intervals
• Environmental impact: CO2 emissions from training
• Versioning: Change log from previous versions

Checklist: Model Card Ready

• [ ] Intended/unsafe uses documented
• [ ] Performance by segment summarized
• [ ] Data limitations and biases acknowledged
• [ ] Owners and maintenance plan listed
• [ ] Compliance requirements met (if applicable)

---

5. Stability & Reproducibility Checks

5.1 Seed Stability

Why it matters:

• Ensure results not due to random luck
• Build confidence in model selection
• Detect overfitting to validation set

Procedure: 1. Train model with 5-10 different random seeds 2. Report mean +/- std for key metrics 3. Verify rankings stable (best model stays best)

Red flags:

• High variance in metrics (std > 5% of mean)
• Model rankings change with different seeds
• Single run significantly outperforms average

5.2 Cross-Validation Stability

When to use:

• Small datasets (< 10k samples)
• Need robust estimates
• Hyperparameter tuning

Procedure: 1. K-fold cross-validation (K = 5 or 10) 2. Report mean +/- std across folds 3. Check consistency of feature importance 4. Validate on final held-out test set

Checklist: Stability Validated

• [ ] Multiple seeds tested (5-10 runs)
• [ ] Variance of metrics acceptable (std < 5% of mean)
• [ ] Model rankings stable across seeds
• [ ] Cross-validation used for small datasets
• [ ] Final test set validates CV results

---

6. Statistical Significance Testing

6.1 When to Test

Use statistical tests when:

• Comparing two models with small performance difference
• Need confidence in improvement (e.g., for production deployment)
• Evaluating A/B test results

Skip when:

• Large performance differences (obvious winner)
• Exploratory phase (many models to try)
• Computationally expensive

6.2 Methods

Paired t-test:

• Use when: Comparing two models on same cross-validation folds
• Null hypothesis: No difference in mean performance
• Threshold: p < 0.05 (or p < 0.01 for stricter test)

Bootstrap confidence intervals:

• Use when: Want uncertainty estimates
• Procedure: Resample test set 1000 times, compute metric, report 95% CI
• Interpretation: If CIs don't overlap, difference is significant

Permutation test:

• Use when: Distribution assumptions unclear
• Procedure: Randomly permute labels, compute metric, compare to actual
• Threshold: p < 0.05

Checklist: Statistical Testing

• [ ] Test method chosen and justified
• [ ] Null hypothesis clearly stated
• [ ] p-value or confidence intervals reported
• [ ] Multiple testing correction applied (if comparing many models)
• [ ] Practical significance considered (not just statistical)

---

Related Resources

• Modelling Patterns - Model selection, hyperparameter tuning, train/test splits
• Reproducibility Checklist - Experiment tracking and versioning
• Production Feedback Loops - Online evaluation and A/B testing

Feature Engineering Patterns

A collection of operational patterns for transforming raw data into model-ready features.

---

1. Numeric Feature Patterns

1.1 Standardization

Use when units vary or model sensitive to scale.

• z-score
• min-max
• robust scaling (median/IQR)

1.2 Outlier Handling

• Winsorize top/bottom 1%
• Cap values at domain limits
• Log transform long-tailed distributions

Checklist - Numeric Features

• [ ] Consistent units
• [ ] Outliers handled
• [ ] Skew addressed

---

2. Categorical Feature Patterns

2.1 Low Cardinality

• One-hot encoding
• Ordinal encoding (only when true order exists)

2.2 High Cardinality

• Target encoding (use CV to avoid leakage)
• Frequency encoding
• Hashing

Checklist - Categorical Features

• [ ] Rare categories grouped/flagged
• [ ] Clear mapping for unseen categories
• [ ] Encoders versioned for training/serving parity

---

3. Text Feature Patterns

3.1 Cleaning

• Strip HTML
• Lowercase or case-preserve based on domain
• Remove excessive whitespace

3.2 Representations

• TF-IDF
• Pretrained embeddings
• Keyword densities
• Text length signals

Checklist - Text Features

• [ ] Deterministic preprocessing
• [ ] PII removed where required
• [ ] Embedding models versioned

---

4. Time-Based Features

4.1 Datetime Decomposition

• Year, month, day
• Day of week
• Hour, minute
• Boolean flags (weekend, holiday)

4.2 Lag Features

• lag_1, lag_7, lag_28
• Rolling windows

Checklist - Time Features

• [ ] Timezone alignment validated
• [ ] Features do not leak future information

---

5. Interaction Patterns

Use carefully to avoid explosion.

• Crossed categorical features
• Numeric x categorical interactions
• Polynomial features (2nd/3rd degree)

Checklist - Interaction Features

• [ ] Interaction justified
• [ ] No combinatorial blow-up
• [ ] Feature importance reviewed

---

6. Train/Serve Consistency Patterns

Ensuring parity between offline and production pipelines

• Use shared feature store when possible
• Encode with version-pinned transformers
• Enforce dtype consistency

Checklist - Consistency

• [ ] Single source of truth for transformations
• [ ] Serving pipeline tested with training artifacts

Feature Freshness, Streaming & Schema Evolution

Operational patterns for managing real-time features, streaming pipelines, and schema changes in production ML systems.

---

Overview

Modern ML systems increasingly require real-time features and streaming data pipelines. This guide covers best practices for maintaining freshness SLAs, ensuring batch-stream parity, and safely evolving schemas.

---

1. Freshness Contracts & SLAs

1.1 Defining Freshness Requirements

Questions to answer:

• What is the maximum acceptable lag between source update and feature availability?
• Are there different SLAs for different features?
• What happens when freshness SLA is violated?

Common SLA tiers:

• Real-time: < 1 minute lag (streaming features)
• Near real-time: 1-15 minutes lag (micro-batch)
• Hourly: < 1 hour lag (batch with frequent updates)
• Daily: < 24 hours lag (overnight batch jobs)

1.2 Freshness Monitoring

Metrics to track:

• Lag: current_time - source_timestamp
• Staleness: Time since last successful update
• Update frequency: Updates per hour/day

Alerting thresholds:

• Critical: Lag > 2x SLA threshold
• Warning: Lag > 1.5x SLA threshold
• Info: Lag approaching SLA threshold

Checklist: Freshness Monitoring

• [ ] Freshness SLAs defined per feature or feature group
• [ ] Lag metrics collected and dashboarded
• [ ] Alerts configured with appropriate thresholds
• [ ] Fallback strategy documented for stale data
• [ ] Historical lag trends analyzed (p50, p95, p99)

---

2. Batch + Stream Parity

2.1 The Parity Challenge

Problem:

• Batch pipelines use different code/frameworks than streaming
• Results can diverge due to:
• Rounding differences
• Aggregation window boundaries
• Late-arriving data handling
• Order-dependent operations

Solution:

• Use shared feature transformation logic
• Implement idempotent upserts
• Handle late-arriving data consistently
• Test parity with synthetic replay

2.2 Shared Feature Logic Patterns

Option 1: Feature store abstraction

# Same code for batch and stream
@feature_definition
def user_7day_spend(events):
    return events.filter(
        lambda e: e.timestamp > now() - timedelta(days=7)
    ).sum("amount")

Option 2: Shared libraries

# features/user_metrics.py (shared by Spark batch + Flink stream)
def compute_rolling_spend(events_df, window_days=7):
    # Deterministic logic works in both contexts
    return events_df.groupBy("user_id").agg(...)

Checklist: Batch-Stream Parity

• [ ] Feature logic shared between batch and streaming pipelines
• [ ] Idempotent upserts implemented (duplicate events handled)
• [ ] Late-arriving data strategy defined and consistent
• [ ] Parity tests: batch vs stream results match within tolerance
• [ ] Differences documented (if unavoidable due to windowing)

---

3. Schema Evolution Strategies

3.1 Compatible Changes

Backward compatible (safe):

• Adding new optional fields
• Widening numeric types (int32 -> int64)
• Relaxing constraints (nullable = true)

Forward compatible (safe):

• Removing optional fields (old code ignores)
• Adding default values for new fields

Incompatible (breaking):

• Renaming fields
• Changing types (string -> int)
• Removing required fields
• Changing semantics

3.2 Migration Process

Step 1: Version schemas

{
  "schema_version": "v2.1",
  "fields": [...],
  "compatible_with": ["v2.0"]
}

Step 2: Dual write/read

• Write to both old and new schema
• Read from new schema with fallback to old
• Validate consistency

Step 3: Backfill

• Run backfill jobs to populate new schema
• Use idempotent writes
• Validate against original data

Step 4: Cutover

• Monitor error rates
• Gradual rollout (1% -> 10% -> 50% -> 100%)
• Keep rollback artifacts

Checklist: Schema Evolution

• [ ] Schema versioned with compatibility metadata
• [ ] Dual-write phase completed successfully
• [ ] Backfill job run and validated
• [ ] Rollback plan documented and tested
• [ ] Deprecated schemas marked with sunset date

---

4. Data Quality Gates

4.1 Quality Checks

PII/Format checks:

• Regex validation (email, phone, SSN patterns)
• PII detection and redaction
• Encoding validation (UTF-8)

Range checks:

• Min/max bounds per feature
• Enum membership checks
• Cross-field constraints (end_date > start_date)

Distribution checks:

• KL divergence vs training distribution
• Kolmogorov-Smirnov test
• Population Stability Index (PSI)

4.2 Gate Enforcement

At ingestion:

• Block writes on critical violations
• Log warnings on minor violations
• Quarantine invalid records

At training:

• Fail pipeline on distribution drift > threshold
• Require manual approval for major changes

At serving:

• Reject invalid requests
• Apply fallback values
• Log and alert

Checklist: Quality Gates Active

• [ ] PII detection rules configured
• [ ] Range checks defined per feature
• [ ] Distribution drift thresholds set
• [ ] Gates active in CI/CD pipeline
• [ ] Gates active in production serving
• [ ] Quarantine process for invalid data

---

5. Late-Arriving Data Handling

5.1 Patterns

Pattern 1: Allowed lateness window

• Accept events up to N hours late
• Drop events beyond window
• Trade-off: completeness vs complexity

Pattern 2: Watermarks

• Track event time watermark
• Trigger computations when watermark advances
• Handle stragglers with side outputs

Pattern 3: Reprocessing

• Periodically recompute features with full data
• Upsert corrected values
• Idempotent operations required

5.2 Implementation

Flink watermark example:

env.fromSource(source)
   .assignTimestampsAndWatermarks(
       WatermarkStrategy
           .<Event>forBoundedOutOfOrderness(Duration.ofMinutes(5))
           .withTimestampAssigner((event, ts) -> event.timestamp)
   )

Checklist: Late Data Handling

• [ ] Allowed lateness window defined
• [ ] Watermark strategy configured
• [ ] Reprocessing cadence determined
• [ ] Idempotency verified for reprocessing
• [ ] Metrics track late arrival rates

---

6. Streaming Architecture Patterns

6.1 Lambda Architecture

Components:

• Batch layer: Complete, accurate, slow
• Speed layer: Approximate, fast, handles recent data
• Serving layer: Merges batch + speed results

When to use:

• Need both accuracy (batch) and low latency (streaming)
• Can tolerate eventual consistency

6.2 Kappa Architecture

Components:

• Single streaming pipeline for all data
• Reprocessing = replay from log (Kafka)

When to use:

• Can express all logic in streaming framework
• Want to avoid maintaining two pipelines

Checklist: Architecture Selection

• [ ] Latency requirements documented
• [ ] Accuracy vs speed trade-offs evaluated
• [ ] Replay/reprocessing needs assessed
• [ ] Framework capabilities validated (Flink, Spark Streaming, Kafka Streams)
• [ ] Operational complexity considered

---

7. Testing Strategies

7.1 Unit Tests

Test feature transformations:

def test_rolling_spend_deterministic():
    events = create_test_events()
    result_batch = compute_rolling_spend_batch(events)
    result_stream = compute_rolling_spend_stream(events)
    assert result_batch == result_stream

7.2 Integration Tests

Test end-to-end pipeline:

• Inject synthetic events
• Wait for processing
• Verify output matches expected

7.3 Chaos Testing

Simulate failures:

• Late-arriving data
• Out-of-order events
• Duplicate events
• Network partitions
• Service restarts

Checklist: Testing Complete

• [ ] Unit tests for feature transformations (batch-stream parity)
• [ ] Integration tests for full pipeline
• [ ] Chaos tests for failure scenarios
• [ ] Performance tests (throughput, latency)
• [ ] Regression tests for schema changes

---

8. Operational Runbooks

8.1 Common Issues

Issue: Features are stale

• Check: Streaming job running?
• Check: Source producing events?
• Check: Network connectivity?
• Mitigation: Restart job, trigger backfill

Issue: Batch-stream parity violation

• Check: Transformation code versions match?
• Check: Late-arriving data handled?
• Mitigation: Align code, replay stream

Issue: Schema mismatch errors

• Check: Producers using latest schema?
• Check: Consumers handle old schema?
• Mitigation: Dual-write mode, schema registry

Checklist: Runbooks Ready

• [ ] Incident response procedures documented
• [ ] Common failure modes catalogued
• [ ] Escalation paths defined
• [ ] Rollback procedures tested
• [ ] On-call training completed

---

Related Resources

• Data Contracts & Lineage - Schema versioning and lineage tracking
• Reproducibility Checklist - Experiment versioning
• Production Feedback Loops - Online learning patterns

Hyperparameter Optimization

Operational guide for systematic hyperparameter tuning using Optuna, Ray Tune, and Bayesian optimization. Covers search space design, pruning, multi-objective optimization, and reproducible tuning recipes for common model families.

Freshness anchor: January 2026 — Optuna 3.6+, Ray Tune 2.9+, scikit-learn 1.5+, LightGBM 4.x

---

Decision Tree: Choosing an Optimization Strategy

START
│
├─ < 10 hyperparameters?
│   ├─ YES → Optuna with TPE sampler (default)
│   └─ NO  → Continue
│
├─ 10–30 hyperparameters?
│   ├─ Training time < 5 min per trial?
│   │   ├─ YES → Optuna TPE, 100–300 trials
│   │   └─ NO  → Optuna with pruning (MedianPruner)
│   └─ Distributed cluster available?
│       ├─ YES → Ray Tune + Optuna integration
│       └─ NO  → Optuna with SQLite storage for resumability
│
├─ Multi-objective (e.g., accuracy + latency)?
│   └─ Optuna with NSGAIISampler → Pareto front
│
├─ Need warmstarting from prior runs?
│   └─ Optuna with enqueue_trial for known-good configs
│
└─ Very expensive trials (>1 hour each)?
    └─ Bayesian optimization (GP) with <50 trials
        OR early stopping with aggressive pruning

---

Quick Reference: Sampler Selection

---

Operational Patterns

Pattern 1: Optuna Basic Setup

• Use when: Starting any tuning task
• Implementation:

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 100, 1000, step=100),
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
    }

    # Cross-validation inside objective
    scores = cross_val_score(model_cls(**params), X, y, cv=5, scoring='average_precision')
    return scores.mean()

study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42),
    study_name='lgbm_tuning_v1',
    storage='sqlite:///optuna_studies.db',  # resumable
)
study.optimize(objective, n_trials=200, timeout=3600)

• Key rules:
• Always use log=True for learning rates, regularization
• Always set seed in sampler for reproducibility
• Use SQLite storage for runs > 30 minutes (crash recovery)

Pattern 2: Pruning for Expensive Models

• Use when: Single trial takes > 2 minutes
• Implementation:

from optuna.pruners import MedianPruner, HyperbandPruner

study = optuna.create_study(
    direction='maximize',
    pruner=MedianPruner(
        n_startup_trials=10,   # don't prune first 10
        n_warmup_steps=20,     # don't prune before 20 epochs
        interval_steps=5,      # check every 5 epochs
    ),
)

def objective(trial):
    params = {... }  # suggest params

    for epoch in range(100):
        train_one_epoch(model, params)
        val_score = evaluate(model)

        trial.report(val_score, epoch)
        if trial.should_prune():
            raise optuna.TrialPruned()

    return val_score

• Pruner selection:

Pattern 3: LightGBM Tuning Recipe

• Use when: Tuning LightGBM for tabular data
• Search space (battle-tested ranges):

def lgbm_objective(trial):
    params = {
        'objective': 'binary',
        'metric': 'average_precision',
        'verbosity': -1,
        'boosting_type': 'gbdt',

        # Tier 1: Highest impact
        'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.2, log=True),
        'n_estimators': trial.suggest_int('n_estimators', 100, 2000, step=100),
        'num_leaves': trial.suggest_int('num_leaves', 15, 255),
        'max_depth': trial.suggest_int('max_depth', 3, 12),

        # Tier 2: Regularization
        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),

        # Tier 3: Stochastic
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.4, 1.0),
        'subsample_freq': trial.suggest_int('subsample_freq', 1, 7),

        # Tier 4: Fine-tuning
        'min_split_gain': trial.suggest_float('min_split_gain', 0.0, 1.0),
        'max_bin': trial.suggest_int('max_bin', 63, 511),
    }

    cv_result = lgb.cv(params, train_set, nfold=5, stratified=True,
                       return_cvbooster=True)
    return cv_result['valid average_precision-mean'][-1]

• Tuning order: Tune Tier 1 first (50 trials), freeze best, then add Tier 2–4

Pattern 4: scikit-learn Tuning Recipe

• Use when: Tuning RandomForest, GradientBoosting, SVM, or other sklearn models

def sklearn_rf_objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 100, 1000, step=50),
        'max_depth': trial.suggest_int('max_depth', 3, 30),
        'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 15),
        'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', 0.3, 0.5, 0.7]),
        'class_weight': trial.suggest_categorical('class_weight', ['balanced', 'balanced_subsample', None]),
    }
    clf = RandomForestClassifier(**params, random_state=42, n_jobs=-1)
    scores = cross_val_score(clf, X, y, cv=5, scoring='average_precision')
    return scores.mean()

Pattern 5: Multi-Objective Optimization

• Use when: Trading off accuracy vs latency, accuracy vs model size, etc.
• Implementation:

study = optuna.create_study(
    directions=['maximize', 'minimize'],  # accuracy UP, latency DOWN
    sampler=optuna.samplers.NSGAIISampler(seed=42),
)

def multi_objective(trial):
    params = {... }
    score = cross_val_score(model(**params), X, y, cv=3).mean()
    latency = measure_inference_latency(model(**params), X[:100])
    return score, latency

study.optimize(multi_objective, n_trials=200)

# Get Pareto front
pareto_trials = study.best_trials
for t in pareto_trials:
    print(f"Score: {t.values[0]:.4f}, Latency: {t.values[1]:.2f}ms")

Pattern 6: Warmstarting with Known-Good Configs

• Use when: You have prior knowledge or production configs to start from

study = optuna.create_study(direction='maximize')

# Seed with known-good config
study.enqueue_trial({
    'learning_rate': 0.05,
    'n_estimators': 500,
    'max_depth': 7,
    'num_leaves': 63,
})

study.optimize(objective, n_trials=150)

Pattern 7: Ray Tune for Distributed Tuning

• Use when: Cluster available, need to parallelize across GPUs/nodes

from ray import tune
from ray.tune.search.optuna import OptunaSearch

search_space = {
    'learning_rate': tune.loguniform(1e-4, 1e-1),
    'batch_size': tune.choice([32, 64, 128, 256]),
    'hidden_size': tune.choice([128, 256, 512]),
}

analysis = tune.run(
    train_fn,
    config=search_space,
    search_alg=OptunaSearch(metric='val_loss', mode='min'),
    num_samples=200,
    resources_per_trial={'cpu': 4, 'gpu': 1},
    scheduler=tune.schedulers.ASHAScheduler(
        metric='val_loss', mode='min',
        max_t=100, grace_period=10,
    ),
)

---

Reproducibility Checklist

# 1. Fix all random seeds
import numpy as np, random, torch
random.seed(42)
np.random.seed(42)
torch.manual_seed(42)

# 2. Use seeded sampler
sampler = optuna.samplers.TPESampler(seed=42)

# 3. Log environment
import optuna, sklearn, lightgbm
env_info = {
    'optuna': optuna.__version__,
    'sklearn': sklearn.__version__,
    'lgbm': lightgbm.__version__,
    'python': sys.version,
}

# 4. Store study to DB
study = optuna.create_study(storage='sqlite:///studies.db', study_name='exp_v1')

# 5. Export best trial
best = study.best_trial
print(f"Best value: {best.value}")
print(f"Best params: {best.params}")

---

Anti-Patterns

---

Convergence Analysis

# Check if study has converged
import optuna.visualization as vis

# Plot optimization history
vis.plot_optimization_history(study)

# Parameter importance (which params matter most)
vis.plot_param_importances(study)

# Slice plot (effect of each param)
vis.plot_slice(study)

# Rule of thumb: if best value hasn't improved in last 30% of trials, stop

---

Cross-References

• ai-ml-data-science/references/class-imbalance-patterns.md — tuning scale_pos_weight and sampling ratio
• ai-ml-data-science/references/interpretability-explainability.md — interpreting tuned models
• ai-mlops/references/experiment-tracking-patterns.md — logging Optuna studies to MLflow/W&B
• ai-mlops/references/automated-retraining-patterns.md — scheduling tuning runs in pipelines

Interpretability and Explainability

Operational guide for explaining ML model predictions. Covers SHAP, LIME, permutation importance, partial dependence, and audience-appropriate communication of model behavior. Focus on actionable interpretation, not theory.

Freshness anchor: January 2026 — SHAP 0.45+, LIME 0.2+, scikit-learn 1.5+, LightGBM 4.x

---

Decision Tree: Choosing an Explanation Method

START
│
├─ Model type?
│   ├─ Tree-based (LightGBM, XGBoost, RF, CatBoost)
│   │   └─ Use TreeSHAP (exact, fast, O(TLD))
│   │
│   ├─ Linear (LogisticRegression, Lasso, Ridge)
│   │   └─ Use LinearSHAP or direct coefficient interpretation
│   │
│   ├─ Neural network
│   │   ├─ Tabular → KernelSHAP (slow) or DeepSHAP
│   │   └─ Image/text → GradientSHAP, Integrated Gradients
│   │
│   └─ Black-box / API-only
│       └─ KernelSHAP or LIME (model-agnostic)
│
├─ Explanation scope?
│   ├─ Global (overall model behavior)
│   │   ├─ Feature importance ranking → SHAP summary plot
│   │   ├─ Feature effect curves → PDP or SHAP dependence
│   │   └─ Feature interactions → SHAP interaction values
│   │
│   └─ Local (single prediction)
│       ├─ Detailed breakdown → SHAP waterfall
│       ├─ Quick approximation → LIME
│       └─ Contrastive ("why not X?") → SHAP force plot
│
└─ Audience?
    ├─ Data scientist → Full SHAP values, interaction plots
    ├─ Business stakeholder → Top 3 drivers, bar charts
    └─ Regulatory / audit → Model cards, stability analysis

---

Quick Reference: Methods Comparison

---

Operational Patterns

Pattern 1: TreeSHAP for Tree-Based Models

• Use when: Using LightGBM, XGBoost, CatBoost, RandomForest
• Implementation:

import shap

# Train model
model = lgb.LGBMClassifier(**params).fit(X_train, y_train)

# Create explainer (auto-detects tree type)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# For binary classification: shap_values is [neg_class, pos_class]
# Use shap_values[1] for positive class explanations

# Global: Summary plot (feature importance + distribution)
shap.summary_plot(shap_values[1], X_test)

# Global: Bar plot (mean absolute SHAP per feature)
shap.summary_plot(shap_values[1], X_test, plot_type='bar')

# Local: Single prediction waterfall
shap.waterfall_plot(shap.Explanation(
    values=shap_values[1][0],
    base_values=explainer.expected_value[1],
    data=X_test.iloc[0],
    feature_names=X_test.columns.tolist()
))

• Performance tip: For large datasets, compute SHAP on a representative sample (5k–10k rows)
• Gotcha: TreeSHAP with feature_perturbation='interventional' gives causal-style attribution but requires background data

Pattern 2: KernelSHAP for Black-Box Models

• Use when: Model is an API, neural network, or ensemble of mixed types
• Implementation:

# Background data: use k-means summary for speed
background = shap.kmeans(X_train, 100)

explainer = shap.KernelExplainer(model.predict_proba, background)

# Compute on subset (KernelSHAP is slow)
shap_values = explainer.shap_values(X_test[:500], nsamples=500)

• Speed tradeoff: nsamples controls accuracy vs speed
• nsamples=100 — fast, rough approximation
• nsamples=500 — good balance
• nsamples=2048 — high accuracy, slow

Pattern 3: LIME for Quick Local Explanations

• Use when: Need fast, single-prediction explanation for stakeholders
• Implementation:

from lime.lime_tabular import LimeTabularExplainer

lime_exp = LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=X_train.columns.tolist(),
    class_names=['Negative', 'Positive'],
    mode='classification',
    discretize_continuous=True,
)

# Explain single prediction
exp = lime_exp.explain_instance(
    X_test.iloc[0].values,
    model.predict_proba,
    num_features=10,
    num_samples=5000,
)

exp.show_in_notebook()
# Or export: exp.as_html(), exp.as_list()

• Stability check: Run LIME 5 times on same instance — if top features change, results are unreliable
• Gotcha: LIME fits a local linear model — fails for highly non-linear local behavior

Pattern 4: Permutation Importance

• Use when: Need global feature ranking, model-agnostic, simple to explain
• Implementation:

from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_test, y_test,
    n_repeats=30,
    random_state=42,
    scoring='average_precision',
    n_jobs=-1,
)

# Sort by importance
sorted_idx = result.importances_mean.argsort()[::-1]
for idx in sorted_idx[:15]:
    print(f"{X_test.columns[idx]:30s}: "
          f"{result.importances_mean[idx]:.4f} +/- {result.importances_std[idx]:.4f}")

• Key advantage: Measures importance on unseen data (test set) — avoids overfitting bias
• Gotcha: Correlated features split importance — consider grouping correlated features

Pattern 5: Partial Dependence and ICE Plots

• Use when: Need to show how a feature affects predictions across its range
• Implementation:

from sklearn.inspection import PartialDependenceDisplay

# PDP for top features
features = ['age', 'income', ('age', 'income')]  # single + interaction
PartialDependenceDisplay.from_estimator(
    model, X_train, features,
    kind='both',      # PDP (average) + ICE (individual)
    subsample=500,    # ICE lines to plot
    grid_resolution=50,
    n_jobs=-1,
)

• PDP vs ICE:
• PDP = average effect (can hide heterogeneity)
• ICE = individual curves (reveals subgroups with different effects)
• Always plot both — if ICE lines are parallel, PDP is reliable; if they cross, PDP is misleading

Pattern 6: Feature Importance Stability Analysis

• Use when: Regulatory or audit context, need confidence in feature rankings
• Implementation:

import numpy as np

# Bootstrap SHAP importance stability
n_bootstrap = 20
importance_ranks = []

for i in range(n_bootstrap):
    sample_idx = np.random.choice(len(X_test), size=len(X_test), replace=True)
    X_sample = X_test.iloc[sample_idx]
    shap_vals = explainer.shap_values(X_sample)

    mean_abs = np.abs(shap_vals[1]).mean(axis=0)
    ranks = np.argsort(-mean_abs)  # descending
    importance_ranks.append(ranks)

# Compute rank stability per feature
from scipy.stats import kendalltau
stability_scores = []
for i in range(n_bootstrap):
    for j in range(i+1, n_bootstrap):
        tau, _ = kendalltau(importance_ranks[i], importance_ranks[j])
        stability_scores.append(tau)

print(f"Mean rank correlation: {np.mean(stability_scores):.3f}")
# > 0.9 = stable rankings; < 0.7 = unstable, report with caveats

---

Audience-Appropriate Explanations

Technical Audience (Data Scientists)

• Full SHAP summary plots with distributions
• Interaction values and dependence plots
• Permutation importance with confidence intervals
• Raw SHAP values for downstream analysis

Business Stakeholders

• Top 3–5 drivers as horizontal bar chart
• Natural language: "This customer was flagged primarily because their account age (2 months) is unusually short, and their transaction frequency (47/day) is 5x the average"
• Avoid: SHAP values, log-odds, probability scores
• Use: directional language ("increases risk", "decreases likelihood")

Regulatory / Audit

• Model card documenting: features used, protected attributes, fairness metrics
• Stability analysis across bootstrap samples
• Monotonicity checks for regulated features
• Feature importance consistency across time periods
• Documentation template:

## Model Explanation Report
- Model type: [type]
- Training date: [date]
- Explanation method: [TreeSHAP/KernelSHAP]
- Top 10 features (stable across 20 bootstrap runs): [list]
- Protected attribute impact: [analysis]
- Monotonicity compliance: [pass/fail per feature]

---

Anti-Patterns

---

Model Card Template (Interpretability Section)

## Interpretability

### Explanation Method
- Primary: TreeSHAP (exact for tree ensemble)
- Secondary: Permutation importance (validation)

### Top Features (Stable)
| Rank | Feature | Mean |SHAP| | Direction |
|------|---------|-------------|-----------|
| 1    | [name]  | [value]     | [+/-]     |

### Stability
- Bootstrap rank correlation (Kendall tau): [value]
- Feature ranking consistent across [N] time periods: [yes/no]

### Limitations
- [Correlated feature groups]
- [Non-monotonic relationships]
- [Protected attribute interactions]

---

Validation Checklist

• [ ] Explanation method matches model type (TreeSHAP for trees, etc.)
• [ ] SHAP values computed on representative sample (not full dataset unless small)
• [ ] Feature importance stable across bootstrap samples (tau > 0.8)
• [ ] Permutation importance confirms SHAP rankings (top 5 agree)
• [ ] PDP/ICE plots reviewed for non-linear effects and interactions
• [ ] Business-appropriate summary prepared (top drivers, directional language)
• [ ] Model card updated with interpretability section
• [ ] No causal claims made from correlational analysis
• [ ] Protected attributes checked for disproportionate importance

---

Cross-References

• ai-ml-data-science/references/class-imbalance-patterns.md — interpreting minority-class predictions
• ai-ml-data-science/references/hyperparameter-optimization.md — feature importance after tuning
• ai-mlops/references/experiment-tracking-patterns.md — logging SHAP artifacts
• ai-rag/references/embedding-model-guide.md — explaining embedding-based features

Modelling Patterns

Operational modelling techniques, baseline-first methodologies, model selection, train/test splits, and model comparison rules.

---

1. Model Selection & Baselines (Modern Best Performers)

1.1 Decision Guide

Based on current benchmarks and best practices:

Key Finding: Tree-based methods (LightGBM) deliver best performance with significant computational efficiency advantage.

1.2 Baseline First Pattern

Always implement simple baselines first:

Classification:

• Majority-class classifier
• Stratified random
• Simple rule-based (if domain knowledge available)

Regression:

• Mean/median predictor
• Linear regression
• Moving average (for time series)

Time series:

• Seasonal naive forecast
• Last-value carry-forward

Why baselines matter:

• Establish minimum performance bar
• Reality check for model complexity
• Fast iteration and debugging
• Interpretability reference

Checklist: Baselines

• [ ] Simple baseline implemented (majority class, mean, naive forecast)
• [ ] LightGBM/XGBoost tried as primary candidate
• [ ] Complexity added only after baselines understood
• [ ] Compute, latency, and explainability constraints considered early
• [ ] Model performance logged in experiment tracker (MLflow/W&B)

---

2. Train/Validation/Test Split Design

2.1 Split Strategies

Random split (IID):

• Use when: Data is independent and identically distributed
• Pros: Simple, maximizes training data
• Cons: Doesn't test temporal generalization

Time-based split:

• Use when: Forecasting or temporal leakage risk
• Pattern: Train on [T0, T1], validate on [T1, T2], test on [T2, T3]
• Pros: Tests realistic deployment scenario
• Cons: Less training data, seasonality may affect splits

Group-based split:

• Use when: User/item/entity leakage risk
• Pattern: Split by user_id, never mix same user across sets
• Examples: Recommendation systems, fraud detection
• Pros: Tests generalization to new entities
• Cons: Reduces effective sample size

Cross-validation:

• Use when: Small datasets, need robust estimates
• K-fold: 5 or 10 folds typical
• Stratified: Preserve class balance in each fold
• Time-series CV: Rolling/expanding window
• Pros: Better variance estimates, more data usage
• Cons: K times slower, risk of data leakage if not careful

2.2 Common Pitfalls

Leakage:

• Same entity in train and test (user, transaction)
• Feature computed using test data
• Future information in training

Imbalance:

• Rare classes missing from validation/test
• Non-representative splits

Size:

• Test set too small for reliable metrics
• Validation set too small for hyperparameter tuning

2.3 Recommended Ratios

Large datasets (>100k samples):

• Train: 80%, Validation: 10%, Test: 10%

Medium datasets (10k-100k):

• Train: 70%, Validation: 15%, Test: 15%

Small datasets (<10k):

• Use cross-validation instead of single split
• Hold out 20% for final test

Checklist: Split Design

• [ ] Split respects time order when needed
• [ ] No record from same entity in both train and test where leakage matters
• [ ] Test/validation sets held out from all model decisions
• [ ] Evaluation method documented and reproducible
• [ ] Class balance validated in all splits
• [ ] Test set size sufficient for statistical significance

---

3. Model Family Selection

3.1 Tabular Data

First choice:

• LightGBM (fast, accurate, handles categorical features)
• XGBoost (mature, well-tested)
• CatBoost (handles high-cardinality categoricals)

Linear models:

• Logistic regression (interpretable baseline)
• Ridge/Lasso (regularized linear)
• Use when: Need interpretability, compliance, or very fast inference

Neural networks:

• Only when: Large datasets (>1M rows), complex interactions
• TabNet, FT-Transformer for tabular data

3.2 Text Data

Start with:

• TF-IDF + linear models (fast baseline)
• Pretrained embeddings (Sentence-BERT) + LightGBM

Advanced:

• Fine-tuned transformers (BERT, RoBERTa)
• Only when: Large labeled dataset, need state-of-art

3.3 When to Avoid Deep Models

Don't use neural networks when:

• Small datasets (<10k samples)
• Highly structured relational data (use tree models)
• Need interpretability for compliance
• Limited compute budget

Checklist: Model Family

• [ ] Model complexity matches data size
• [ ] Baseline -> interpretable model -> complex model progression
• [ ] Compute and latency constraints considered
• [ ] Interpretability requirements documented

---

4. Hyperparameter Tuning

4.1 Tuning Strategy

Level 1: Manual scan (fast)

• Test 3-5 values per key parameter
• Use domain knowledge and defaults
• Time: Minutes to hours

Level 2: Grid search (thorough)

• Small grid on important parameters
• Use when: Need reproducibility
• Time: Hours to day

Level 3: Random search (efficient)

• Sample random combinations
• Better than grid for high-dimensional spaces
• Time: Hours to day

Level 4: Bayesian optimization (smart)

• Use Optuna, Ray Tune, Hyperopt
• Learns from previous trials
• Time: Hours to days

4.2 Key Parameters by Model

LightGBM:

• num_leaves (31-255)
• learning_rate (0.01-0.3)
• min_data_in_leaf (20-100)
• feature_fraction (0.7-1.0)

XGBoost:

• max_depth (3-10)
• learning_rate (0.01-0.3)
• min_child_weight (1-10)
• subsample (0.7-1.0)

Neural networks:

• Learning rate (1e-5 to 1e-2, log scale)
• Batch size (16, 32, 64, 128)
• Dropout rate (0.1-0.5)
• Number of layers (2-6)

4.3 Stability and Reproducibility

Best practices:

• Set random seeds (model, data split, sampling)
• Run multiple seeds for final model (e.g., 5 seeds)
• Report mean +/- std across seeds
• Log all hyperparameters to experiment tracker

Checklist: Hyperparameter Tuning

• [ ] Parameters logged in experiment tracker
• [ ] Seeds logged and controlled
• [ ] Overfitting checked (train vs validation)
• [ ] Multiple runs for stability (3-5 seeds minimum)
• [ ] Best parameters documented with justification

---

5. Overfitting Control

5.1 Detection

Indicators of overfitting:

• Train error decreases while validation error increases
• Large gap between train and validation metrics
• Model performs well on training data but poorly on new data

Monitoring:

• Plot train vs validation loss/metric over epochs/iterations
• Check learning curves
• Validate on held-out test set

5.2 Mitigation Techniques

For tree models:

• Limit max_depth (3-10)
• Increase min_data_in_leaf / min_child_weight
• Reduce num_leaves
• Use feature subsampling (feature_fraction, colsample_bytree)

For neural networks:

• Dropout (0.2-0.5)
• L2 regularization (weight decay)
• Early stopping (patience = 5-10 epochs)
• Data augmentation

For linear models:

• L1 (Lasso) or L2 (Ridge) regularization
• Reduce number of features (feature selection)

Universal:

• Get more training data
• Simplify model architecture
• Cross-validation for robust estimates

Checklist: Overfitting Control

• [ ] Train vs validation gap monitored
• [ ] Regularization applied (appropriate to model type)
• [ ] Early stopping configured (if applicable)
• [ ] Learning curves analyzed
• [ ] Test set performance validates generalization

---

6. CatBoost for Categorical-Heavy Data

6.1 When to Choose CatBoost

CatBoost often outperforms LightGBM/XGBoost when:

• Dataset contains many categorical features (>30% of features)
• High-cardinality categoricals (cities, product IDs, user IDs)
• Limited time for feature engineering (native handling reduces preprocessing)
• Need robust defaults with minimal hyperparameter tuning

Key advantages:

• Ordered target encoding: Prevents target leakage automatically
• Built-in overfitting detection: Automatic early stopping
• GPU support: Native CUDA implementation for training
• Symmetric trees: Better generalization on some datasets

6.2 CatBoost vs LightGBM vs XGBoost

6.3 CatBoost Key Parameters

from catboost import CatBoostClassifier

model = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.1,
    depth=6,  # 4-10 typical
    l2_leaf_reg=3,  # L2 regularization
    cat_features=['city', 'product_id', 'category'],  # Specify categorical columns
    early_stopping_rounds=50,
    verbose=100
)

Checklist: CatBoost

• [ ] Categorical features identified and passed to cat_features
• [ ] Compared against LightGBM/XGBoost baseline
• [ ] Early stopping configured
• [ ] GPU enabled for large datasets (task_type='GPU')

---

7. GPU Scaling for Large Datasets

7.1 When to Use GPU Training

Indicators:

• Dataset exceeds 10M+ rows
• Training time >1 hour on CPU
• Need rapid experimentation cycles
• Production requires frequent retraining

Benchmark reference (H100 GPUs):

• 1.2B rows, 120 features: ~7 minutes with 6x H100 GPUs
• 100M rows: ~30-60 seconds

7.2 GPU Training with LightGBM

import lightgbm as lgb

params = {
    'device': 'gpu',
    'gpu_platform_id': 0,
    'gpu_device_id': 0,
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 63,
    'learning_rate': 0.05,
    'feature_fraction': 0.8
}

train_data = lgb.Dataset(X_train, label=y_train)
model = lgb.train(params, train_data, num_boost_round=500)

7.3 Distributed Training with Ray

For datasets that don't fit in memory or require horizontal scaling:

from ray.train.lightgbm import LightGBMTrainer
from ray.train import ScalingConfig

trainer = LightGBMTrainer(
    label_column="target",
    params={
        "objective": "binary",
        "metric": "auc",
        "num_leaves": 63
    },
    scaling_config=ScalingConfig(
        num_workers=4,
        use_gpu=True,
        resources_per_worker={"GPU": 1}
    ),
    datasets={"train": train_ds, "valid": valid_ds}
)

result = trainer.fit()

7.4 XGBoost GPU Training

import xgboost as xgb

params = {
    'tree_method': 'hist',
    'device': 'cuda',
    'objective': 'binary:logistic',
    'eval_metric': 'auc',
    'max_depth': 6,
    'learning_rate': 0.1
}

dtrain = xgb.DMatrix(X_train, label=y_train)
model = xgb.train(params, dtrain, num_boost_round=500)

Checklist: GPU Scaling

• [ ] GPU availability verified (nvidia-smi)
• [ ] CUDA drivers and libraries installed
• [ ] Memory requirements estimated (GPU VRAM)
• [ ] Fallback to CPU configured for debugging
• [ ] Ray cluster configured for distributed training (if needed)
• [ ] Training time benchmarked: CPU vs GPU

---

8. Model Comparison

8.1 Fair Comparison Rules

Requirements:

• Same train/validation/test split (same random seed)
• Same evaluation metric
• Same feature set (or document differences)
• Same hardware (for latency comparisons)

What to compare:

• Primary metric (accuracy, RMSE, etc.)
• Compute cost (training time, memory)
• Inference latency (p50, p95, p99)
• Model size (disk, memory)
• Interpretability (if relevant)

8.2 Statistical Significance

When to test:

• Comparing two models
• Small performance differences
• Need confidence in improvement

Methods:

• Paired t-test (cross-validation folds)
• Bootstrap confidence intervals
• Permutation test

Checklist: Model Comparison

• [ ] Apples-to-apples comparison (same data, metric, hardware)
• [ ] Primary metric differences documented
• [ ] Secondary metrics considered (latency, cost, interpretability)
• [ ] Statistical significance tested (if differences small)
• [ ] Documented reasons for final choice

---

9. Thresholding for Classification

9.1 Threshold Selection

Methods:

• ROC curve: Maximize TPR, minimize FPR
• PR curve: Precision-recall trade-off (better for imbalanced)
• F1 score: Harmonic mean of precision and recall
• Cost-sensitive: Assign costs to FP and FN, minimize total cost

Context-specific:

• Fraud detection: High recall (catch fraudsters), tolerate FP
• Spam filtering: High precision (don't block legitimate emails)
• Medical diagnosis: Balance based on cost of FN vs FP

9.2 Per-Segment Validation

Why it matters:

• Optimal threshold may vary by segment
• Fairness: ensure performance across demographics
• Business logic: different risk tolerances

Checklist: Thresholding

• [ ] Threshold selection method documented (ROC, PR, cost)
• [ ] ROC and PR curves generated
• [ ] Threshold chosen with business justification
• [ ] Per-segment thresholds validated (if applicable)
• [ ] Trade-offs documented (precision vs recall)

Reproducibility Checklist

Ensuring ML experiments are reproducible, trackable, and production-ready with modern MLOps practices (CI/CD, CT, CM).

---

1. Experiment Tracking & Versioning

1.1 What to Track

Every training run must log:

• Code version: Git commit hash
• Data version: Dataset snapshot ID or hash
• Feature set version: From feature store
• Hyperparameters: All model and training config
• Random seeds: For reproducibility
• Metrics: Primary and guardrail metrics
• Artifacts: Model weights, preprocessors, encoders
• Drift statistics: Distribution comparison vs training data

1.2 Experiment Tracking Tools

MLflow:

• Open-source, self-hosted
• Experiment tracking + model registry
• Integrates with popular frameworks

Weights & Biases (W&B):

• Cloud-hosted, polished UI
• Real-time metrics visualization
• Sweep/hyperparameter optimization

Neptune:

• Metadata store for ML
• Advanced experiment comparison
• Team collaboration features

DVC (Data Version Control):

• Git for data
• Pipeline tracking
• Reproducible experiments

Checklist: Experiment Tracking

• [ ] Experiments logged with code + data + params + feature version
• [ ] Best runs easily identifiable with tagged metrics
• [ ] Re-running yields same metrics within noise
• [ ] Model registry entry created for candidate models
• [ ] Drift statistics logged for production monitoring

---

2. Modern MLOps Integration (CI/CD/CT/CM)

2.1 Continuous Integration (CI)

Automated testing and validation:

• Unit tests for data preprocessing and feature engineering
• Integration tests for training pipeline
• Code quality checks (linting, type checking)
• Data validation (schema checks, distribution tests)

Tools:

• GitHub Actions, GitLab CI, Jenkins
• Great Expectations (data validation)
• pytest, unittest

2.2 Continuous Delivery (CD)

Automated deployment:

• Environment-specific model promotion (dev -> staging -> prod)
• Automated model packaging (Docker, model serving format)
• Canary deployment with gradual rollout
• Rollback on regression

Tools:

• Kubernetes, Docker
• MLflow Model Registry
• BentoML, Seldon, KServe

2.3 Continuous Training (CT)

Automated retraining:

• Triggered by drift detection (data or performance)
• Scheduled retraining (weekly, monthly)
• New data availability triggers
• Automated evaluation and promotion

Triggers:

• Drift exceeds threshold (PSI, KL divergence)
• Performance degradation (accuracy drop > 5%)
• Calendar schedule (monthly refresh)
• Manual trigger (emergency retrain)

2.4 Continuous Monitoring (CM)

Real-time production monitoring:

• Data drift (input distribution changes)
• Concept drift (target distribution changes)
• Model performance (accuracy, latency, errors)
• System health (CPU, memory, throughput)

Metrics:

• Data drift: KL divergence, PSI, KS test
• Performance: Online accuracy, solve rate, calibration
• Operational: Latency (p50, p95, p99), error rate, cost

Checklist: MLOps Integration

• [ ] CI/CD pipeline integrated for automated testing
• [ ] CT configured with drift-based and scheduled triggers
• [ ] CM dashboards active with drift and performance metrics
• [ ] Automated retraining and promotion workflow tested
• [ ] Rollback procedure documented and tested

---

3. Environment & Dependency Management

3.1 Python Environment

Requirements:

• Python version pinned (e.g., 3.10.12)
• Package versions locked (requirements.txt, poetry.lock, Pipfile.lock)
• Virtual environment (venv, conda, poetry)

Best practices:

• Use pip freeze > requirements.txt or poetry
• Pin all dependencies, including transitive ones
• Test installation on clean environment

3.2 System Dependencies

Document:

• Operating system (Ubuntu 22.04, macOS 14.2)
• CUDA version (for GPU training)
• System libraries (libgeos, GDAL, etc.)
• Hardware requirements (CPU cores, RAM, GPU)

3.3 Docker for Reproducibility

Benefits:

• Complete environment specification
• Portable across machines
• Consistent training and serving

Example Dockerfile:

FROM python:3.10.12-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app
CMD ["python", "train.py"]

Checklist: Environment Pinned

• [ ] Python version documented and pinned
• [ ] All package versions locked (requirements.txt or equivalent)
• [ ] System dependencies documented
• [ ] Docker image built and tested (if applicable)
• [ ] Environment reproducible on fresh machine

---

4. Data Versioning

4.1 What to Version

Datasets:

• Training, validation, test splits
• Raw data snapshots (before preprocessing)
• Processed features (after transformations)
• Data lineage (source -> intermediate -> final)

Metadata:

• Extraction timestamp
• Data quality metrics (nulls, outliers, distribution)
• Sampling strategy
• Label quality (inter-annotator agreement)

4.2 Data Versioning Tools

DVC (Data Version Control):

• Git-like interface for data
• Store data in S3, GCS, Azure Blob
• Track data lineage and pipelines

LakeFS:

• Git for data lakes
• Branching and merging for datasets
• Time-travel queries

Feature stores:

• Feast, Tecton, Databricks Feature Store
• Centralized feature management
• Version features alongside models

Checklist: Data Versioned

• [ ] Dataset snapshots tracked with version IDs
• [ ] Train/validation/test splits documented and versioned
• [ ] Data lineage captured (source -> transformations -> features)
• [ ] Metadata logged (quality metrics, extraction time)
• [ ] Feature store used for centralized versioning (if applicable)

---

5. Random Seed Management

5.1 Sources of Randomness

Control seeds for:

• NumPy (np.random.seed())
• Python random (random.seed())
• Model libraries (LightGBM, XGBoost, PyTorch, TensorFlow)
• Data sampling and train/test splits
• Data augmentation

5.2 Setting Seeds

Example (Python):

import random
import numpy as np
import torch

def set_seed(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # For deterministic behavior (slower)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

LightGBM/XGBoost:

params = {
    'seed': 42,
    'feature_fraction_seed': 42,
    'bagging_seed': 42
}

Checklist: Randomness Controlled

• [ ] All random seeds set at script start
• [ ] Seeds logged in experiment tracker
• [ ] Multiple seed runs for stability (5-10 seeds)
• [ ] Deterministic behavior verified (same input -> same output)

---

6. Model Artifacts & Registry

6.1 What to Archive

For each model:

• Model weights (.pkl, .h5, .pt, .onnx)
• Preprocessors (scalers, encoders, tokenizers)
• Feature transformations (versioned with feature store)
• Hyperparameters (JSON config)
• Training metadata (metrics, data version, git commit)

6.2 Model Registry

Purpose:

• Centralized model storage
• Version management
• Stage promotion (dev -> staging -> prod)
• Metadata and lineage

Tools:

• MLflow Model Registry
• W&B Model Registry
• Cloud-specific (SageMaker Model Registry, Vertex AI Model Registry)

Checklist: Model Artifacts Managed

• [ ] Model weights and preprocessors saved
• [ ] Artifacts uploaded to model registry
• [ ] Model versioned with semantic versioning (v1.0.0, v1.1.0)
• [ ] Stage annotations (dev, staging, production)
• [ ] Metadata linked (training data version, metrics, owner)

---

7. Documentation & Model Cards

7.1 Code Documentation

Requirements:

• README with setup instructions
• Docstrings for functions and classes
• Inline comments for complex logic
• Architecture diagrams (for complex systems)

7.2 Model Card

Essential sections:

• Model overview and intended use
• Training data description and biases
• Performance metrics and limitations
• Operational requirements (latency, dependencies)
• Owners and maintenance plan

Checklist: Documentation Complete

• [ ] README with environment setup and training instructions
• [ ] Model card created with all sections
• [ ] Runbooks for common issues
• [ ] Architecture diagrams (if applicable)

---

8. End-to-End Reproducibility Workflow

8.1 Reproducibility Test

Validate reproducibility by: 1. Clone repository on fresh machine 2. Set up environment from requirements.txt or Dockerfile 3. Download data using DVC or data versioning tool 4. Run training script with documented seed 5. Verify metrics match within tolerance (+/- 1%)

8.2 Continuous Validation

Automated checks:

• CI pipeline runs reproducibility test on PRs
• Periodic re-training to validate pipeline
• Drift detection triggers investigation

Checklist: Reproducibility Validated

• [ ] Reproducibility test passes on fresh environment
• [ ] Same code + data + seed -> same metrics (+/- 1%)
• [ ] CI pipeline validates reproducibility
• [ ] Documentation sufficient for new team member

---

9. Production Readiness Checklist

Before deploying to production:

• [ ] All randomness seeded and logged
• [ ] Data and code versioned
• [ ] Experiments logged with full context (code, data, params, metrics)
• [ ] Model registry entry created with stage annotation
• [ ] CI/CD pipeline integrated and tested
• [ ] CT (continuous training) configured with triggers
• [ ] CM (continuous monitoring) dashboards active
• [ ] Drift monitoring enabled (data + concept + performance)
• [ ] Feature store tracks all transformations and versions
• [ ] Model card created and approved
• [ ] Rollback procedure tested
• [ ] Reproducibility validated on fresh environment

---

Related Resources

• Data Contracts & Lineage - Data versioning and lineage tracking
• Feature Freshness & Streaming - Real-time feature updates
• Production Feedback Loops - Online learning and continuous improvement
• Evaluation Patterns - Metrics and model evaluation