Data Scientist

Data Scientist

Overview

Execute data science workflows from exploration to production. This skill covers machine learning modeling, statistical analysis, A/B testing, causal inference, feature engineering, model evaluation, and MLOps patterns.

Features

• ML modeling lifecycle: problem framing, data prep, model selection, training, evaluation
• Statistical analysis: hypothesis testing, regression, ANOVA, Bayesian methods
• A/B testing: experiment design, sample size calculation, statistical power, result interpretation
• Causal inference: propensity score matching, difference-in-differences, instrumental variables
• Feature engineering: encoding, scaling, selection, dimensionality reduction
• MLOps: model deployment, monitoring, drift detection, retraining triggers

Usage

1. Identify the user's data science need (modeling, analysis, experimentation, or MLOps) 2. Follow the corresponding workflow below 3. Produce structured outputs: model cards, experiment reports, feature engineering pipelines, or MLOps runbooks

Examples

• User: "Build a churn prediction model"

Agent: Runs ML Modeling workflow, frames problem, selects features, trains classifier, evaluates with precision/recall, produces model card

• User: "Design an A/B test"

Agent: Runs Experiment Design workflow, calculates sample size, defines success metrics, sets up randomization, produces experiment plan

• User: "Monitor model drift"

Agent: Runs MLOps workflow, defines drift metrics, sets up monitoring dashboard, configures retraining triggers

When to Use

• Scoping ML problems, baselines, feature engineering, and model evaluation
• Designing A/B tests, power analysis, or causal inference when experiments are infeasible
• Productionizing models (batch, real-time, monitoring, retraining triggers)
• Selecting ML, stats, or MLOps tools for a given problem and data regime

When NOT to Use

• Executive dashboards, KPI definitions, or BI storytelling → use bi-analyst
• Warehouse dimensional modeling or ETL idempotency patterns → use data-warehouse-engineer
• Prompt design, LLM agents, or guardrailed GenAI features → use prompt-engineer
• RL training platform, rollout workers, distributed PPO/SAC jobs → use ml-systems-engineer-rl-engineering
• Revenue metrics (ARR, NRR) or ASC 606 accounting → use senior-revenue-accountant

Core Workflows

1. End-to-End ML Project Workflow

Phase checklist:

1. Problem definition

• Define the business metric to optimize
• Determine if ML is needed (rule-based may suffice)
• Set success criteria and failure modes

2. Data exploration & validation

• Profile distributions, missing values, duplicates
• Check for leakage (future information in training data)
• Validate data freshness and coverage

3. Feature engineering

• Create domain-relevant features
• Encode categoricals, scale numerics
• Document feature definitions and dependencies

4. Modeling

• Baseline: simple model first (linear regression, logistic)
• Iterate: tree-based, then ensembles, then deep learning if needed
• Cross-validate properly (time-based for temporal data)

5. Evaluation

• Hold-out test set, never used for hyperparameter tuning
• Check calibration, fairness, robustness
• Compare against baseline and business threshold

6. Production

• Serialize model, build inference API
• Add monitoring (prediction drift, latency)
• Document retraining triggers

2. Statistical Analysis & Experimentation

A/B testing workflow:

1. Define hypothesis, primary metric, and minimal detectable effect (MDE) 2. Calculate sample size (power analysis) 3. Randomize and run experiment 4. Check invariant metrics (randomization sanity) 5. Analyze primary metric with proper statistical test 6. Correct for multiple comparisons if needed 7. Document and socialize results

Causal inference when A/B test is impossible:

• Difference-in-differences
• Propensity score matching
• Instrumental variables
• Regression discontinuity

3. Productionizing Models (MLOps)

Deployment patterns:

Monitoring checklist:

• [ ] Prediction distribution drift vs training
• [ ] Feature drift (incoming data changes)
• [ ] Latency and throughput
• [ ] Error rate and fallback behavior
• [ ] Business metric tracking

4. Tool Selection

Analytics & Statistics

Hypothesis Testing

Test Selection Guide

Always check assumptions:

• Independence of observations
• Normality (for parametric tests; use Shapiro-Wilk or Q-Q plot)
• Homogeneity of variance (Levene's test)
• Sample size (central limit theorem helps for n > 30)

Python Examples

from scipy import stats

# Two-sample t-test
t_stat, p_value = stats.ttest_ind(group_a, group_b, equal_var=False)

# Mann-Whitney U (non-parametric)
u_stat, p_value = stats.mannwhitneyu(group_a, group_b, alternative='two-sided')

# Chi-square test
contingency = pd.crosstab(df['category'], df['outcome'])
chi2, p_value, dof, expected = stats.chi2_contingency(contingency)

# Effect size (Cohen's d)
def cohens_d(x, y):
    nx, ny = len(x), len(y)
    dof = nx + ny - 2
    return (np.mean(x) - np.mean(y)) / np.sqrt(((nx-1)*np.std(x, ddof=1)**2 + (ny-1)*np.std(y, ddof=1)**2) / dof)

A/B Testing

Experiment Design Checklist

• [ ] Primary metric defined (single metric to optimize)
• [ ] Secondary metrics identified (guardrails, not for decision)
• [ ] Minimal Detectable Effect (MDE) set with business input
• [ ] Sample size calculated (power = 0.8, alpha = 0.05)
• [ ] Randomization unit matches analysis unit (usually user)
• [ ] Duration covers at least 1 full business cycle
• [ ] No confounding launches during experiment

Sample Size Calculation

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# For proportions (conversion rate)
baseline = 0.10  # 10% conversion
mde = 0.02       # Detect 12% vs 10%
effect_size = proportion_effectsize(baseline + mde, baseline)

power_analysis = NormalIndPower()
sample_size = power_analysis.solve_power(
    effect_size=effect_size,
    alpha=0.05,
    power=0.8,
    ratio=1
)
# sample_size per group

Analysis Best Practices

1. Intent-to-treat: Analyze all randomized users, even if they didn't engage 2. Check invariant metrics: Confirm randomization worked (e.g., device split unchanged) 3. Use proper statistical test: t-test for continuous, z-test for proportions 4. Confidence intervals: Report with estimates, not just p-values 5. Segment analysis: Pre-defined segments only; avoid data dredging 6. Multiple comparisons: Bonferroni or FDR correction if testing many metrics

Early Stopping

Avoid peeking without correction:

• Fixed-horizon testing: Decide duration upfront, don't stop early
• Sequential testing: Use proper sequential boundaries (e.g., Group Sequential, Always Valid P-values)
• Bayesian monitoring: Credible intervals, stop when decisive

Causal Inference

Methods by Context

Difference-in-Differences Example

import statsmodels.formula.api as smf

# Panel data with treated and post indicators
model = smf.ols('outcome ~ treated * post + entity_fe + time_fe',
                data=panel_data).fit()
# Coefficient on treated:post is the ATT

Propensity Score Matching

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors

# Estimate propensity scores
ps_model = LogisticRegression().fit(X_confounders, treatment)
propensity_scores = ps_model.predict_proba(X_confounders)[:, 1]

# Match treated to control
nn = NearestNeighbors(n_neighbors=1).fit(propensity_scores[treatment==0].reshape(-1, 1))
distances, indices = nn.kneighbors(propensity_scores[treatment==1].reshape(-1, 1))

Exploratory Data Analysis (EDA)

Systematic EDA Checklist

1. Structure: Rows, columns, types, missingness pattern 2. Distributions: Histograms, boxplots, summary stats 3. Relationships: Correlation matrix, pairplots, groupby aggregations 4. Anomalies: Outliers, impossible values, data entry errors 5. Temporal: Trends, seasonality, gaps in time series 6. Geographic: Spatial distributions if location data exists

Missing Data Strategy

Imputation methods:

• Simple: mean, median, mode
• Advanced: KNN imputation, iterative imputer (MICE), MissForest
• Domain: business rule-based (e.g., "no purchase → spend = 0")

Statistical Power & Errors

Practical significance vs statistical significance:

• p < 0.05 with 1M samples can detect trivial effects
• Always report effect sizes and confidence intervals
• Ask: "Is this difference meaningful for the business?"

Bayesian Basics

When to use Bayesian methods:

• Small sample sizes (priors help regularize)
• Sequential updating (new data refines beliefs)
• Hierarchical structure (users within segments)
• Need full posterior distribution, not just point estimate

Bayesian A/B test:

import pymc as pm

with pm.Model() as model:
    p_a = pm.Beta('p_a', alpha=1, beta=1)
    p_b = pm.Beta('p_b', alpha=1, beta=1)
    
    obs_a = pm.Binomial('obs_a', n=n_a, p=p_a, observed=conversions_a)
    obs_b = pm.Binomial('obs_b', n=n_b, p=p_b, observed=conversions_b)
    
    diff = pm.Deterministic('diff', p_b - p_a)
    trace = pm.sample(2000)

# Probability that B is better than A
prob_better = (trace.posterior.diff > 0).mean()

Machine Learning Modeling

Algorithm Selection Guide

Feature Engineering Patterns

Numeric Features

# Log transform for skewed distributions
import numpy as np
df['log_revenue'] = np.log1p(df['revenue'])

# Binning for non-linear relationships
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 50, 65, 100])

# Interaction terms
df['price_per_unit'] = df['price'] / df['quantity']

Categorical Features

# High cardinality: target encoding (with regularization)
from category_encoders import TargetEncoder
encoder = TargetEncoder(cols=['city'])

# Ordinal: maintain order if exists
df['size_encoded'] = df['size'].map({'S': 1, 'M': 2, 'L': 3})

# One-hot for low cardinality (<10)
pd.get_dummies(df['color'], prefix='color')

Temporal Features

# Cyclical encoding for time
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)

# Time since event
df['days_since_signup'] = (df['event_date'] - df['signup_date']).dt.days

Text Features

# TF-IDF for classical ML
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(max_features=10000, ngram_range=(1, 2))

# Embeddings for deep learning
# Use sentence-transformers or LLM APIs

Cross-Validation Strategies

Time series split example:

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    # train is always before test in time
    X_train, X_test = X[train_idx], X[test_idx]

Evaluation Metrics

Classification

Regression

Hyperparameter Tuning

Search Strategies

Optuna example:

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 50, 500)
    max_depth = trial.suggest_int('max_depth', 3, 20)
    min_samples_split = trial.suggest_float('min_samples_split', 0.01, 0.3)

    clf = RandomForestClassifier(n_estimators=n_estimators,
                                  max_depth=max_depth,
                                  min_samples_split=min_samples_split)
    score = cross_val_score(clf, X, y, cv=5, scoring='f1').mean()
    return score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

Model Interpretability

Global Interpretation

• Feature importance: Built-in (tree-based) or permutation importance
• SHAP summary plot: Feature impact distribution
• Partial dependence plots: Feature effect on prediction

Local Interpretation

• SHAP force plot: Why this specific prediction?
• LIME: Local surrogate model explanation
• Counterfactuals: What would change the prediction?

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

Common Pitfalls

MLOps & Production

Model Deployment Patterns

Batch Scoring

# Scheduled job (Airflow, cron, cloud scheduler)
def batch_predict(model, input_path, output_path):
    df = pd.read_parquet(input_path)
    df['prediction'] = model.predict(df[features])
    df[['id', 'prediction']].to_parquet(output_path, partition_cols=['date'])

Real-Time API

from fastapi import FastAPI
import joblib

app = FastAPI()
model = joblib.load('model.pkl')

@app.post("/predict")
def predict(input_data: dict):
    features = preprocess(input_data)
    prediction = model.predict([features])[0]
    return {"prediction": prediction, "model_version": "1.0.0"}

Embedded (Database Native)

-- BigQuery ML example
CREATE OR REPLACE MODEL `project.dataset.model`
OPTIONS(model_type='LOGISTIC_REG')
AS SELECT * FROM `project.dataset.training_data`;

-- Predict in SQL
SELECT * FROM ML.PREDICT(MODEL `project.dataset.model`,
  (SELECT * FROM `project.dataset.new_data`));

Model Monitoring

What to Monitor

Drift Detection Example

from scipy.stats import ks_2samp
import numpy as np

def detect_drift(reference, production, threshold=0.01):
    """Returns True if distributions are significantly different"""
    statistic, p_value = ks_2samp(reference, production)
    return p_value < threshold

# For each feature
for col in features:
    if detect_drift(train[col], production[col]):
        alert(f"Drift detected in {col}")

Shadow Deployment

Run new model alongside production without serving its predictions:

• Log predictions from both models
• Compare offline metrics
• Switch traffic when new model is validated

Feature Stores

When to Use

• Same features needed by multiple models
• Complex feature engineering (streaming aggregations)
• Need point-in-time correctness (prevent leakage)

Architecture

Raw Data → Feature Pipelines → Feature Store → Models
                               (online + offline)

Point-in-Time Correctness

# Critical: feature values as they were at prediction time
# NOT as they are now
features_at_time = feature_store.get_features(
    entity_ids=['user_123'],
    timestamp='2024-01-15T10:00:00Z'  # NOT now()
)

Model Versioning & Registry

MLflow Tracking

import mlflow

mlflow.set_experiment("churn_prediction")

with mlflow.start_run():
    mlflow.log_params({"n_estimators": 100, "max_depth": 6})
    mlflow.log_metrics({"f1": 0.85, "auc": 0.92})
    mlflow.sklearn.log_model(model, "model")
    mlflow.set_tag("version", "v1.2.0")

Model Registry Stages

1. Staging: Candidate model under evaluation 2. Production: Currently serving traffic 3. Archived: Retired, kept for audit

CI/CD for ML

Pipeline Stages

Code commit → Unit tests → Integration tests → Train model → Evaluate → Register → Deploy

Key differences from software CI/CD:

• Model training is non-deterministic (set seeds, log everything)
• Evaluation needs hold-out data (not just unit tests)
• Deployment may require data validation, not just code validation
• Rollback means reverting to previous model artifact

Testing Strategy

Git-Based Workflows

Git-flow for ML:

• main: Production model + code
• develop: Integration branch
• experiment/<name>: Model experiments (may not merge)
• feature/<name>: Code features (merge to develop)

Model promotion via PR:

• Training run triggered on PR
• Evaluation metrics posted as PR comment
• Human review + automated gates → merge → deploy

Retraining Strategy

Retraining checklist:

• [ ] New data passes quality checks
• [ ] Feature pipeline unchanged (or updated and tested)
• [ ] New model beats current production on hold-out
• [ ] A/B test or shadow mode before full cutover
• [ ] Update model metadata and documentation

Cost Optimization

Tools & Frameworks

Environment Setup

Python Environment

# Recommended: uv or conda
uv venv .venv
source .venv/bin/activate

# Core stack
uv pip install pandas polars numpy scikit-learn xgboost

# Optional by use case
uv pip install torch transformers  # deep learning
uv pip install statsmodels scipy  # statistics
uv pip install mlflow optuna     # experiment tracking + HPO
uv pip install fastapi uvicorn    # API serving
uv pip install shap lime          # interpretability

R Environment

install.packages(c("tidyverse", "caret", "randomForest", "xgboost"))
install.packages(c("broom", "infer", "rsample"))  # tidy stats

Library Comparison

Data Manipulation

Machine Learning

Experiment Tracking

Code Patterns

Reproducible Experiment

import random
import numpy as np
from sklearn.model_selection import cross_val_score

def set_seed(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    # Add framework-specific seeds
    # torch.manual_seed(seed)
    # tf.random.set_seed(seed)

set_seed(42)

# Log everything
config = {
    "model": "XGBClassifier",
    "n_estimators": 100,
    "max_depth": 6,
    "seed": 42
}
# mlflow.log_params(config)

model = XGBClassifier(**config)
scores = cross_val_score(model, X, y, cv=5)
# mlflow.log_metric("cv_f1", scores.mean())

Pipeline Pattern

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer([
    ('num', StandardScaler(), numeric_features),
    ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features)
])

pipeline = Pipeline([
    ('preprocess', preprocessor),
    ('model', XGBClassifier())
])

# Prevents leakage: preprocessing fit only on training fold
pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)

Time Series Split

from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_squared_error

tscv = TimeSeriesSplit(n_splits=5)
scores = []

for train_idx, test_idx in tscv.split(X):
    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    scores.append(mean_squared_error(y_test, preds))

Cloud ML Services

Performance Tips

Pandas:

• Use category dtype for low-cardinality strings
• Vectorize with .apply() only as last resort
• Use .loc[] for assignment to avoid SettingWithCopy
• Chunk large files: pd.read_csv(file, chunksize=100000)

scikit-learn:

• Use n_jobs=-1 for parallelizable operations
• Prefer joblib over pickle for model serialization
• Use Pipeline to prevent data leakage

XGBoost/LightGBM:

• Use early_stopping_rounds to prevent overfitting
• Set tree_method='hist' for large datasets
• Use feature_name and feature_types for interpretability

Common Errors & Fixes