Data Science

Data Science Expert

Comprehensive data science frameworks for analytics, machine learning, and data-driven decision making.

Data Strategy

Data Maturity Model

Analytics Value Chain

DATA → INFORMATION → INSIGHT → ACTION → VALUE

PROGRESSION:
Descriptive: What happened?
Diagnostic: Why did it happen?
Predictive: What will happen?
Prescriptive: What should we do?
Autonomous: Self-optimizing systems

Statistical Analysis

Descriptive Statistics

CENTRAL TENDENCY:
- Mean: Sum / Count (sensitive to outliers)
- Median: Middle value (robust to outliers)
- Mode: Most frequent value

DISPERSION:
- Range: Max - Min
- Variance: Average squared deviation
- Standard Deviation: √Variance
- IQR: Q3 - Q1 (robust)

DISTRIBUTION SHAPE:
- Skewness: Asymmetry (0 = symmetric)
- Kurtosis: Tail heaviness (3 = normal)

For detailed inferential statistics and hypothesis testing, see Statistical Methods Reference.

Machine Learning

Algorithm Selection

For detailed ML pipelines, feature engineering, and model monitoring, see ML Pipelines Reference.

Data Governance

Data Governance Framework

GOVERNANCE PILLARS:

POLICIES:
- Data ownership
- Data classification
- Data retention
- Data access
- Data quality standards

ROLES:
- Data Owner: Accountable for data domain
- Data Steward: Day-to-day quality management
- Data Custodian: Technical implementation
- Data Consumer: End user

PROCESSES:
- Data cataloging
- Metadata management
- Data lineage
- Issue resolution
- Change management

METRICS:
- Data quality scores
- Policy compliance
- Data access requests
- Issue resolution time

Data Quality Dimensions

Business Intelligence

BI Architecture

ARCHITECTURE LAYERS:

DATA SOURCES:
- Operational systems
- External data
- IoT/streaming

DATA INTEGRATION:
- ETL/ELT pipelines
- Data lakes
- Data warehouses

SEMANTIC LAYER:
- Business definitions
- Calculated metrics
- Hierarchies
- Relationships

PRESENTATION:
- Dashboards
- Reports
- Ad-hoc analysis
- Embedded analytics

Dashboard Design Principles

DESIGN PRINCIPLES:

PURPOSE:
- One clear objective per dashboard
- Know your audience
- Enable decisions

LAYOUT:
- Most important top-left
- Related items grouped
- Progressive disclosure
- Whitespace for clarity

VISUALS:
- Right chart for data type
- Consistent formatting
- Minimal decoration
- Color with purpose

INTERACTIVITY:
- Filters for exploration
- Drill-down capability
- Cross-filtering
- Tooltip details

Metric Design

METRIC DEFINITION TEMPLATE:

NAME: [Metric name]
DEFINITION: [Clear business definition]
FORMULA: [Precise calculation]
OWNER: [Responsible person]
DATA SOURCE: [Where it comes from]
GRAIN: [Level of detail]
FREQUENCY: [Update cadence]
DIMENSIONS: [Slicing attributes]
TARGETS: [Goals/benchmarks]
RELATED: [Related metrics]

Predictive Modeling

Use Case Framework

Data Ethics & Privacy

Ethical AI Framework

PRINCIPLES:

FAIRNESS:
- No discriminatory outcomes
- Bias testing across groups
- Regular auditing

ACCOUNTABILITY:
- Clear ownership
- Decision audit trails
- Escalation process

TRANSPARENCY:
- Explainable decisions
- Clear documentation
- User communication

PRIVACY:
- Data minimization
- Consent management
- Security controls

Bias Detection

BIAS TYPES:

HISTORICAL: Reflects past discrimination
REPRESENTATION: Training data not representative
MEASUREMENT: Proxy variables correlate with protected attributes
AGGREGATION: Single model for diverse populations
EVALUATION: Inappropriate benchmarks

FAIRNESS METRICS:
- Demographic Parity: Equal positive rates
- Equalized Odds: Equal TPR and FPR
- Individual Fairness: Similar inputs, similar outputs
- Calibration: Equal accuracy across groups

Analytics Team Structure

Team Roles

Operating Models

References

• ML Pipelines Reference - Detailed ML pipeline, feature engineering, model development
• Statistical Methods Reference - Inferential statistics, hypothesis testing, evaluation metrics

See Also

• Fortune 50 Product Management
• Fortune 50 Business Strategy
• Fortune 50 Finance

Data Science Expert

This skill provides Fortune 50-caliber Data Science and Analytics expertise across the complete data lifecycle. Use this skill when working on statistical analysis, machine learning, business intelligence, data governance, predictive modeling, data visualization, or analytics strategy.

When to Use This Skill

Invoke this skill when the user needs help with:

• Designing and executing statistical analyses
• Building machine learning pipelines and models
• Establishing data governance frameworks
• Creating business intelligence dashboards and reports
• Developing predictive models for forecasting, classification, or clustering
• Designing effective data visualizations
• Ensuring data ethics, privacy, and compliance
• Defining analytics strategies and roadmaps
• Setting up experimentation and causal inference frameworks
• Building data infrastructure and architecture
• Creating data quality frameworks

Core Data Science Frameworks

1. Analytics Strategy & Roadmap

Analytics Maturity Model:

Level 1 - Descriptive (What Happened?):

• Characteristics: Backward-looking, manual reporting, siloed data
• Capabilities: Basic dashboards, historical reports, simple aggregations
• Tools: Spreadsheets, basic BI tools
• Organizational Readiness: Limited data culture, ad-hoc requests
• Next Steps: Centralize data, automate reporting, establish KPIs

Level 2 - Diagnostic (Why Did It Happen?):

• Characteristics: Root cause analysis, drill-downs, comparative analysis
• Capabilities: Interactive dashboards, segmentation, cohort analysis
• Tools: Intermediate BI tools, SQL databases
• Organizational Readiness: Data-aware, some self-service
• Next Steps: Build data warehouse, enable self-service BI, train analysts

Level 3 - Predictive (What Will Happen?):

• Characteristics: Forward-looking, statistical models, forecasting
• Capabilities: Predictive models, propensity scoring, trend analysis
• Tools: Statistical software, ML platforms, data science notebooks
• Organizational Readiness: Data-driven decisions, dedicated DS team
• Next Steps: Industrialize ML pipelines, A/B testing infrastructure

Level 4 - Prescriptive (What Should We Do?):

• Characteristics: Optimization, simulation, recommendation engines
• Capabilities: Real-time decisioning, automated actions, optimization algorithms
• Tools: Advanced ML platforms, production ML systems, optimization solvers
• Organizational Readiness: Data-first culture, embedded analytics
• Next Steps: Real-time ML, automated decision systems, continuous learning

Analytics Strategy Framework:

1. Business Alignment:

• Identify key business questions and decisions
• Map analytics capabilities to strategic priorities
• Define success metrics for analytics initiatives
• Estimate business value and ROI

2. Use Case Prioritization:

• Value: Business impact if successful (revenue, cost, risk)
• Feasibility: Data availability, technical complexity, timeline
• Data Readiness: Quality and completeness of required data
• Organizational Readiness: Stakeholder buy-in, change management
• Prioritization Matrix: Plot use cases on Value vs. Feasibility; start with high-value, high-feasibility

3. Capability Building:

• People: Hire/train analysts, data scientists, engineers, translators
• Process: Define workflows, governance, documentation standards
• Technology: Data infrastructure, tools, platforms
• Data: Quality, accessibility, integration, governance

4. Measurement & Iteration:

• Track analytics KPIs: adoption, time-to-insight, accuracy, business impact
• Regular retrospectives and continuous improvement
• Scale successful pilots, kill unsuccessful ones quickly

2. Statistical Analysis Frameworks

Hypothesis Testing Framework:

1. Formulate Hypotheses:

• Null Hypothesis (H₀): No effect or no difference (status quo)
• Alternative Hypothesis (H₁): Effect exists or difference is present
• Example: H₀: New feature has no effect on retention; H₁: New feature increases retention

2. Choose Test:

• Comparing Means:
• Two groups: Independent t-test (independent samples) or Paired t-test (related samples)
• Multiple groups: ANOVA (Analysis of Variance)
• Non-normal data: Mann-Whitney U test or Kruskal-Wallis test
• Comparing Proportions:
• Two proportions: Z-test for proportions or Chi-square test
• Multiple proportions: Chi-square test
• Correlation:
• Linear relationship: Pearson correlation
• Monotonic relationship: Spearman correlation
• Categorical association: Chi-square test

3. Set Significance Level (α):

• Standard: α = 0.05 (5% false positive rate)
• Conservative: α = 0.01 (1% false positive rate)
• Multiple comparisons: Apply Bonferroni correction (divide α by number of tests)

4. Calculate Test Statistic & P-Value:

• Test statistic measures how far observed data deviates from null hypothesis
• P-value: Probability of observing data this extreme if null hypothesis is true
• Interpretation:
• p < α: Reject null hypothesis (statistically significant)
• p ≥ α: Fail to reject null hypothesis (not statistically significant)

5. Assess Practical Significance:

• Effect Size Measures:
• Cohen's d (standardized mean difference): Small=0.2, Medium=0.5, Large=0.8
• Correlation coefficient (r): Small=0.1, Medium=0.3, Large=0.5
• Odds ratio or relative risk for proportions
• Business Impact: Translate statistical finding to business outcome
• Example: 5% conversion rate increase is statistically significant (p=0.01) with medium effect size (d=0.4), translating to $2M annual revenue

6. Check Assumptions:

• Normality: Q-Q plots, Shapiro-Wilk test
• Homogeneity of Variance: Levene's test
• Independence: Check for autocorrelation, ensure random sampling
• Sample Size: Power analysis to ensure adequate sample

Common Statistical Pitfalls:

• P-hacking: Running multiple tests until finding significance
• Stopping Early: Peeking at results and stopping when significant
• Cherry-Picking: Reporting only significant results
• Confusing Correlation and Causation: Association ≠ causation
• Ignoring Practical Significance: Statistically significant but trivial effect

Causal Inference Methods:

1. Randomized Controlled Trials (RCTs):

• Gold Standard: Random assignment eliminates confounding
• Design: Treatment and control groups, measure outcomes
• Analysis: Compare means/proportions between groups
• Limitations: Expensive, time-consuming, sometimes unethical/infeasible

2. Quasi-Experimental Methods:

• Difference-in-Differences (DiD):
• Compare change in treatment group vs. control group over time
• Requires parallel trends assumption (groups would have changed similarly without treatment)
• Example: Policy change in one state vs. another
• Regression Discontinuity Design (RDD):
• Treatment assigned based on cutoff (e.g., credit score > 700)
• Compare outcomes just above vs. just below cutoff
• Assumes continuity around cutoff
• Propensity Score Matching:
• Match treated units to similar control units based on covariates
• Balance observed confounders
• Does not address unobserved confounders

3. Instrumental Variables (IV):

• Use variable that affects treatment but not outcome (except through treatment)
• Addresses endogeneity and confounding
• Example: Distance to hospital as instrument for medical treatment

4. Synthetic Control:

• Create weighted combination of control units to match treatment unit pre-treatment
• Compare post-treatment outcomes
• Useful for single treated unit (e.g., policy change in one city)

3. Machine Learning Pipeline Design

ML Lifecycle Framework:

Phase 1: Problem Definition

• Business Objective: What decision will this model support?
• Success Metrics: How will we measure model success?
• Business metrics: Revenue impact, cost savings, user satisfaction
• Model metrics: Accuracy, precision, recall, AUC, RMSE, etc.
• Use Case Type: Classification, regression, clustering, ranking, recommendation
• Baseline: What's the current approach? What's naive baseline performance?

Phase 2: Data Preparation

1. Data Collection:

• Identify data sources (internal databases, APIs, third-party data)
• Assess data availability and accessibility
• Determine required sample size (more complex models need more data)
• Consider data labeling needs (supervised learning)

2. Exploratory Data Analysis (EDA):

• Univariate Analysis: Distributions, summary statistics, missing values
• Bivariate Analysis: Correlations, relationships with target variable
• Multivariate Analysis: Interactions, clustering, dimensionality
• Visualizations: Histograms, box plots, scatter plots, correlation matrices
• Key Questions:
• What's the distribution of the target variable?
• Are there outliers or anomalies?
• What features correlate with the target?
• Is there class imbalance?
• How much missing data?

3. Data Cleaning:

• Missing Values:
• Remove: If small percentage (<5%) and missing at random
• Impute: Mean/median (numerical), mode (categorical), model-based
• Flag: Create indicator variable for missingness
• Outliers:
• Investigate: Are they errors or valid extreme values?
• Remove: If data errors or invalid
• Transform: Log/sqrt transformation to reduce impact
• Cap: Winsorize to percentile thresholds
• Duplicates: Identify and remove exact or near-duplicates
• Data Types: Ensure correct types (numerical, categorical, datetime)

4. Feature Engineering:

• Transformations:
• Log/sqrt for skewed distributions
• Polynomial features for non-linear relationships
• Binning continuous variables
• Encoding:
• One-hot encoding for nominal categoricals
• Ordinal encoding for ordered categoricals
• Target encoding for high-cardinality categoricals
• Derived Features:
• Aggregations (sums, counts, averages)
• Ratios and differences
• Time-based features (day of week, hour, time since event)
• Domain-specific features
• Interactions: Product of features, especially for tree-based models
• Dimensionality Reduction: PCA, t-SNE for high-dimensional data

5. Data Splitting:

• Training Set (60-80%): Fit model parameters
• Validation Set (10-20%): Tune hyperparameters, select features
• Test Set (10-20%): Final evaluation, never used during development
• Stratification: Maintain class distribution across splits
• Time-Based Split: For time-series, use chronological split

Phase 3: Model Development

1. Model Selection:

Classification:

• Logistic Regression: Interpretable, good baseline, works well with many features
• Decision Trees: Non-linear, interpretable, handles mixed data types
• Random Forest: Robust, handles non-linearity, feature importance, less overfitting than single tree
• Gradient Boosting (XGBoost, LightGBM, CatBoost): High performance, handles complex patterns, risk of overfitting
• Neural Networks: Complex patterns, large datasets, less interpretable
• Support Vector Machines: Good for high-dimensional data, kernel trick for non-linearity

Regression:

• Linear Regression: Interpretable, fast, assumes linear relationship
• Ridge/Lasso: Regularized linear, handles multicollinearity, feature selection (Lasso)
• Decision Trees/Random Forest/Gradient Boosting: Handle non-linearity well
• Neural Networks: Complex non-linear relationships, large datasets

Clustering:

• K-Means: Fast, spherical clusters, need to specify k
• DBSCAN: Arbitrary shapes, automatic outlier detection, need to tune eps and min_samples
• Hierarchical: Dendrogram, no need to specify k upfront
• Gaussian Mixture Models: Probabilistic, soft clustering, assumes Gaussian distributions

Time Series:

• ARIMA: Linear time dependencies, interpretable
• Prophet (Facebook): Handles seasonality, holidays, missing data
• LSTM/GRU: Neural networks for complex patterns, long sequences
• Exponential Smoothing: Simple, good baseline

2. Training & Validation:

• Cross-Validation: k-fold (typically 5 or 10) to assess generalization
• Hyperparameter Tuning:
• Grid Search: Exhaustive search over parameter grid (slow but thorough)
• Random Search: Random combinations (faster, often as good)
• Bayesian Optimization: Intelligent search using past results
• Regularization: L1 (Lasso), L2 (Ridge), Elastic Net to prevent overfitting
• Early Stopping: Stop training when validation performance plateaus

3. Model Evaluation:

Classification Metrics:

• Accuracy: (TP + TN) / Total - Good for balanced classes
• Precision: TP / (TP + FP) - How many predicted positives are correct?
• Recall (Sensitivity): TP / (TP + FN) - How many actual positives did we catch?
• F1 Score: Harmonic mean of precision and recall
• AUC-ROC: Area under ROC curve - Overall classification performance across thresholds
• Confusion Matrix: Detailed breakdown of predictions vs. actuals
• Choose Based on Context:
• Fraud detection: High recall (catch all fraud)
• Spam filter: High precision (don't mark real email as spam)
• Balanced: F1 score

Regression Metrics:

• MAE (Mean Absolute Error): Average absolute difference, same units as target
• RMSE (Root Mean Squared Error): Square root of average squared difference, penalizes large errors
• R² (Coefficient of Determination): Proportion of variance explained (0-1, higher better)
• MAPE (Mean Absolute Percentage Error): Average percentage error, scale-independent

4. Model Interpretation:

• Feature Importance: Which features most influence predictions?
• Tree-based: Built-in feature importance
• Linear models: Coefficient magnitudes
• SHAP values: Unified approach across model types
• Partial Dependence Plots: How predictions change with one feature
• LIME (Local Interpretable Model-Agnostic Explanations): Explain individual predictions
• SHAP (SHapley Additive exPlanations): Game theory-based feature attribution

Phase 4: Deployment & Monitoring

1. Model Deployment Patterns:

• Batch Prediction: Scheduled predictions on static data (nightly scoring)
• Online Prediction: Real-time predictions via API
• Streaming: Continuous predictions on data streams
• Edge Deployment: Model on device (mobile, IoT)

2. Infrastructure:

• Model Serving: REST API, gRPC, or batch job
• Scaling: Handle traffic spikes, latency requirements
• Versioning: Track model versions, enable rollback
• A/B Testing: Gradually roll out new models

3. Monitoring:

• Data Drift: Input feature distributions changing over time
• Concept Drift: Relationship between features and target changing
• Performance Metrics: Track accuracy, latency, errors in production
• Alerts: Trigger when metrics exceed thresholds
• Retraining Triggers: Automate retraining when performance degrades

4. Feedback Loop:

• Collect ground truth labels for predictions
• Retrain models with new data
• Evaluate and redeploy improved models
• Continuous learning systems

ML Pipeline Best Practices:

• Reproducibility: Version control code, track data versions, log hyperparameters
• Modularity: Separate data prep, training, evaluation, deployment
• Testing: Unit tests for data processing, integration tests for pipeline
• Documentation: Document assumptions, decisions, trade-offs
• Ethics: Consider fairness, bias, privacy throughout

4. Business Intelligence & Dashboards

Dashboard Design Framework:

1. Define Purpose & Audience:

• Purpose: Monitoring, exploration, storytelling, decision-support
• Audience: Executives, managers, analysts, operational users
• Context: When and where will it be used? (daily standups, board meetings, real-time operations)

2. Identify Key Metrics:

• Primary Metrics: 1-3 most important (North Star, primary KPIs)
• Secondary Metrics: Supporting context (3-7 metrics)
• Dimensions: How to slice data (time, geography, segment, product)
• Avoid: Metric overload, vanity metrics, metrics without action

3. Dashboard Layout Principles:

• Visual Hierarchy: Most important metrics top-left, less important bottom-right
• Grouping: Related metrics together, use whitespace to separate
• Simplicity: Remove clutter, maximize data-ink ratio
• Consistency: Same metric types use same visualizations
• Responsiveness: Adapt to different screen sizes

4. Visualization Selection:

Comparison:

• Bar chart: Compare categories (horizontal for long labels)
• Column chart: Compare over time (chronological x-axis)
• Grouped/stacked bar: Compare subcategories

Trend/Time-Series:

• Line chart: Continuous time series, multiple metrics
• Area chart: Show magnitude over time, cumulative values
• Sparklines: Small, inline trends

Part-to-Whole:

• Pie chart: Only for 2-4 categories, avoid 3D
• Stacked bar/area: Parts over time
• Treemap: Hierarchical proportions

Distribution:

• Histogram: Frequency distribution
• Box plot: Summary statistics, outliers
• Violin plot: Distribution shape

Correlation:

• Scatter plot: Relationship between two variables
• Bubble chart: Three variables (x, y, size)
• Heatmap: Correlation matrix

Geospatial:

• Choropleth map: Color-coded regions
• Point map: Locations as dots
• Flow map: Movement between locations

5. Interactivity:

• Filters: Date range, segments, products
• Drill-down: Click to see more detail
• Tooltips: Show exact values on hover
• Linked Views: Selection in one chart filters others
• Avoid: Excessive interactivity that confuses users

6. Color & Design:

• Purpose-Driven Color:
• Categorical: Distinct hues for different categories
• Sequential: Light to dark for ordered data
• Diverging: Two colors for data around midpoint (e.g., negative/positive)
• Accessibility:
• Colorblind-friendly palettes
• Don't rely solely on color
• Sufficient contrast
• Branding: Company colors for consistency

7. Performance:

• Aggregation: Pre-aggregate data for large datasets
• Caching: Cache queries and results
• Incremental Refresh: Update only new data
• Lazy Loading: Load visuals as needed

BI Tool Selection Criteria:

• User Base: Analysts (SQL-based), business users (drag-and-drop), executives (curated dashboards)
• Data Sources: Variety and complexity of connections needed
• Deployment: Cloud vs. on-premise
• Cost: Licensing model, scalability
• Capabilities: Real-time, embedded analytics, mobile, AI features
• Examples: Tableau (exploration, ease of use), Power BI (Microsoft ecosystem), Looker (SQL-based, embedded), Metabase (open-source, simple)

5. Data Governance Framework

Data Governance Operating Model:

1. Governance Structure:

• Data Governance Council: Executive sponsors, strategic direction, policy approval
• Data Stewards: Domain experts, data quality ownership, business glossary
• Data Custodians: IT/engineering, technical implementation, access control
• Data Users: Consumers of data, feedback on quality and usability

2. Policies & Standards:

Data Quality Framework (Six Dimensions):

• Accuracy: Data correctly represents real-world entities
• Validation rules, cross-referencing, data profiling
• Completeness: All required data present
• Mandatory fields, completeness metrics, gap analysis
• Consistency: Data is uniform across systems
• Master data management, standardization, reconciliation
• Timeliness: Data is current and available when needed
• Freshness SLAs, update frequency, latency monitoring
• Validity: Data conforms to defined formats and rules
• Schema validation, business rules, referential integrity
• Uniqueness: No unintended duplicates
• Deduplication, primary keys, matching algorithms

Data Classification:

• Public: Can be freely shared (marketing materials)
• Internal: For internal use only (internal reports)
• Confidential: Restricted access (financial data, strategies)
• Restricted: Highly sensitive (PII, PHI, PCI)

3. Data Catalog:

• Metadata Management: Technical (schemas, data types), business (definitions, owners), operational (lineage, quality)
• Business Glossary: Standardized definitions, avoid ambiguity
• Data Lineage: Track data from source to consumption
• Discovery: Search and browse available datasets

4. Access Control:

• Role-Based Access Control (RBAC): Access based on role
• Attribute-Based Access Control (ABAC): Access based on attributes (department, clearance level)
• Data Masking: Obfuscate sensitive data for non-privileged users
• Audit Logs: Track who accessed what, when

5. Data Quality Monitoring:

• Automated Checks: Run validation rules on data pipelines
• Data Quality Dashboards: Track quality metrics over time
• Issue Management: Log, prioritize, and resolve quality issues
• Root Cause Analysis: Prevent recurrence

6. Master Data Management (MDM):

• Golden Records: Single source of truth for key entities (customers, products)
• Data Matching: Identify same entities across systems
• Data Merging: Consolidate duplicates, resolve conflicts
• Data Synchronization: Propagate changes across systems

6. Data Ethics & Privacy

Privacy Framework:

1. Privacy Principles:

• Data Minimization: Collect only what's necessary
• Purpose Limitation: Use data only for stated purposes
• Transparency: Inform users about data collection and use
• User Control: Enable access, correction, deletion
• Security: Protect data from unauthorized access
• Accountability: Assign responsibility for privacy

2. Regulatory Compliance:

GDPR (General Data Protection Regulation - EU):

• Scope: EU residents' data, regardless of company location
• Key Rights: Access, rectification, erasure, portability, restriction, objection
• Consent: Explicit, informed, revocable
• Data Protection Officer: Required for large-scale processing
• Breach Notification: Within 72 hours
• Penalties: Up to 4% of global revenue

CCPA/CPRA (California Consumer Privacy Act/Rights Act):

• Scope: California residents' data
• Key Rights: Know, delete, opt-out of sale, non-discrimination
• Do Not Sell: Opt-out mechanism required
• Penalties: Up to $7,500 per intentional violation

HIPAA (Health Insurance Portability and Accountability Act - US Healthcare):

• Scope: Protected Health Information (PHI)
• Key Requirements: Privacy rule, security rule, breach notification
• Business Associate Agreements: Third parties handling PHI
• Penalties: Up to $1.5M per violation category per year

Other Regulations: PCI-DSS (payment cards), COPPA (children's data), sector-specific regulations

3. Privacy-Preserving Techniques:

Anonymization:

• Remove or transform personally identifiable information
• Techniques: Suppression, generalization, perturbation
• Challenge: Re-identification risk from combinations of attributes

Pseudonymization:

• Replace identifiers with pseudonyms
• Reversible with key, less privacy protection than anonymization
• GDPR encourages as safeguard

Differential Privacy:

• Add statistical noise to data or query results
• Privacy Budget (ε): Lower ε = more privacy, less accuracy
• Formal privacy guarantee: Individual records don't significantly affect output
• Used by Apple, Google, US Census

Federated Learning:

• Train ML models on decentralized data (devices, organizations)
• Share model updates, not raw data
• Enables collaboration without sharing data

Homomorphic Encryption:

• Perform computations on encrypted data
• Results decrypt to same as computation on unencrypted data
• Computationally expensive, limited operations

4. Algorithmic Ethics & Fairness:

Bias Sources:

• Historical Bias: Training data reflects historical discrimination
• Representation Bias: Training data doesn't represent population
• Measurement Bias: Proxies used instead of true construct
• Aggregation Bias: Model doesn't account for group differences
• Evaluation Bias: Test data unrepresentative

Fairness Definitions:

• Demographic Parity: Equal positive prediction rates across groups
• Equal Opportunity: Equal true positive rates across groups (same recall)
• Equalized Odds: Equal TPR and FPR across groups
• Predictive Parity: Equal precision across groups
• Individual Fairness: Similar individuals receive similar predictions
• Note: Some definitions are mathematically incompatible; choose based on context

Bias Mitigation:

• Pre-Processing: Reweight or resample training data, transform features
• In-Processing: Constrain model during training, adversarial debiasing
• Post-Processing: Adjust decision thresholds by group
• Testing: Evaluate fairness metrics across demographic groups
• Auditing: Regular reviews, external audits

Transparency & Explainability:

• Model Cards: Document intended use, performance, fairness metrics
• Datasheets for Datasets: Document data collection, composition, biases
• Explainable AI: LIME, SHAP, attention mechanisms
• Human-in-the-Loop: Humans review high-stakes decisions

5. Data Ethics Checklist:

• [ ] Informed consent obtained for data collection
• [ ] Data collection limited to stated purposes
• [ ] Sensitive attributes protected or removed
• [ ] Model evaluated for fairness across demographic groups
• [ ] Potential harms identified and mitigated
• [ ] Transparency provided about automated decisions
• [ ] Human oversight for high-stakes decisions
• [ ] Opt-out mechanisms available
• [ ] Data retention policies defined and enforced
• [ ] Regular audits and reviews conducted

7. Advanced Analytics Techniques

1. Segmentation & Clustering:

K-Means Clustering:

• Process: Assign points to k clusters, minimize within-cluster variance
• Choosing k: Elbow method (plot inertia vs. k), silhouette score
• Strengths: Fast, scalable, simple
• Limitations: Assumes spherical clusters, sensitive to outliers, need to specify k

RFM Analysis (Recency, Frequency, Monetary):

• Purpose: Customer segmentation for marketing
• Metrics:
• Recency: Days since last purchase
• Frequency: Number of purchases
• Monetary: Total spend
• Process: Score each metric (1-5), combine into segments
• Segments: Champions (555), Loyal (X5X), At Risk (5XX with low recency), Lost (low scores)

Hierarchical Clustering:

• Agglomerative: Bottom-up (start with individual points, merge)
• Divisive: Top-down (start with all points, split)
• Dendrogram: Visualize hierarchy, choose cut height for clusters
• Linkage: Single (closest points), complete (farthest points), average, Ward (minimize variance)

2. Dimensionality Reduction:

PCA (Principal Component Analysis):

• Purpose: Reduce dimensions while retaining variance
• Process: Find orthogonal directions of maximum variance
• Interpretation: First PC captures most variance, subsequent PCs capture remaining variance
• Use Cases: Visualization, noise reduction, feature extraction
• Limitations: Linear, assumes variance = importance

t-SNE (t-Distributed Stochastic Neighbor Embedding):

• Purpose: Visualize high-dimensional data in 2D/3D
• Process: Preserve local structure, similar points stay close
• Use Cases: Exploratory visualization, cluster visualization
• Limitations: Non-deterministic, computationally expensive, only for visualization (not feature extraction)

UMAP (Uniform Manifold Approximation and Projection):

• Purpose: Dimensionality reduction for visualization and feature extraction
• Advantages over t-SNE: Faster, preserves global structure, deterministic
• Use Cases: Visualization, preprocessing for ML

3. Time Series Forecasting:

Components:

• Trend: Long-term increase or decrease
• Seasonality: Regular periodic fluctuations
• Cyclical: Long-term oscillations (not fixed period)
• Residual: Random noise

Decomposition:

• Additive: Y = Trend + Seasonality + Residual (constant seasonality)
• Multiplicative: Y = Trend × Seasonality × Residual (seasonality proportional to level)

Forecasting Methods:

Naive Methods:

• Mean: Forecast = historical average
• Naive: Forecast = last observed value
• Seasonal Naive: Forecast = same season last cycle
• Drift: Linear trend from first to last observation

ARIMA (AutoRegressive Integrated Moving Average):

• AR (p): Regression on past values
• I (d): Differencing to make stationary
• MA (q): Regression on past errors
• Process: Check stationarity, identify p/d/q (ACF/PACF plots), fit model, validate residuals

Exponential Smoothing:

• Simple: Weighted average with exponentially decreasing weights
• Holt: Add trend component
• Holt-Winters: Add seasonality component
• Automatic: Algorithms like ETS to select components

Prophet (Facebook):

• Components: Trend (piecewise linear or logistic), seasonality (Fourier series), holidays
• Strengths: Handles missing data, outliers, trend changes, holidays
• Use Cases: Business forecasting with irregular events

Machine Learning for Time Series:

• Feature Engineering: Lags, rolling statistics, date features
• Models: Random Forest, XGBoost, LSTM/GRU
• Cross-Validation: Time series split (no random shuffling)

Forecast Evaluation:

• MAE: Mean absolute error
• RMSE: Root mean squared error (penalizes large errors)
• MAPE: Mean absolute percentage error
• Forecast Horizon: Evaluate at different horizons (short-term vs. long-term accuracy)

4. Survival Analysis:

• Purpose: Model time until event (churn, conversion, failure)
• Censoring: Some observations haven't experienced event yet
• Kaplan-Meier: Non-parametric survival curve estimation
• Cox Proportional Hazards: Model effect of covariates on hazard rate
• Use Cases: Customer churn, equipment failure, clinical trials

5. Recommendation Systems:

Collaborative Filtering:

• User-Based: Recommend items liked by similar users
• Item-Based: Recommend items similar to what user liked
• Matrix Factorization: Decompose user-item matrix (SVD, ALS)
• Strengths: No item features needed, discovers latent preferences
• Limitations: Cold start (new users/items), sparsity

Content-Based Filtering:

• Approach: Recommend items with features similar to user's past preferences
• Strengths: No need for other users' data, explains recommendations
• Limitations: Limited discovery, requires rich item features

Hybrid Approaches:

• Combine collaborative and content-based
• Use deep learning to learn features and interactions

Evaluation:

• Offline: Precision@K, Recall@K, NDCG (ranking quality), coverage, diversity
• Online: CTR, conversion rate, engagement time, long-term retention

8. Data Visualization Best Practices

Visualization Principles:

1. Clarity:

• Clear title and axis labels
• Appropriate scale (don't truncate y-axis to exaggerate differences)
• Remove chart junk (unnecessary gridlines, 3D effects, decorations)
• Direct labeling instead of legends when possible

2. Accuracy:

• Use appropriate chart type for data
• Start y-axis at zero for bar charts (not required for line charts)
• Avoid dual y-axes unless absolutely necessary (confusing)
• Show uncertainty (confidence intervals, error bars)

3. Efficiency:

• Maximize data-ink ratio (data pixels / total pixels)
• Use small multiples for comparisons (same chart repeated for different groups)
• Simplify where possible without losing information

4. Aesthetics:

• Consistent color scheme
• Readable fonts (sans-serif for digital, sufficient size)
• Whitespace for breathing room
• Alignment and grid for structure

Common Visualization Mistakes:

• Pie Charts: Avoid for >4 categories, humans bad at comparing angles
• 3D Charts: Distort perception, add no information
• Dual Y-Axes: Confusing, can mislead
• Too Many Colors: Overwhelming, reduces clarity
• Chartjunk: Decorative elements that don't add information
• Misleading Scales: Truncated axes, inconsistent intervals

Storytelling with Data:

• Context: Why should audience care? What's the question?
• Conflict: What's the problem or insight?
• Resolution: What action should be taken?
• Structure: Beginning (setup), middle (evidence), end (conclusion)
• Annotations: Guide viewer's attention to key insights

9. Data Infrastructure & Architecture

Data Architecture Patterns:

1. Data Warehouse:

• Purpose: Centralized repository for structured data from multiple sources
• Characteristics: Schema-on-write, optimized for read queries, historical data
• Schema: Star schema (fact and dimension tables) or snowflake schema
• ETL: Extract from sources, Transform (clean, aggregate), Load into warehouse
• Use Cases: BI reporting, historical analysis, structured data
• Examples: Snowflake, Redshift, BigQuery, Synapse

2. Data Lake:

• Purpose: Store raw data in native format (structured, semi-structured, unstructured)
• Characteristics: Schema-on-read, flexible, scalable, low cost per GB
• Storage: Object storage (S3, Azure Blob, GCS)
• Use Cases: ML training data, data science exploration, unstructured data
• Challenges: Data swamp risk (ungoverned, poor quality)
• Examples: S3 + Athena, Azure Data Lake, GCS + BigQuery

3. Data Lakehouse:

• Purpose: Combine warehouse structure with lake flexibility
• Characteristics: ACID transactions, schema enforcement, unified batch and streaming
• Technologies: Delta Lake, Iceberg, Hudi on top of data lake storage
• Benefits: Single platform for BI and ML, lower cost than separate warehouse and lake

4. Lambda Architecture:

• Batch Layer: Process historical data (accurate, slow)
• Speed Layer: Process real-time data (fast, approximate)
• Serving Layer: Merge batch and speed views
• Use Cases: Real-time analytics with historical context
• Challenges: Complexity, duplicate logic

5. Kappa Architecture:

• Single Streaming Pipeline: Everything is a stream
• Reprocessing: Replay events from stream for corrections
• Simpler than Lambda: No batch/speed layer duplication
• Use Cases: Event-driven systems, real-time analytics

Data Pipeline Best Practices:

• Idempotency: Re-running same operation produces same result
• Incremental Processing: Process only new/changed data
• Data Quality Checks: Validate at ingestion and transformation steps
• Monitoring: Track latency, failures, data volumes
• Error Handling: Dead letter queues, retry logic, alerts
• Testing: Unit tests for transformations, integration tests for pipelines

10. Emerging Trends & Advanced Topics

1. AutoML (Automated Machine Learning):

• Purpose: Automate model selection, hyperparameter tuning, feature engineering
• Benefits: Lower barrier to ML, faster iteration, good baselines
• Limitations: Less control, interpretability, may not find novel approaches
• Tools: H2O AutoML, Google AutoML, DataRobot, Auto-sklearn

2. MLOps (Machine Learning Operations):

• Purpose: Productionize and maintain ML systems
• Components: Model versioning, CI/CD for models, monitoring, retraining automation
• Tools: MLflow, Kubeflow, SageMaker, Vertex AI
• Best Practices: Treat models as code, version data, monitor performance, automate

3. Causal AI:

• Purpose: Move beyond correlation to understand causation
• Methods: RCTs, quasi-experiments, causal graphs, do-calculus
• Use Cases: Policy evaluation, personalization, root cause analysis
• Tools: DoWhy, CausalML, EconML

4. Graph Analytics:

• Purpose: Analyze relationships and networks
• Use Cases: Social networks, fraud detection, recommendation, knowledge graphs
• Algorithms: PageRank, community detection, shortest path, centrality
• Databases: Neo4j, Amazon Neptune, TigerGraph

5. Real-Time Analytics:

• Purpose: Analyze data as it arrives
• Technologies: Kafka, Flink, Spark Streaming, Kinesis
• Use Cases: Fraud detection, personalization, monitoring, IoT
• Challenges: Latency, scalability, consistency

6. Edge Analytics:

• Purpose: Process data on devices (IoT, mobile) instead of cloud
• Benefits: Lower latency, reduced bandwidth, privacy
• Use Cases: Autonomous vehicles, industrial IoT, mobile ML
• Challenges: Limited compute, model size, update mechanisms

How to Apply This Skill

When helping with data science and analytics tasks:

1. Assess Maturity Level: Understand current analytics maturity and organization readiness 2. Start with Business Questions: Analytics should answer questions and drive decisions, not exist for its own sake 3. Choose Appropriate Methods: Match statistical/ML methods to problem type, data availability, interpretability needs 4. Emphasize Data Quality: Garbage in, garbage out - prioritize data quality over model complexity 5. Consider Ethics Early: Privacy, fairness, and transparency should be designed in, not bolted on 6. Focus on Actionability: Insights are useless if they don't drive decisions or actions 7. Iterate and Validate: Use experimentation to validate hypotheses, measure actual impact 8. Communicate Effectively: Tailor communication to audience (technical depth, visualizations, business framing) 9. Build for Production: Consider scalability, monitoring, maintenance from the start 10. Stay Pragmatic: Simple, interpretable models often outperform complex ones; choose simplest that works

Common Data Science Questions:

• "How do I prioritize analytics initiatives?" → Use Value vs. Feasibility framework
• "What model should I use?" → Consider problem type, data size, interpretability needs, start simple
• "How do I measure model performance?" → Choose metrics based on business context and class balance
• "Is my model biased?" → Evaluate fairness metrics across demographic groups, use bias mitigation techniques
• "How do I deploy this model?" → Consider latency, scale, monitoring needs; use MLOps best practices
• "How do I visualize this data?" → Match chart type to data type and question, follow visualization best practices
• "How do I ensure data privacy?" → Apply privacy frameworks, anonymization/pseudonymization, differential privacy
• "How do I know if this result is statistically significant?" → Use appropriate hypothesis test, check assumptions, assess practical significance

This skill embodies world-class Data Science and Analytics expertise. Apply it rigorously to drive data-informed decisions and create measurable business value.

Machine Learning Pipelines Reference

Detailed ML Pipeline

PIPELINE STAGES:

1. PROBLEM DEFINITION
   - Business objective
   - Success metrics
   - Constraints

2. DATA COLLECTION
   - Data sources
   - Sampling strategy
   - Labeling (if supervised)

3. DATA PREPARATION
   - Cleaning
   - Feature engineering
   - Train/validation/test split

4. MODEL DEVELOPMENT
   - Algorithm selection
   - Training
   - Hyperparameter tuning

5. EVALUATION
   - Performance metrics
   - Cross-validation
   - Error analysis

6. DEPLOYMENT
   - Integration
   - Monitoring
   - Maintenance

Feature Engineering Details

NUMERICAL FEATURES:
- Scaling (StandardScaler, MinMaxScaler)
- Log transformation (skewed data)
- Binning (continuous to categorical)
- Polynomial features
- Interaction terms

CATEGORICAL FEATURES:
- One-hot encoding
- Label encoding
- Target encoding
- Frequency encoding
- Embedding (high cardinality)

TEXT FEATURES:
- Bag of words
- TF-IDF
- Word embeddings (Word2Vec, GloVe)
- Transformers (BERT embeddings)

TIME FEATURES:
- Cyclical encoding (sin/cos)
- Lag features
- Rolling statistics
- Time since events

DOMAIN-SPECIFIC:
- Domain knowledge features
- Aggregations (customer-level from transactions)
- Ratios and differences

Model Development Process

MODEL LIFECYCLE:

1. DEFINE
   - Business problem
   - Success metrics
   - Constraints
   - Timeline

2. EXPLORE
   - Data assessment
   - EDA
   - Feature candidates
   - Baseline model

3. DEVELOP
   - Feature engineering
   - Model selection
   - Hyperparameter tuning
   - Validation

4. VALIDATE
   - Holdout test set
   - Business validation
   - Bias testing
   - Stress testing

5. DEPLOY
   - Production integration
   - A/B testing
   - Monitoring setup
   - Documentation

6. MONITOR
   - Performance tracking
   - Data drift detection
   - Model retraining
   - Feedback loop

Model Monitoring

MONITORING DIMENSIONS:

DATA DRIFT:
- Input feature distributions changing
- Detect with KS test, PSI
- Triggers: retraining

MODEL DRIFT:
- Model performance degrading
- Track prediction accuracy
- Triggers: investigation, retraining

CONCEPT DRIFT:
- Underlying relationships changing
- Harder to detect
- Requires domain knowledge

OPERATIONAL:
- Latency
- Error rates
- Resource usage
- API health

Statistical Methods Reference

Inferential Statistics

Hypothesis Testing

HYPOTHESIS TESTING PROCESS:

1. State hypotheses
   H0: Null hypothesis (no effect)
   H1: Alternative hypothesis (effect exists)

2. Choose significance level
   α = 0.05 (typically)

3. Calculate test statistic
   (depends on test type)

4. Find p-value
   P(observed | H0 true)

5. Make decision
   p < α → Reject H0
   p ≥ α → Fail to reject H0

COMMON ERRORS:
Type I (False Positive): Reject H0 when true
Type II (False Negative): Fail to reject when false

EFFECT SIZE:
Cohen's d = (M1 - M2) / SD_pooled
Small: 0.2, Medium: 0.5, Large: 0.8

Model Evaluation Metrics

CLASSIFICATION:

Accuracy = (TP + TN) / Total
Precision = TP / (TP + FP) - Of predicted positive, how many correct?
Recall = TP / (TP + FN) - Of actual positive, how many found?
F1 = 2 * (Precision * Recall) / (Precision + Recall)
AUC-ROC = Area under ROC curve

REGRESSION:

MAE = Mean(|actual - predicted|)
MSE = Mean((actual - predicted)²)
RMSE = √MSE
R² = 1 - (SS_res / SS_tot)
MAPE = Mean(|actual - predicted| / actual) * 100

RANKING:

Precision@K = Relevant in top K / K
Recall@K = Relevant in top K / Total Relevant
MAP = Mean of Average Precision across queries
NDCG = Normalized Discounted Cumulative Gain