Ml System Design Interview

ML System Design Interview

End-to-end ML pipeline design coaching for staff+ engineers. Covers the full arc from problem definition through production monitoring -- the scope expected at L6+ interviews at top-tier ML organizations.

This skill assumes 15+ years of ML/CV/AI/NLP experience. It does not teach fundamentals. It structures the knowledge you already have into the format interviewers reward.

---

When to Use

Use for:

• Practicing 45-minute ML system design rounds
• Structuring whiteboard presentations for recommendation, ranking, RAG, fraud, perception systems
• Analyzing serving architecture tradeoffs (batch vs online vs streaming)
• Identifying L6+ differentiation signals (problem ownership, org constraints, data flywheels)
• Reviewing and critiquing ML system design answers

NOT for:

• Coding interviews (use senior-coding-interview)
• Behavioral / leadership questions (use interview-loop-strategist)
• ML theory or math derivations
• Implementing models or writing training code
• Paper reading or research review

---

The 7-Stage Design Framework

Every ML system design answer follows this arc. The stages are sequential but you will loop back as constraints emerge. The Mermaid diagram below is your whiteboard skeleton.

flowchart TD
    R[1. Requirements\n- Business goal\n- Users and scale\n- Latency/throughput SLA\n- Constraints] --> M[2. Metrics\n- Offline: precision, recall, NDCG\n- Online: CTR, conversion, revenue\n- Guardrails: latency p99, fairness]
    M --> D[3. Data\n- Sources and collection\n- Labeling strategy\n- Pipeline: ETL, validation\n- Freshness and staleness]
    D --> F[4. Features\n- Engineering and transforms\n- Feature store architecture\n- Online vs offline features\n- Freshness requirements]
    F --> Mo[5. Model\n- Architecture selection\n- Training pipeline\n- Iteration strategy\n- Baseline and ablation]
    Mo --> S[6. Serving\n- Batch vs online vs streaming\n- Caching and precomputation\n- Scaling and cost\n- Canary and shadow mode]
    S --> Mon[7. Monitoring\n- Data drift detection\n- Model degradation alerts\n- A/B testing framework\n- Rollback strategy\n- Feedback loops]
    Mon -.->|Feedback loop| D
    Mon -.->|Retrain trigger| Mo

Stage Details

Stage 1 -- Requirements (5 minutes) Ask clarifying questions before designing anything. Establish: Who is the user? What is the business metric? What is the latency SLA? What scale (QPS, data volume)? What are hard constraints (cost, privacy, regulation)? An L6+ candidate owns the problem definition -- do not wait for the interviewer to hand you requirements.

Stage 2 -- Metrics (3 minutes) Define offline metrics that you can measure before deployment AND online metrics that matter to the business. Explain the gap: "NDCG improvement offline does not always translate to CTR lift online because of position bias and novelty effects." Define guardrail metrics: latency p99, fairness across user segments, cost per prediction.

Stage 3 -- Data (7 minutes) Where does training data come from? How is it labeled (human, weak supervision, implicit signals)? What is the class balance? How fresh does data need to be? What is the data pipeline (batch ETL vs streaming)? What data quality checks exist? This stage separates L6+ candidates from L5 -- junior candidates assume clean labeled data.

Stage 4 -- Features (5 minutes) What features does the model need? Which are precomputed (offline) vs computed at request time (online)? Feature store architecture: online store (low-latency lookups) vs offline store (batch training). Feature freshness: user features update daily, item features update hourly, contextual features are real-time.

Stage 5 -- Model (8 minutes) Start with a simple baseline (logistic regression, XGBoost) and explain why. Then propose the production architecture (two-tower, transformer, etc.) and justify the upgrade. Discuss training pipeline: how often, how much data, how to handle distribution shift. Iteration strategy: what experiments to run first.

Stage 6 -- Serving (8 minutes) This is where system design and ML intersect. Discuss: inference latency requirements, batch precomputation vs online inference, GPU/CPU tradeoffs, model serving framework, caching strategy, cost optimization (quantization, distillation, spot instances). Draw the serving architecture.

Stage 7 -- Monitoring (5 minutes) What happens after deployment? Data drift detection (PSI, KL divergence). Model degradation alerts (metric decay over time). A/B testing framework (sample size, duration, novelty effects). Rollback strategy (shadow mode, canary percentage). Feedback loops that improve the model over time.

---

45-Minute Time Budget

If the interviewer cuts in with questions, adapt -- but cover all 7 stages even briefly. Skipping monitoring is the most common L5 mistake.

---

Canonical Problem Set

---

Serving Architecture Comparison

Detailed serving tradeoffs, framework comparisons, and cost optimization strategies are in references/serving-tradeoffs.md.

---

L6+ Differentiation Signals

What separates a staff+ answer from a senior answer:

1. Own the Problem Definition Do not accept the problem as stated. Ask: "What business metric are we optimizing? Is this a revenue problem or an engagement problem? What is the current solution and why is it insufficient?" L5 candidates accept "build a recommendation system." L6+ candidates ask "what are we recommending, to whom, and what does success look like?"

2. Discuss Organizational Constraints Real systems live inside organizations. Address: team size (can we maintain a custom model or should we use a managed service?), on-call burden, cross-team data dependencies, compliance requirements, migration path from legacy system.

3. Data Flywheel Strategy Show that you think about the virtuous cycle: better model -> more engagement -> more data -> better model. Discuss how to accelerate it: active learning, implicit feedback loops, exploration strategies, cold-start bootstrapping.

4. Build vs Buy Decisions Not everything should be custom. Argue for managed services where appropriate (embedding APIs, feature stores, serving platforms) and custom solutions where competitive advantage demands it. Show you understand the total cost of ownership.

5. Multi-Objective Thinking Real systems optimize multiple objectives simultaneously: relevance AND diversity, accuracy AND fairness, quality AND latency. Discuss how to handle conflicts: Pareto optimization, constrained optimization, multi-task learning, business-rule post-processing.

---

Whiteboard Strategy

What to draw and when:

Use boxes for components, arrows for data flow, and annotate with latency/throughput numbers. The diagram should be readable by someone who walks in at minute 30.

---

Anti-Patterns

Model-First Thinking

Novice: Jumps to "I would use a transformer" or "Let me describe the attention mechanism" in the first 2 minutes, before understanding the problem, defining metrics, or discussing data. Spends 70% of time on model architecture and 0% on serving.

Expert: Spends the first 10 minutes on requirements, metrics, and data before mentioning any model. Names a simple baseline first (logistic regression on handcrafted features), then argues for complexity only when the baseline's limitations are clear. Allocates equal time to serving and monitoring.

Detection: Architecture diagram has a detailed model box but no data pipeline, no feature store, no serving layer, and no monitoring component. Mentions model architecture in the first sentence.

Ignoring the Data

Novice: Assumes clean, labeled data exists at scale. Says "we would train on millions of labeled examples" without discussing where labels come from, how much they cost, what the class distribution looks like, or how stale the data gets.

Expert: Asks about data sources, labeling strategy (human vs weak supervision vs implicit signals), class imbalance handling, data freshness SLA, and data quality monitoring. Discusses the cost of labeling and proposes strategies to reduce it (active learning, semi-supervised methods, synthetic data).

Detection: No discussion of data collection, labeling costs, class imbalance, data quality checks, or data freshness anywhere in the answer. The word "label" does not appear.

No Monitoring Story

Novice: Design ends at the serving layer. No mention of what happens after the model is deployed. Does not discuss how to detect degradation, how to roll back, or how to improve the model over time.

Expert: Discusses data drift detection (population stability index, feature distribution monitoring), model performance decay alerts, A/B testing framework with proper statistical rigor, canary deployment strategy, shadow mode for safe rollouts, and explicit feedback loops that flow data back into retraining.

Detection: Architecture diagram has no monitoring component. No feedback arrows from production back to training. No mention of A/B testing, canary deployment, or rollback.

---

Reference Files

Consult these for deep dives -- they are NOT loaded by default:

Evaluation Metrics Guide

Comprehensive reference for choosing, computing, and aligning ML metrics. Covers offline metrics, online metrics, metric alignment problems, A/B testing, and generative AI evaluation.

---

Offline Metrics

Classification Metrics

Ranking Metrics

Regression / Forecasting Metrics

NLP / Generation Metrics

---

Online Metrics

Engagement Metrics

Business Metrics

System Metrics (Guardrails)

---

Metric Alignment Problem

The gap between offline metrics and online metrics is the central challenge of ML evaluation. Improving offline metrics does NOT guarantee online metric improvement.

Why Offline Metrics Fail to Predict Online Performance

Closing the Alignment Gap

1. Use multiple offline metrics: Never rely on a single metric. Track precision AND recall AND NDCG. 2. Evaluate on fresh data: Use time-based splits (train on past, evaluate on future) not random splits. 3. Counterfactual evaluation: Use logged data with inverse propensity weighting to estimate what would happen with a different policy. 4. Interleaving tests: Faster signal than A/B testing for ranking comparisons. Requires less traffic. 5. Guardrail metrics: Define metrics that must NOT regress even if primary metric improves. 6. Composite metrics: Combine multiple signals into a single optimization target (e.g., 0.6watch_time + 0.3satisfaction - 0.1*regret).

---

A/B Testing for ML

Sample Size Calculation

Before running any A/B test, calculate required sample size:

Inputs needed:

• Baseline metric value (e.g., 3.5% conversion rate)
• Minimum detectable effect (MDE) (e.g., 0.2% absolute lift = ~5.7% relative lift)
• Significance level (alpha, typically 0.05)
• Power (1-beta, typically 0.80)

Rule of thumb: For a 1% relative lift in a metric with 5% baseline, you need ~1.6M samples per variant. For a 5% relative lift, ~64K samples per variant.

Practical implication: Small metric improvements require massive traffic. A 0.1% CTR improvement on 2% CTR baseline needs millions of impressions to detect.

Duration Considerations

Common A/B Testing Mistakes in ML

1. Peeking at results Checking results daily and stopping when significant inflates false positive rate from 5% to 20-30%. Use sequential testing (always-valid p-values) if you must peek.

2. Running too many experiments simultaneously Multiple experiments on the same users create interaction effects. Use layer-based experimentation (Google's Overlapping Experiment Infrastructure).

3. Wrong randomization unit For ML systems: randomize by user, not by request. Same user should always see the same variant. Exception: latency experiments can randomize by request.

4. Ignoring long-term effects ML model improves short-term CTR but degrades long-term user satisfaction. Run holdback groups (1% of users never see new model) for long-term measurement.

5. Survivorship bias Only measuring users who complete a session ignores users who bounced. Include all randomized users in analysis.

6. Multiple comparison correction Testing 20 segments for significance without correction means ~1 will be "significant" by chance. Use Bonferroni correction or FDR control.

ML-Specific A/B Testing Challenges

Label delay: For fraud detection, true labels arrive 30-90 days after prediction. Cannot measure model quality in real-time. Use proxy metrics (review rate, rule trigger rate) with known correlation to true fraud rate.

Cold-start interaction: New model has no personalization for existing users. First-session performance may not reflect steady-state. Use burn-in period.

Model-data feedback loop: A/B testing a new recommendation model changes what data you collect. Control model's data quality degrades over time if fewer users see it. Time-bound experiments.

---

Evaluating Generative AI

Human Evaluation Frameworks

Side-by-Side Comparison (SxS) Show human raters output from Model A and Model B for same input. Rater chooses: A better, B better, or tie. Requires 200-500 comparisons for statistical significance. Expensive but gold standard.

Likert Scale Rating Rate each output independently on 1-5 scale across dimensions:

• Relevance (does it answer the question?)
• Faithfulness (is it supported by the context?)
• Completeness (does it cover all aspects?)
• Coherence (is it well-organized and readable?)
• Harmlessness (does it avoid harmful content?)

Task Completion Measure whether the generated output actually solves the task. Most objective but requires well-defined tasks with verifiable outcomes.

LLM-as-Judge

Use a capable LLM to evaluate another model's output. Faster and cheaper than human evaluation. Increasingly standard.

Setup:

• Judge model should be more capable than evaluated model (or at least different)
• Provide clear evaluation criteria in the judge prompt
• Use structured output (1-5 score + reasoning)
• Calibrate against human judgments on a sample

Biases to mitigate:

• Position bias: judge prefers the first response in SxS. Randomize order.
• Verbosity bias: judge prefers longer responses. Instruct to evaluate quality, not length.
• Self-preference bias: judge prefers outputs similar to its own style. Use different model families.

Validation:

• Compute agreement rate between LLM-judge and human judges
• Expect 70-85% agreement for well-calibrated judges
• Track cases where LLM-judge disagrees with humans -- these are your blind spots

Factuality and Hallucination Metrics

Recommended approach for RAG systems: 1. Extract claims from the generated answer (LLM call) 2. For each claim, check if any retrieved chunk supports it (NLI model or LLM) 3. Compute faithfulness score = supported claims / total claims 4. Flag unsupported claims for human review

RAG-Specific Evaluation

Evaluation frameworks: RAGAS, TruLens, DeepEval, Phoenix (Arize). All automate the above metrics.

---

Metric Decision Tree

Use this to choose metrics for your ML system design interview answer.

flowchart TD
    Start[What is the task?] --> Class{Classification?}
    Class -->|Yes| Imbalance{Class Imbalance?}
    Imbalance -->|Yes| AUCPR[AUC-PR + F-beta]
    Imbalance -->|No| AUCROC[AUC-ROC + F1]
    Class -->|No| Rank{Ranking?}
    Rank -->|Yes| Graded{Graded Relevance?}
    Graded -->|Yes| NDCG[NDCG@K]
    Graded -->|No| SingleAnswer{Single Correct Answer?}
    SingleAnswer -->|Yes| MRR_M[MRR]
    SingleAnswer -->|No| MAP_M[MAP + Precision@K]
    Rank -->|No| Gen{Generation?}
    Gen -->|Yes| HasRef{Has Reference Text?}
    HasRef -->|Yes| RefMetrics[BERTScore + ROUGE]
    HasRef -->|No| RAG{RAG / Grounded?}
    RAG -->|Yes| RAGMetrics[Faithfulness + Context Relevance\n+ Human Eval SxS]
    RAG -->|No| HumanEval[LLM-as-Judge + Human SxS]
    Gen -->|No| Regression{Regression?}
    Regression -->|Yes| Outliers{Outliers Matter?}
    Outliers -->|Yes| RMSE_M[RMSE]
    Outliers -->|No| MAE_M[MAE]

Pairing Offline with Online Metrics

---

Quick Reference: Formulas

Precision: TP / (TP + FP)

Recall: TP / (TP + FN)

F1: 2 Precision Recall / (Precision + Recall)

NDCG@K: DCG@K / IDCG@K, where DCG@K = sum(rel_i / log2(i+1)) for i in 1..K

MRR: 1/|Q| * sum(1/rank_i) for each query

MAP: mean of average precisions across queries

AUC-ROC: Probability that model scores a random positive higher than a random negative

Log Loss: -1/N sum(ylog(p) + (1-y)*log(1-p))

PSI (Population Stability Index): sum((actual% - expected%) * ln(actual% / expected%)) -- used for data drift detection. PSI < 0.1 = no significant shift, 0.1-0.2 = moderate, >0.2 = significant.

ML System Design Templates

Six fully worked ML system designs. Each follows the 7-stage framework from SKILL.md. Use these as skeletons for practice -- adapt, do not memorize.

---

1. YouTube-Scale Video Recommendation System

Requirements

• Recommend videos to 2B+ monthly active users
• Homepage feed, "Up Next" sidebar, notifications
• Optimize for watch time (primary), satisfaction (guardrail via surveys), diversity (guardrail)
• Latency: <200ms for homepage, <100ms for "Up Next"
• Must handle cold-start for new users and new videos

Metrics

Data Pipeline

• Implicit signals: watch time, completion rate, likes, shares, skips, scroll-past
• Explicit signals: thumbs up/down, "not interested", survey responses
• Video features: title embeddings, visual embeddings (from frames), audio features, metadata (category, creator, duration, upload date)
• User features: watch history, search history, demographics, device, time-of-day
• Pipeline: Kafka streaming for real-time events -> Spark for batch aggregation -> Feature store (online: Redis, offline: Hive)

Feature Engineering

• User embedding: learned from interaction history (updated daily)
• Video embedding: multi-modal (text + visual + audio, updated on upload)
• Cross features: user-category affinity, time-of-day preference, device-format preference
• Real-time features: session watch history, current time, user location
• Feature freshness: user features daily, video features on upload, context features real-time

Model Architecture

flowchart LR
    subgraph Retrieval["Retrieval (1000 candidates)"]
        TT[Two-Tower Model\nUser Tower + Video Tower\nANN Index: HNSW]
    end
    subgraph Ranking["Ranking (score 1000)"]
        DR[Deep Ranking Model\nWide & Deep / DCN-V2\nMulti-task: watch time +\nsatisfaction + engagement]
    end
    subgraph Reranking["Reranking (top 50)"]
        RR[Business Rules +\nDiversity Injection +\nFreshness Boost +\nCreator Fairness]
    end
    TT --> DR --> RR --> Feed[Final Feed]

• Retrieval: Two-tower with approximate nearest neighbor (HNSW via FAISS). User tower encodes user features + recent history. Video tower encodes video features. Inner product scoring. Retrain daily.
• Ranking: Deep cross network (DCN-V2) with multi-task heads for watch time prediction, satisfaction prediction, and engagement prediction. Combines with Wide component for memorization. Retrain every 6 hours on streaming data.
• Reranking: Post-processing layer for diversity (MMR), freshness boost, creator fairness constraints, and business rules (e.g., promote Shorts, suppress near-duplicates).

Serving Architecture

• Retrieval: precomputed user/video embeddings, ANN index updated hourly, served via custom C++ service
• Ranking: online inference on GPU cluster (TensorRT), batched requests, <50ms p99
• Feature serving: Redis cluster for online features (<5ms), Hive for offline training features
• Caching: prediction cache for returning users (invalidate on new watch event), embedding cache (TTL 1 hour)
• Scale: ~500K QPS peak, sharded by user ID hash

Monitoring

• Data drift: monitor feature distributions daily (PSI threshold 0.1)
• Model metrics: track NDCG on holdout set hourly, watch time per session in real-time
• A/B testing: 1% canary -> 5% ramp -> 50/50 split, minimum 7-day test, account for novelty effect
• Rollback: shadow mode for new models, instant rollback via feature flag, 5-minute detection latency
• Feedback loop: user interactions feed back into training data within 24 hours

---

2. Real-Time Search Ranking (E-Commerce)

Requirements

• Rank product search results for 100M+ daily queries
• Optimize for conversion rate (primary) and revenue per search (secondary)
• Latency: <50ms total (retrieval + ranking)
• Handle query intent: navigational, informational, transactional
• Support personalization, seasonal trends, real-time inventory

Metrics

Data Pipeline

• Query logs: query text, clicked products, purchased products, dwell time, add-to-cart events
• Product catalog: title, description, images, price, category, brand, stock level, seller rating
• User signals: purchase history, browsing history, wishlist, location, device
• Real-time: inventory levels, price changes, flash sales, trending queries
• Pipeline: Kafka for click streams -> Flink for real-time aggregation -> Elasticsearch for retrieval -> Feature store for ML features

Feature Engineering

• Query features: query embedding (BERT-based), query intent classifier, query frequency, seasonal signal
• Product features: product embedding, price percentile in category, review score, conversion rate, return rate, stock level
• Query-product features: BM25 score, semantic similarity, historical CTR for query-product pair
• User features: category affinity, price sensitivity, brand preference, purchase recency
• Real-time: session clicks, cart contents, time since last purchase

Model Architecture

flowchart TD
    Q[Query] --> IC[Intent Classifier\nNavigational / Informational /\nTransactional]
    Q --> ES[Elasticsearch\nBM25 + Embedding Retrieval\n1000 candidates]
    ES --> L1[L1 Ranker\nLightweight GBDT\n1000 -> 100 candidates\n<10ms]
    L1 --> L2[L2 Ranker\nCross-Encoder Transformer\n100 -> 30 candidates\n<30ms]
    IC --> L2
    L2 --> PP[Post-Processing\nDiversity, Sponsored,\nInventory Filter]
    PP --> Results[Search Results Page]

• Retrieval: Elasticsearch with BM25 + dense embedding retrieval (hybrid). 1000 candidates in <10ms.
• L1 Ranker: LightGBM on precomputed features. Scores 1000 candidates in <10ms. Features: BM25 score, semantic similarity, product popularity, price, reviews.
• L2 Ranker: Cross-encoder (DistilBERT fine-tuned) for query-product relevance. Scores 100 candidates with full attention. Multi-task: relevance + purchase probability.
• Post-processing: Diversity injection (no more than 3 results from same brand), sponsored placement, out-of-stock demotion.

Serving Architecture

• Retrieval: Elasticsearch cluster with custom plugin for hybrid search
• L1: CPU inference, model cached in memory, feature lookup from Redis
• L2: GPU inference (Triton), batched across concurrent queries, quantized INT8
• Feature store: Redis for real-time features, DynamoDB for user features
• Caching: query-level cache for popular queries (LRU, 15-min TTL), embedding cache for products
• Scale: 3K QPS average, 15K QPS peak (flash sales)

Monitoring

• Relevance: daily NDCG on human-judged query sets (100 queries, 3 judges per query)
• Business: conversion rate by query intent, revenue per search, zero-result rate
• Latency: p50/p95/p99 per ranking stage, alert on p99 >50ms
• A/B testing: interleaving for ranking comparison, minimum 3-day test
• Feedback loop: click-through and purchase data feed L1/L2 retraining weekly

---

3. Content Moderation Pipeline (Text + Image + Video)

Requirements

• Moderate user-generated content across text, images, and video
• Categories: hate speech, violence, nudity, spam, misinformation, self-harm
• Precision >95% for auto-removal (minimize false positives), recall >90% for review queue
• Latency: text <100ms, image <500ms, video <5s (for first-frame check, async for full)
• Handle adversarial evasion (Unicode tricks, steganography, text-in-image)
• Support appeals workflow with human review

Metrics

Data Pipeline

• Training data: human-labeled content (internal + vendor), synthetic adversarial examples, public datasets (HatEval, MMHS150K)
• Labeling: 3-annotator consensus, specialist reviewers for edge cases, regular calibration sessions
• Data challenges: label subjectivity (cultural context), class imbalance (harmful content is <1%), evolving policy
• Pipeline: content upload -> feature extraction -> model inference -> decision -> (optional) human review -> label back to training

Model Architecture

flowchart TD
    Upload[Content Upload] --> Router{Content Type}
    Router -->|Text| TE[Text Encoder\nFine-tuned RoBERTa\nMulti-label Classification]
    Router -->|Image| IE[Image Encoder\nCLIP + Custom CNN\nMulti-label Classification]
    Router -->|Video| VE[Video Pipeline\nKeyframe Extraction +\nImage Model + Audio Model +\nTemporal Aggregation]
    TE --> Ensemble[Ensemble + Rules Engine\nCategory-specific thresholds\nHigh-confidence auto-decide\nLow-confidence -> human review]
    IE --> Ensemble
    VE --> Ensemble
    Ensemble -->|High confidence harmful| Remove[Auto-Remove\n+ Notify User]
    Ensemble -->|Low confidence| Queue[Human Review Queue\nPrioritized by severity]
    Ensemble -->|High confidence safe| Publish[Publish Content]
    Queue --> Decision[Human Decision]
    Decision -->|Label| Retrain[Retrain Pipeline]

• Text: Fine-tuned RoBERTa for multi-label classification. Handles: hate speech, threats, spam, self-harm. Adversarial robustness via character-level augmentation and homoglyph normalization.
• Image: CLIP embeddings + custom CNN head for NSFW/violence/hate-symbol detection. Perceptual hashing for known-bad content matching.
• Video: Keyframe extraction (1 FPS + scene change detection), per-frame image model, audio transcription + text model, temporal aggregation for context.
• Rules engine: Policy-specific thresholds per category, auto-escalation for high-severity (CSAM, terrorism), geographic policy variations.

Serving Architecture

• Text: CPU inference, batched, <100ms p99
• Image: GPU inference (TorchServe), precomputed hash lookup for known-bad content (<10ms), model inference for unknown content (<500ms)
• Video: async pipeline, first-frame check synchronous (<5s), full video analysis async (<5min)
• Scaling: auto-scale GPU pods on queue depth, priority queue for reported content
• Caching: perceptual hash database (billions of entries), embedding similarity cache for near-duplicates

Monitoring

• Precision tracking: sample 1% of auto-removed content for human review weekly
• Recall tracking: monitor appeal rate and appeal success rate
• Adversarial testing: red team generates evasion attempts monthly, measure detection rate
• Policy drift: new policy additions trigger retraining within 48 hours
• Regional monitoring: per-region precision/recall because policies vary

---

4. Enterprise RAG System (with Evaluation)

Requirements

• Answer questions over enterprise knowledge base (100K+ documents, 10M+ chunks)
• Sources: Confluence, Google Docs, Slack, Jira, code repositories, PDFs
• Accuracy: cited answers with verifiable sources, hallucination rate <5%
• Latency: <3s for answer generation (streaming first token <500ms)
• Support multi-turn conversation with context carryover
• Handle access control (user can only see documents they have permission for)

Metrics

Data Pipeline

• Ingestion: connectors for Confluence, Google Workspace, Slack, Jira, GitHub, S3 (PDFs)
• Chunking strategy: semantic chunking (respect section boundaries) + sliding window overlap, chunk size 512-1024 tokens
• Embedding: text-embedding-3-large (OpenAI) or E5-large-v2 (self-hosted), re-embed on document update
• Index: vector database (Pinecone/Weaviate/Qdrant) + BM25 index (Elasticsearch) for hybrid search
• Access control: document-level ACL stored alongside vectors, filtered at query time
• Refresh: webhook-triggered re-indexing on document update, full re-index weekly

Feature Engineering

• Query features: query embedding, query type classifier (factual, procedural, comparative), entity extraction
• Document features: document embedding, recency score, authority score (based on author/source), access frequency
• Retrieval features: BM25 score, semantic similarity, document recency, source diversity
• Conversation features: conversation history embedding, topic continuity score

Model Architecture

flowchart TD
    Q[User Query + Conversation History] --> QR[Query Rewriting\nResolve pronouns,\nexpand abbreviations]
    QR --> HybridSearch[Hybrid Search\nSemantic: HNSW ANN\nLexical: BM25\nRRF Fusion]
    HybridSearch --> ACL[Access Control Filter\nUser permissions check]
    ACL --> Rerank[Cross-Encoder Reranker\ncohere-rerank-v3 or\nBGE-reranker-v2-m3\nTop 20 -> Top 5]
    Rerank --> Gen[LLM Generation\nSystem prompt with\ncitation instructions\nStreaming response]
    Gen --> FC[Faithfulness Check\nVerify claims against\nretrieved chunks\nFlag unsupported claims]
    FC --> Response[Response with\nInline Citations\n+ Source Links]

• Query rewriting: LLM rewrites multi-turn queries into standalone queries. Resolves "it", "that", abbreviations.
• Hybrid search: Reciprocal Rank Fusion of semantic (embedding similarity) and lexical (BM25) retrieval. 50-100 candidates.
• Reranking: Cross-encoder model scores query-document pairs with full attention. Top 20 -> top 5. Critical for precision.
• Generation: GPT-4o / Claude with system prompt enforcing citation format. Streaming for responsiveness.
• Faithfulness check: Post-generation verification that each claim is supported by a retrieved chunk. Flag or remove unsupported claims.

Serving Architecture

• Embedding service: batch embedding for ingestion, online embedding for queries (<100ms)
• Vector DB: managed service (Pinecone/Weaviate), ~50ms for ANN search
• Reranker: GPU inference, batched, <200ms for 20 candidates
• LLM: API call with streaming, first token <500ms
• Caching: query embedding cache, popular query answer cache (TTL 1 hour, invalidate on source doc update)
• Total latency budget: query rewrite (100ms) + search (100ms) + rerank (200ms) + generation (streaming from 500ms)

Monitoring

• Retrieval quality: weekly human evaluation of retrieval relevance on 50 query sample
• Answer quality: LLM-as-judge for faithfulness (automated daily), human evaluation weekly
• Hallucination monitoring: automated claim extraction + verification pipeline, alert on >5% hallucination rate
• User feedback: thumbs up/down per answer, track satisfaction trend
• Index freshness: monitor document staleness, alert on stale sources cited
• A/B testing: test retrieval strategies, reranker models, generation prompts independently

---

5. Real-Time Fraud Detection

Requirements

• Detect fraudulent transactions in real-time for a payment platform
• Transaction volume: 10K TPS peak, 3K TPS average
• Latency: <100ms per transaction (decision before authorization)
• Precision >99% for auto-block (minimize false positives on legitimate transactions)
• Recall >85% for fraud (catch most fraud, accept some false negatives for review)
• Handle adversarial adaptation (fraudsters change tactics)

Metrics

Data Pipeline

• Transaction data: amount, merchant, category, location, timestamp, device, IP, card info
• User history: transaction history, account age, velocity patterns, historical fraud flags
• External data: IP geolocation, device fingerprint, merchant risk scores, consortium data
• Labels: confirmed fraud (chargebacks, investigations), confirmed legitimate (no dispute after 60 days)
• Label delay: fraud labels arrive 30-90 days after transaction (chargeback cycle)
• Pipeline: Kafka streaming -> Flink for real-time feature computation -> model serving -> decision engine

Feature Engineering

• Transaction features: amount, merchant category, time of day, day of week, is_international
• Velocity features (real-time): transactions in last 1h/24h/7d, unique merchants in 24h, amount in 24h
• Behavioral features: deviation from user's typical amount, new merchant flag, new device flag, new location flag
• Graph features: merchant risk score (fraud rate), device cluster score, IP reputation
• Derived: amount / average_amount ratio, time_since_last_transaction, geographic_velocity (distance/time)

Model Architecture

flowchart TD
    Tx[Transaction Event] --> FE[Real-Time Feature\nComputation\nFlink + Redis]
    FE --> Rules[Rules Engine\nHard blocks:\nblacklisted merchants,\nstolen cards, velocity limits]
    Rules -->|Pass| Ensemble[ML Ensemble\nXGBoost: tabular features\nGraph NN: network features\nSequence Model: behavior]
    Rules -->|Block| Block[Auto-Block]
    Ensemble --> Decision{Score Threshold}
    Decision -->|High risk > 0.9| Block
    Decision -->|Medium 0.5-0.9| Review[Manual Review Queue]
    Decision -->|Low risk < 0.5| Approve[Approve Transaction]
    Review --> Analyst[Fraud Analyst Decision]
    Analyst -->|Fraud confirmed| Retrain[Retrain Pipeline +\nUpdate Rules]

• Rules engine: Deterministic checks first (blacklists, velocity limits, amount limits). Fast and interpretable. Catches ~30% of fraud.
• XGBoost: Primary model on tabular features. Fast inference (<5ms). Handles feature interactions well.
• Graph neural network: Captures device/IP/merchant network patterns. Batch-computed graph features updated hourly, fed into XGBoost.
• Sequence model: LSTM on user's transaction history for behavioral anomaly detection. Precomputed user state updated per transaction.
• Ensemble: Weighted average with XGBoost dominant. Three threshold tiers: auto-block, review, approve.

Serving Architecture

• Feature store: Redis for real-time features (velocity, running averages), updated per transaction
• Rules engine: in-memory, <2ms latency
• ML model: CPU inference (XGBoost), batched, <10ms
• Total budget: feature lookup (10ms) + rules (2ms) + ML inference (10ms) + decision logic (5ms) = ~27ms p50
• Scaling: stateless model servers, auto-scale on TPS, geographically distributed
• Fallback: if model service is down, rules-only mode (higher false positive rate, acceptable for availability)

Monitoring

• Real-time dashboards: fraud rate, false positive rate, decision distribution, latency
• Model drift: daily comparison of feature distributions between training and production
• Adversarial detection: cluster analysis of blocked transactions, alert on new attack patterns
• Feedback loop: chargeback data feeds retraining within 24 hours (challenge: 30-90 day label delay, use early fraud indicators)
• A/B testing: careful -- cannot A/B test by approving known fraud. Use shadow scoring and historical replay.

---

6. Autonomous Vehicle Perception Stack

Requirements

• Real-time 3D object detection and tracking for autonomous vehicles
• Sensors: 6 cameras (360-degree), 1 LiDAR, 5 radars, IMU, GPS
• Object classes: vehicles, pedestrians, cyclists, traffic signs, lane markings, road edges
• Latency: <100ms end-to-end (sensor input to 3D bounding boxes)
• Safety: miss rate <0.01% for pedestrians within 50m (safety-critical)
• Handle: rain, snow, night, glare, construction zones, emergency vehicles

Metrics

Data Pipeline

• Collection: fleet of test vehicles with calibrated sensor suites, data recorded at 10Hz
• Labeling: 3D bounding box annotation (LiDAR point cloud + camera images), lane marking annotation
• Scale: petabytes of raw sensor data, millions of labeled frames
• Long-tail: active learning to find rare scenarios (emergency vehicles, construction, animals)
• Simulation: synthetic data generation (CARLA, internal simulator) for corner cases
• Pipeline: raw data -> calibration + synchronization -> labeling -> quality review -> training dataset

Model Architecture

flowchart TD
    subgraph Sensors
        C[6 Cameras\n30 FPS each]
        L[LiDAR\n10 Hz, 150m range]
        R[5 Radars\n20 Hz, velocity]
    end
    subgraph Perception["Perception Stack"]
        C --> BEV[BEV Encoder\nLSS / BEVFormer\nProject images to\nbird's eye view]
        L --> VX[Voxel Encoder\nVoxelNet / PointPillars\n3D feature grid]
        BEV --> Fusion[Multi-Modal Fusion\nTransformer-based\nBEV feature fusion]
        VX --> Fusion
        R --> Fusion
        Fusion --> Det[3D Object Detection\nCenterPoint heads\nPer-class detection]
        Fusion --> Seg[Semantic Segmentation\nRoad, lane, sidewalk]
        Det --> Track[Multi-Object Tracking\nKalman filter +\nHungarian matching]
    end
    Track --> Predict[Motion Prediction\nTrajectory forecasting\n3-8 second horizon]
    Predict --> Plan[Planning Module]

• Camera backbone: BEVFormer or LSS (Lift-Splat-Shoot) to project 2D images into bird's-eye-view features. Run on GPU.
• LiDAR backbone: PointPillars or VoxelNet for 3D voxel encoding. Efficient point cloud processing.
• Fusion: Transformer-based attention over BEV features from camera + LiDAR + radar. Learns to weight modalities by scenario (camera better for signs, LiDAR better for geometry, radar better for velocity).
• Detection: CenterPoint heads for 3D bounding box prediction. Per-class heads for different object types.
• Tracking: Extended Kalman filter for state estimation, Hungarian algorithm for association, handle occlusion and reappearance.

Serving Architecture

• Hardware: NVIDIA Orin / Thor SoC, dedicated inference accelerator
• Model optimization: TensorRT, INT8 quantization, pruning (must maintain safety metrics)
• Latency budget: camera processing (20ms) + LiDAR processing (15ms) + fusion (10ms) + detection (15ms) + tracking (5ms) + prediction (15ms) = ~80ms
• Redundancy: dual compute units, fallback to radar-only mode if camera/LiDAR fails
• OTA updates: model updates deployed via over-the-air, staged rollout (1% fleet -> 10% -> 100%)

Monitoring

• Per-vehicle telemetry: inference latency, detection confidence distributions, sensor health
• Fleet-wide analytics: disengagement analysis (what caused human takeover), near-miss detection
• Shadow mode: new model runs in parallel, predictions compared but not acted upon
• Simulation regression: every model update tested against 10K+ scenarios in simulation before deployment
• Safety: formal verification for safety-critical paths, redundant perception for ASIL-D compliance
• Feedback loop: disengagement events trigger automatic data collection and prioritized labeling

Serving Architecture Tradeoffs

Deep reference for ML serving decisions. Covers patterns, frameworks, caching, cost optimization, and deployment strategies.

---

Serving Patterns

Batch Prediction (Precomputation)

How it works: Run inference on all possible inputs (or likely inputs) on a schedule. Store predictions in a key-value store. Serve predictions via simple lookup.

When to use:

• Finite, enumerable input space (e.g., recommendations for all users)
• Prediction freshness tolerance is hours or days
• High-throughput, low-latency serving requirement (lookup is O(1))
• Limited GPU budget (amortize compute overnight)

When NOT to use:

• Input space is combinatorial (e.g., search queries are infinite)
• Real-time features are critical (user just added item to cart)
• Model updates must take effect immediately

Architecture:

Training pipeline -> Model artifact -> Batch job (Spark/Dataflow) ->
Prediction store (DynamoDB/Redis) -> Serving API (simple lookup)

Latency: <5ms (key-value lookup) Freshness: Hours to days (depends on batch frequency) Cost: Low compute (off-peak GPU), high storage (precomputed predictions)

---

Online Inference

How it works: Model loaded in memory on serving nodes. Each request triggers real-time inference with current features.

When to use:

• Input space is infinite or unpredictable (search queries, free text)
• Real-time features materially improve predictions
• Low-latency requirement (<100ms) with fresh predictions
• Model must react to current context (time, location, session)

When NOT to use:

• Latency budget cannot accommodate inference time
• Input space is small enough for batch precomputation
• Cost of always-on GPU fleet is unjustifiable

Architecture:

Request -> Feature service (Redis) -> Model server (Triton/TorchServe) ->
Post-processing -> Response

Latency: 10-500ms (depends on model size and hardware) Freshness: Real-time Cost: High compute (always-on GPU/CPU fleet)

---

Near-Real-Time (Micro-Batch)

How it works: Small batch jobs run every 1-60 minutes. Combines some freshness benefits of online with cost benefits of batch.

When to use:

• Minutes-stale predictions are acceptable
• Features change frequently but not per-request
• Cost-sensitive but need more freshness than daily batch

Architecture:

Streaming events (Kafka) -> Micro-batch processor (Flink/Spark Streaming) ->
Feature update -> Model inference -> Prediction store update

Latency: Minutes (depends on batch interval) Freshness: 1-60 minutes Cost: Medium (scheduled compute, smaller fleet than online)

---

Streaming Inference

How it works: Model consumes events from a stream and produces predictions continuously. No request-response cycle -- predictions are pushed.

When to use:

• Event-driven systems (fraud detection, anomaly detection, bidding)
• Prediction must happen on every event without explicit request
• Complex event processing with temporal patterns

Architecture:

Event stream (Kafka) -> Stream processor (Flink) with embedded model ->
Feature computation + inference in same pipeline -> Output stream/alert

Latency: Sub-second Freshness: Continuous Cost: High (always-on stream processing infrastructure)

---

Pattern Comparison Matrix

---

Feature Store Architectures

Dual-Store Pattern (Industry Standard)

Most production ML systems use a dual-store architecture:

Offline store (batch training):

• Technology: Hive, S3/Parquet, BigQuery, Delta Lake
• Stores: historical feature values with timestamps (point-in-time correctness)
• Used for: training data generation, batch feature computation, backfill
• Access pattern: full table scans, time-range queries

Online store (low-latency serving):

• Technology: Redis, DynamoDB, Bigtable, Cassandra
• Stores: latest feature values per entity (user, item, etc.)
• Used for: real-time feature retrieval during inference
• Access pattern: key-value lookup by entity ID, <5ms p99

Synchronization: batch job materializes offline features to online store, streaming job updates real-time features

Feature Store Frameworks

Feature Freshness Categories

---

Model Serving Frameworks

Framework Comparison

When to Use What

Triton: You need maximum throughput, serve multiple model types (PyTorch + TensorFlow + ONNX), or need concurrent model execution. Industry standard for high-scale.

TorchServe: Pure PyTorch shop, want tight integration with PyTorch ecosystem, simpler setup than Triton.

vLLM: Serving LLMs specifically. PagedAttention gives 2-4x throughput over naive serving. Best for text generation workloads.

TGI: HuggingFace models, want production-ready LLM serving with minimal configuration. Good default for transformer models.

Ray Serve: Complex serving graphs (multiple models in a pipeline), need autoscaling, want Python-native composition.

---

Caching Strategies for ML

Embedding Cache

What: Cache computed embeddings for items/users/queries. Hit rate: High for items (millions of items, reusable), medium for queries (power-law distribution). Invalidation: TTL-based (1-24 hours) + event-based (item metadata change, user activity). Storage: Redis with vector support, or dedicated vector cache. Impact: Skip embedding computation (50-200ms saved per cache hit).

Prediction Cache

What: Cache final predictions for exact input combinations. Hit rate: High for popular queries/items, low for long-tail. Invalidation: TTL (15 min - 24 hours) + model version change. Storage: Redis or Memcached. Impact: Skip entire inference pipeline (10-500ms saved per cache hit). Risk: Stale predictions for fast-changing contexts (user just bought the item).

Feature Cache

What: Cache precomputed features for entities. Hit rate: Very high (features change slowly relative to request rate). Invalidation: Event-driven (user action triggers feature update) + periodic refresh. Storage: Online feature store (Redis/DynamoDB) IS the cache. Impact: Skip feature computation (10-100ms saved).

Cache Hierarchy

Request -> Prediction cache (hit? return) ->
Feature cache (hit? compute prediction) ->
Feature computation -> Model inference ->
Cache prediction -> Return

---

Cost Optimization Strategies

Model-Level Optimization

Infrastructure-Level Optimization

Architecture-Level Optimization

---

Latency Targets by Use Case

---

Deployment Strategies for ML

Shadow Mode

What: New model runs alongside production model. Both receive same traffic. New model predictions are logged but not served. When: First deployment of a new model architecture. Want to verify predictions are reasonable before serving. Duration: 1-7 days depending on traffic volume. Success criteria: New model predictions are comparable to production, no latency regression, no error spikes.

Canary Deployment

What: Route a small percentage of traffic to new model. Monitor metrics closely. Progression: 1% -> 5% -> 25% -> 50% -> 100% (each step 1-3 days) Rollback trigger: Online metric regression beyond threshold (e.g., CTR drops >1%) Gotcha: 1% traffic may not be enough to detect small metric changes. Use statistical power analysis.

A/B Testing

What: Controlled experiment with random user assignment to control (old model) vs treatment (new model). Duration: Determined by sample size calculation. Typical: 1-4 weeks. Pitfalls:

• Novelty effect: users interact differently with new things initially
• Network effects: treatment users influence control users (social platforms)
• Multiple testing: running too many experiments inflates false positive rate
• Interference: seasonal effects, external events confound results

Interleaving (for Ranking)

What: Merge results from two ranking models into a single list. Measure which model's results users prefer via clicks. When: Faster than A/B testing for ranking (needs 10x less traffic to detect differences). How: Team-Draft Interleaving -- alternate placing results from each model, credit clicks to the contributing model. Limitation: Only works for ranking/recommendation, not classification or generation.

Blue-Green Deployment

What: Two identical environments. One serves traffic (blue), one is idle (green). Deploy to green, switch traffic, keep blue as instant rollback. When: Model serving infrastructure changes (not just model weights). Need zero-downtime deployment. Cost: 2x infrastructure during deployment window.

---

Model Versioning and Rollback

Version Everything

• Model weights (with hash)
• Training data snapshot (or pointer to immutable dataset version)
• Feature pipeline code and configuration
• Serving configuration (batch size, timeout, device)
• Preprocessing/postprocessing code

Rollback Checklist

1. Feature flag to switch between model versions (instant rollback) 2. Previous model version always loaded (warm standby) or loadable within SLA 3. Monitoring alert triggers automatic rollback consideration 4. Rollback preserves prediction logs for post-mortem analysis 5. Feature pipeline backward-compatible (new model features should not break old model)

Model Registry

Use a model registry (MLflow, Weights & Biases, Vertex AI Model Registry) to track:

• Model version, training date, training data version
• Offline metrics at training time
• Promotion history: staging -> canary -> production
• Rollback history with reasons
• Associated feature pipeline version