Marketplace Recsys Feature Engineering

Marketplace Engineering Recsys Feature Engineering Best Practices

Comprehensive first-principles guide for deriving usable recommender features from the raw assets of a two-sided trust marketplace — listing photos, owner-supplied listing metadata, and sitter wizard responses — for item-to-item, user-to-item, and user-to-user solutions. Contains 44 rules across 8 categories ordered by cascade impact on the feature-engineering lifecycle, plus one playbook that composes the rules into an end-to-end feature discovery workflow.

This skill is the upstream precursor to marketplace-personalisation (AWS Personalize) and marketplace-search-recsys-planning (OpenSearch retrieval). Those skills treat features as inputs they already have; this skill is about deciding what features to build from the raw assets, which decisions they serve, and how to prove each one is worth its maintenance cost.

When to Apply

Reference this skill when:

• Planning what to extract from listing photos, descriptions, or amenity lists to power i2i similarity or u2i ranking
• Designing or revising the sitter onboarding wizard with recsys features as the primary output
• Deciding whether to build a vision embedding pipeline, a text encoder, or neither — and in what order
• Composing existing base features into item-to-item, user-to-item, or user-to-user scoring
• Auditing an existing feature store for coverage, drift, PII, duplication, or orphan features
• Choosing a ship/kill criterion for a new recsys feature and designing the ablation A/B test
• Answering the question: "we want to improve the similar-homes shelf — what feature should we build?"

Setup

This skill has no user-specific configuration — it is self-contained. References are live URLs to engineering blogs from Airbnb, Pinterest, DoorDash, Uber, Netflix, and Google, to open-source libraries (Feast, Sentence-Transformers, Hugging Face CLIP, H3), to foundational academic papers (Airbnb KDD 2018, Pinterest ItemSage, YouTube Semantic IDs, PinSage), and to Google's Rules of Machine Learning.

Rule Categories

Categories are ordered by cascade impact on the feature-engineering lifecycle: auditing mistakes build features on data that does not exist, first-principles mistakes produce features that do not map to real decisions, extraction mistakes poison everything downstream, and so on. Fix earlier-stage problems before later-stage problems.

Quick Reference

1. Asset Audit and Inventory (CRITICAL)

• `audit-measure-coverage-before-modelling` — reject fields below 80% coverage from the feature plan
• `audit-sample-every-asset-type-end-to-end` — pull 100 real instances through the real fetch path before planning
• `audit-verify-rights-and-privacy-before-extraction` — ToS, GDPR, consent, face blur before encoding
• `audit-quantify-freshness-per-asset` — age distribution + expiry + refresh bucket
• `audit-separate-raw-assets-from-derived-features` — raw immutable in object store, derived versioned in feature store

2. First-Principles Feature Decomposition (CRITICAL)

• `firstp-start-from-the-decision-not-the-algorithm` — decision first, sub-judgments second, tools last
• `firstp-ask-what-signal-a-human-uses` — interview 8-12 owners and sitters; features trace back to quotes
• `firstp-tie-every-feature-to-a-specific-solution` — no feature without a named i2i/u2i/u2u consumer
• `firstp-prefer-directly-observed-over-learned` — observed columns first, learned embeddings second
• `firstp-reject-features-you-cannot-serve-at-inference` — training-serving parity starts at design time
• `firstp-kill-features-a-popularity-baseline-already-captures` — correlation screen before registration

3. Image Feature Extraction (HIGH)

• `vision-use-clip-for-zero-shot-listing-embeddings` — zero-shot CLIP ships in a week
• `vision-detect-room-types-before-detecting-amenities` — room prior conditions the amenity threshold
• `vision-quantify-image-quality-separately-from-content` — blur, lighting, aesthetic as their own features
• `vision-extract-per-object-counts-not-just-presence` — n_bed = 4 beats has_bed = true
• `vision-pool-embeddings-across-a-listings-photo-set` — pooled listing vector; per-photo stored alongside
• `vision-fine-tune-on-your-domain-when-clip-underperforms` — contrastive fine-tune only after zero-shot plateaus

4. Listing Text and Metadata Extraction (HIGH)

• `listing-declare-categorical-fields-for-bounded-vocabularies` — bounded vocab → categorical, validated on write
• `listing-multi-hot-encode-amenity-lists` — fixed amenity vocabulary → multi-hot vector
• `listing-hash-geo-to-hierarchies-not-raw-lat-lon` — H3 at multiple resolutions
• `listing-embed-description-with-pretrained-sentence-encoder` — all-MiniLM-L6-v2 for cheap semantic text features
• `listing-extract-stay-duration-shape-not-just-length` — bin + holiday overlap + flexibility, not raw day count
• `listing-encode-pet-requirements-as-structured-triples` — (species, count, special_needs) triples plus free text alongside

5. Sitter Wizard and Profile Extraction (HIGH)

• `wizard-order-questions-by-information-gain` — discriminative questions first, narrative last
• `wizard-prefer-multiple-choice-over-free-text` — categorical features by construction
• `wizard-make-skips-genuine-and-log-them` — skip is signal; defaults destroy it
• `wizard-capture-experience-as-counts-and-dates` — numbers, not adjectives; platform history overrides self-declaration
• `wizard-separate-hard-constraints-from-soft-preferences` — filters vs ranking features

6. Derived Similarity and Affinity (MEDIUM-HIGH)

• `derive-precompute-i2i-nearest-neighbours-offline` — ANN shelf built nightly, served from KV in <5ms
• `derive-fuse-modalities-before-item-similarity` — vision + text + structured, weighted and normalised
• `derive-use-two-tower-for-user-item-affinity` — dual encoder trained on interactions; ANN-retrieval-ready
• `derive-score-u2u-as-symmetric-mutual-fit` — min(P(owner), P(sitter)); one-sided scoring produces wasted requests
• `derive-decompose-affinity-into-interpretable-subscores` — fit/safety/logistics/price subscores + blend
• `derive-cache-user-embedding-with-short-ttl` — session-level cache, 60-300s TTL

7. Feature Quality and Governance (MEDIUM-HIGH)

• `quality-version-feature-definitions-in-one-registry` — one name, one implementation, one owner
• `quality-serve-training-and-inference-from-one-store` — feature store as the single source of truth
• `quality-gate-features-on-coverage-and-drift` — coverage floor + PSI alarm
• `quality-scrub-pii-before-features-leave-secure-zone` — face blur and regex scrubbing before encoding
• `quality-freeze-feature-schemas-per-model-version` — schema hash pinned to model artifact

8. Incremental Rollout and Value Proof (MEDIUM)

• `prove-ship-one-feature-at-a-time` — one feature, one experiment, one decision
• `prove-measure-lift-against-feature-ablated-variant` — ablation isolates the feature from incidental changes
• `prove-kill-features-that-dont-earn-maintenance` — quarterly kill review on attributed lift
• `prove-dedicate-random-exploration-slice-to-new-features` — 3-5% slice catches offline-close-to-tied winners
• `prove-retain-feature-free-baseline-permanently` — popularity baseline as drift anchor

Discovering New Features

One playbook composes the rules into an end-to-end workflow:

• `references/playbooks/discovering.md` — Discover new features from raw marketplace assets: a seven-step workflow that starts with an asset audit and a decision decomposition and ends with a shipped ablation A/B against a feature-ablated baseline. Use when the task is "what should we build next?" rather than "fix this specific feature."

Read the playbook first when the task is an open-ended "how do we extract more signal from X?" Read individual rules when a specific implementation question arises.

How to Use

• Read `references/_sections.md` for category structure and cascade rationale
• Read `gotchas.md` for accumulated diagnostic lessons before suggesting interventions
• Read `references/playbooks/discovering.md` to plan a new feature discovery cycle
• Read individual rule files under references/ when a specific task matches the rule title
• Use `assets/templates/_template.md` to author new rules as the skill grows

Related Skills

• `marketplace-personalisation` — Post-extraction personalisation on AWS Personalize: event tracking, schema design, two-sided matching, cold start, feedback loops. Hand off once your features are in the store and you are ready to train a ranker.
• `marketplace-search-recsys-planning` — OpenSearch retrieval planning: query understanding, index design, ranking, search-plus-recs blending. Hand off when the bottleneck is retrieval rather than feature availability.
• `marketplace-pre-member-personalisation` — Pre-member journey from anonymous visit to paid membership: anonymous signal inference, onboarding intent capture, pre-member measurement. Hand off at the paid-member boundary.

Reference Files

Two-Sided Recsys Feature Engineering

Version 0.1.0 Marketplace Engineering April 2026

Note:

This document is mainly for agents and LLMs to follow when maintaining,

generating, or refactoring codebases. Humans may also find it useful,

but guidance here is optimized for automation and consistency by AI-assisted workflows.

---

Abstract

First-principles guide for deriving usable recommender features from the raw assets of a two-sided trust marketplace — listing photos, owner-entered listing metadata, and sitter wizard responses — for item-to-item, user-to-item, and user-to-user solutions. Contains 44 rules across 8 categories ordered by cascade impact on the feature-engineering lifecycle, from asset auditing and first-principles decomposition through vision, text, and wizard extraction, multi-modal composition into i2i/u2i/u2u scores, feature-store governance and training-serving parity, and incremental online value proof. Includes one playbook that composes the rules into an end-to-end feature discovery workflow. Functions as the upstream precursor to the companion marketplace-personalisation, marketplace-search-recsys-planning, and marketplace-pre-member-personalisation skills.

---

Table of Contents

1. Asset Audit and Inventory — CRITICAL

• 1.1 Measure Coverage Before Declaring a Field a Feature — CRITICAL (prevents modelling features that only exist for 10% of items)
• 1.2 Quantify Freshness Per Asset Type — CRITICAL (prevents stale assets from poisoning similarity and affinity scores)
• 1.3 Sample Every Asset Type End-to-End Before Planning Features — CRITICAL (prevents silent garbage inputs to extraction pipelines)
• 1.4 Separate Raw Assets from Derived Features — CRITICAL (prevents 1-way data loss that blocks re-extraction with better models)
• 1.5 Verify Rights and Privacy Before Running Extraction — CRITICAL (prevents irreversible privacy and ToS violations)

2. First-Principles Feature Decomposition — CRITICAL

• 2.1 Ask What Signal a Human Uses to Make the Same Decision — CRITICAL (prevents guessing — surfaces 5-15 evidence-backed candidates per interview round)
• 2.2 Kill Features a Popularity Baseline Already Captures — CRITICAL (prevents redundant features inflating the portfolio)
• 2.3 Prefer Directly Observed Features over Learned Features at Launch — CRITICAL (delivers 80% of the lift at 10% of the system complexity)
• 2.4 Reject Features You Cannot Compute at Inference Time — CRITICAL (prevents the #1 cause of training-serving skew)
• 2.5 Start from the Decision, Not the Algorithm — CRITICAL (eliminates 60-80% of features that add cost without moving the outcome)
• 2.6 Tie Every Feature to a Specific Solution and Metric — CRITICAL (prevents orphan features that cost maintenance without lift)

3. Image Feature Extraction — HIGH

• 3.1 Apply Domain Fine-Tuning Only When Zero-Shot CLIP Plateaus — HIGH (closes 10-30% of the i2i relevance gap on domain taxonomies)
• 3.2 Detect Room Type Before Detecting Amenities — HIGH (makes amenity counts per-room, cutting false positives by 50%)
• 3.3 Extract Per-Object Counts, Not Just Presence — HIGH (prevents conflating a studio with a 6-bedroom villa)
• 3.4 Pool Embeddings Across a Listing's Photo Set — HIGH (reduces i2i variance by 2-4x versus single-photo features)
• 3.5 Quantify Image Quality Separately from Content — HIGH (prevents low-quality photos from flattening content embeddings)
• 3.6 Use CLIP for Zero-Shot Listing Embeddings Before Fine-Tuning — HIGH (ships the vision pipeline 10-15x faster than training from scratch)

4. Listing Text and Metadata Extraction — HIGH

• 4.1 Declare Categorical Fields for Bounded Vocabularies — HIGH (enables per-value learned features instead of text-bag processing)
• 4.2 Embed Description Text with a Pretrained Sentence Encoder — HIGH (prevents TF-IDF sparsity and synonym drift with 0 training cost)
• 4.3 Encode Amenity Lists as Multi-Hot Vectors, Not Free-Text Strings — HIGH (prevents string-tokenization drift across training and serving)
• 4.4 Encode Pet Requirements as Structured Triples — HIGH (enables per-axis matching that free text cannot)
• 4.5 Extract Stay Duration Shape, Not Just Length — HIGH (unlocks 3-5 sitter preference segments over a single integer)
• 4.6 Hash Geo to Hierarchies, Not Raw Lat/Lon — HIGH (prevents the model from treating geo as an arbitrary 2D plane)

5. Sitter Wizard and Profile Extraction — HIGH

• 5.1 Capture Experience as Counts and Dates, Not Adjectives — HIGH (prevents aspirational self-rating that flattens the feature)
• 5.2 Make Optional Questions Genuinely Skippable and Log the Skip — HIGH (preserves the "did not answer" signal instead of destroying it)
• 5.3 Order Wizard Questions by Information Gain — HIGH (2-3x feature usefulness per completed wizard question)
• 5.4 Prefer Multiple-Choice over Free Text in the Wizard — HIGH (prevents downstream NLP cost and training-serving drift)
• 5.5 Separate Hard Constraints from Soft Preferences in the Wizard — HIGH (prevents 30-50% of requests ending in owner rejection)

6. Derived Similarity and Affinity — MEDIUM-HIGH

• 6.1 Cache the User Embedding with a Short TTL, Not Per-Request — MEDIUM-HIGH (drops u2i latency from 80ms to 5ms per request)
• 6.2 Decompose Affinity into Interpretable Subscores — MEDIUM-HIGH (cuts rank-debug investigation time by 3-5x)
• 6.3 Fuse Modalities Before Computing Item Similarity — MEDIUM-HIGH (multi-modal i2i beats any single modality alone)
• 6.4 Precompute Item-to-Item Nearest Neighbours Offline — MEDIUM-HIGH (turns i2i from 500ms per request to 5ms)
• 6.5 Score User-to-User Compatibility as Symmetric Mutual Fit — MEDIUM-HIGH (prevents the 30-50% of requests that end in owner rejection)
• 6.6 Use a Two-Tower Model for User-to-Item Affinity — MEDIUM-HIGH (learned u2i affinity beats hand-crafted scoring 2-5x on NDCG)

7. Feature Quality and Governance — MEDIUM-HIGH

• 7.1 Freeze Feature Schemas per Model Version — MEDIUM-HIGH (prevents mid-flight schema drift from silently retraining the wrong model)
• 7.2 Gate Every Feature on Coverage and Drift Alarms — MEDIUM-HIGH (catches coverage collapse 10-20x earlier than metric drift)
• 7.3 Scrub PII Before Features Leave the Secure Zone — MEDIUM-HIGH (prevents GDPR exposure through embedding leaks)
• 7.4 Serve Training and Inference Features from One Store — MEDIUM-HIGH (eliminates the #1 cause of silent model regression)
• 7.5 Version Feature Definitions in a Single Registry — MEDIUM-HIGH (prevents two models silently computing the same feature differently)

8. Incremental Rollout and Value Proof — MEDIUM

• 8.1 Dedicate a Random Exploration Slice to New Features — MEDIUM (prevents offline-metric overfitting from blocking good features)
• 8.2 Kill Features That Do Not Earn Their Maintenance Cost — MEDIUM (removes 20-40% of features over the first year of portfolio maturity)
• 8.3 Measure Lift Against a Feature-Ablated Variant, Not the Old Model — MEDIUM (prevents attribution confounds from hyperparameter or data changes)
• 8.4 Retain a Feature-Free Baseline Permanently — MEDIUM (prevents silent ML-vs-baseline gap collapse)
• 8.5 Ship One Feature at a Time in the First Year — MEDIUM (prevents bundled-release attribution confounds)

---

References

1. https://developers.google.com/machine-learning/guides/rules-of-ml 2. https://eugeneyan.com/writing/system-design-for-discovery/ 3. https://eugeneyan.com/writing/patterns-for-personalization/ 4. https://eugeneyan.com/writing/real-time-recommendations/ 5. https://medium.com/airbnb-engineering/amenity-detection-and-beyond-new-frontiers-of-computer-vision-at-airbnb-144a4441b72e 6. https://medium.com/airbnb-engineering/when-a-picture-is-worth-more-than-words-17718860dcc2 7. https://medium.com/airbnb-engineering/airbnbs-ai-powered-photo-tour-using-vision-transformer-e470535f76d4 8. https://medium.com/airbnb-engineering/widetext-a-multimodal-deep-learning-framework-31ce2565880c 9. https://medium.com/airbnb-engineering/listing-embeddings-for-similar-listing-recommendations-and-real-time-personalization-in-search-601172f7603e 10. https://medium.com/airbnb-engineering/embedding-based-retrieval-for-airbnb-search-aabebfc85839 11. https://arxiv.org/pdf/1810.09591 12. https://www.kdd.org/kdd2018/accepted-papers/view/real-time-personalization-using-embeddings-for-search-ranking-at-airbnb 13. https://medium.com/pinterest-engineering/pinsage-a-new-graph-convolutional-neural-network-for-web-scale-recommender-systems-88795a107f48 14. https://medium.com/pinterest-engineering/pinnersage-multi-modal-user-embedding-framework-for-recommendations-at-pinterest-bfd116b49475 15. https://cs.stanford.edu/people/jure/pubs/itemsage-kdd22.pdf 16. https://arxiv.org/abs/2306.08121 17. https://huggingface.co/docs/transformers/model_doc/clip 18. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 19. https://www.width.ai/post/product-similarity-search-with-fashion-clip 20. https://aws.amazon.com/blogs/machine-learning/implement-unified-text-and-image-search-with-a-clip-model-using-amazon-sagemaker-and-amazon-opensearch-service/ 21. https://www.shaped.ai/blog/the-two-tower-model-for-recommendation-systems-a-deep-dive 22. https://www.hopsworks.ai/dictionary/two-tower-embedding-model 23. https://h3geo.org/ 24. https://docs.feast.dev 25. https://feast.dev/blog/what-is-a-feature-store/ 26. https://medium.com/@scoopnisker/solving-the-training-serving-skew-problem-with-feast-feature-store-3719b47e23a2 27. https://careersatdoordash.com/blog/building-a-gigascale-ml-feature-store-with-redis/ 28. https://careersatdoordash.com/blog/homepage-recommendation-with-exploitation-and-exploration/ 29. https://www.uber.com/blog/michelangelo-machine-learning-platform/ 30. https://www.uber.com/us/en/blog/michelangelo-machine-learning-model-representation/ 31. https://greatexpectations.io/blog/ml-ops-data-quality/ 32. https://www.hopsworks.ai/post/data-validation-for-enterprise-ai-using-great-expectations-with-hopsworks 33. https://www.nngroup.com/articles/progressive-disclosure/ 34. https://www.nngroup.com/articles/required-fields/ 35. https://docs.aws.amazon.com/personalize/latest/dg/item-dataset-requirements.html 36. https://research.netflix.com/research-area/recommendations 37. https://experimentguide.com/

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

{Rule Title}

{1-3 sentences explaining WHY this matters in the context of marketplace recsys feature engineering — what goes wrong without this pattern, what the cascade effect is on downstream i2i/u2i/u2u systems, and what the concrete consequence looks like in production. Focus on reasoning the model can generalise, not rigid dictation.}

Incorrect ({what is wrong}):

# production-realistic code showing the problem
# comment explaining the specific cost
def example_bad():
    ...

Correct ({what is right}):

# production-realistic code showing the fix
# minimal diff from the incorrect example — only the key insight changes
# comment explaining the specific benefit
def example_good():
    ...

{Optional sections as needed:}

Alternative ({context}): {Alternative approach when applicable}

When NOT to use this pattern:

• {Exception 1}
• {Exception 2}

Reference: {Title}

Gotchas

Append-only lessons about feature engineering for recsys in a two-sided trust marketplace. Each entry is a specific failure point and the resolution, with a date. Read before recommending an intervention — the problem you are about to solve may already be documented here.

---

No known gotchas yet. Add them here as they are discovered.

Template

### {Short, specific title}
{1-3 sentences describing what went wrong, what the symptom looked like, and
what the root cause turned out to be.}
**Fix:** {The specific action that resolved it.}
**Rules that prevent this:** [`rule-name`](references/rule-name.md)
Added: YYYY-MM-DD

Marketplace Recsys Feature Engineering Skill

First-principles best-practices skill for deriving usable recommender features from the raw assets of a two-sided trust marketplace — listing photos, owner-entered listing metadata, and sitter wizard responses — for item-to-item, user-to-item, and user-to-user solutions.

Overview

This skill is a distillation of authoritative guidance from engineering blogs at Airbnb, Pinterest, DoorDash, Uber, and Netflix, open-source libraries (Feast, Sentence-Transformers, Hugging Face CLIP, H3), foundational academic papers (Airbnb KDD 2018, Pinterest ItemSage, YouTube Semantic IDs, PinSage), and Google's Rules of Machine Learning. It contains 44 rules across 8 categories, ordered by cascade impact on the feature-engineering lifecycle, and one playbook that composes the rules into an end-to-end feature discovery workflow.

This skill is the upstream precursor to the sibling marketplace-personalisation, marketplace-search-recsys-planning, and marketplace-pre-member-personalisation skills. Use it when the question is "what features should we build?"; hand off to the others when the question becomes "how do we rank, retrieve, or convert?"

Structure

marketplace-recsys-feature-engineering/
├── SKILL.md                    # Entry point with category index and quick reference
├── AGENTS.md                   # Compiled navigation document (built by script)
├── metadata.json               # Version, discipline, authoritative references
├── README.md                   # This file
├── gotchas.md                  # Accumulated diagnostic lessons (living)
├── references/
│   ├── _sections.md            # Category definitions and impact ordering
│   ├── audit-*.md              # Asset Audit and Inventory (5 rules)
│   ├── firstp-*.md             # First-Principles Feature Decomposition (6 rules)
│   ├── vision-*.md             # Image Feature Extraction (6 rules)
│   ├── listing-*.md            # Listing Text and Metadata Extraction (6 rules)
│   ├── wizard-*.md             # Sitter Wizard and Profile Extraction (5 rules)
│   ├── derive-*.md             # Derived Similarity and Affinity (6 rules)
│   ├── quality-*.md            # Feature Quality and Governance (5 rules)
│   ├── prove-*.md              # Incremental Rollout and Value Proof (5 rules)
│   └── playbooks/
│       └── discovering.md      # End-to-end feature discovery workflow
└── assets/
    └── templates/
        └── _template.md        # Template for authoring new rules

Getting Started

From the repo root, install plugin dependencies and run the skill validator:

pnpm install
pnpm build
pnpm validate

The validator runs structural and substance checks against the skill:

node scripts/validate-skill.js skills/.experimental/marketplace-recsys-feature-engineering

Build the compiled navigation document:

node scripts/build-agents-md.js skills/.experimental/marketplace-recsys-feature-engineering

Creating a New Rule

Rules go in references/ with a filename of the form {prefix}-{slug}.md, where {prefix} matches an existing category in references/_sections.md. Copy assets/templates/_template.md as a starting point and fill in the frontmatter and body.

A rule must include:

• YAML frontmatter: title, impact, impactDescription, tags (first tag is the prefix)
• One-to-three sentence explanation of why the rule matters and its cascade effect
• An **Incorrect (specific description):** code block with production-realistic code
• A **Correct (specific description):** code block that differs minimally from the incorrect
• A Reference: line linking to an authoritative source

Run pnpm validate after adding or editing rules.

Rule File Structure

Each rule has a strict structure enforced by the validator:

---
title: Use Two-Tower for User-to-Item Affinity
impact: MEDIUM-HIGH
impactDescription: 2-5x NDCG over hand-crafted scoring
tags: derive, u2i, two-tower, dual-encoder
---

## Use Two-Tower for User-to-Item Affinity

Explanation paragraph — why the rule matters, what goes wrong without it,
and how the cascade effect plays out downstream.

**Incorrect (concrete failure mode):**

Production-realistic bad example

**Correct (concrete solution):**

Production-realistic good example

Reference: [Source Title](https://example.com/source)

File Naming Convention

• Skill directory: kebab-case matching the skill name (marketplace-recsys-feature-engineering)
• Rule files: {category-prefix}-{slug}.md with kebab-case slugs (audit-measure-coverage-before-modelling.md)
• Playbook files: references/playbooks/{name}.md
• Templates: assets/templates/_template.md (underscore prefix to exclude from rule listings)
• Category prefixes are 3-8 lowercase letters and defined once in _sections.md

Impact Levels

Categories and rules use six impact levels ordered from highest to lowest cascade impact:

Target distribution for a 40-60 rule distillation: 2-3 CRITICAL categories, 2-4 HIGH, the rest MEDIUM or lower. This skill has 2 CRITICAL, 3 HIGH, 2 MEDIUM-HIGH, and 1 MEDIUM category.

Scripts

The dev-skill plugin provides two scripts used by this skill:

• scripts/validate-skill.js — runs structural and substance validation (required before shipping)
• scripts/build-agents-md.js — compiles the navigation document (never write AGENTS.md manually)

Example invocations:

node scripts/validate-skill.js skills/.experimental/marketplace-recsys-feature-engineering
node scripts/validate-skill.js skills/.experimental/marketplace-recsys-feature-engineering --sections-only
node scripts/build-agents-md.js skills/.experimental/marketplace-recsys-feature-engineering

Contributing

• New rules must follow the structure above and pass pnpm validate with zero errors
• Every rule must have incorrect and correct examples with specific annotations
• References must be from primary maintainers, peer-reviewed research, or engineering blogs with data
• Avoid hedging language (might, perhaps, it is recommended) — use imperative form
• Quantify impact where possible (2-10×, 200ms, prevents stale closures, O(n) to O(1))
• Playbooks in references/playbooks/ compose rules into end-to-end workflows
• Never edit AGENTS.md manually — it is regenerated by build-agents-md.js

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

Categories are ordered by cascade impact on the feature-engineering lifecycle for a two-sided marketplace recommender. A mistake at an earlier stage silently corrupts every stage that depends on it. Auditing mistakes produce features built on imaginary data. First-principles mistakes produce features borrowed from other products that do not match this marketplace's decisions. Extraction mistakes produce garbage signals that no downstream composition can rescue. Composition mistakes produce similarity and affinity scores that look reasonable offline and collapse online. Quality mistakes let drift eat the model silently. Rollout mistakes let a feature portfolio accumulate maintenance cost without lift.

---

1. Asset Audit and Inventory (audit)

Impact: CRITICAL Description: Every downstream stage is built on assumptions about raw data, so an honest audit of what actually exists — coverage, freshness, quality, privacy status — is the only thing that separates a feature plan from wishful thinking.

2. First-Principles Feature Decomposition (firstp)

Impact: CRITICAL Description: Features must be derived from the specific decision a buyer or seller is making in this marketplace, not copied from other products, so every candidate feature is reasoned backwards from the outcome it is supposed to move and the solution (i2i, u2i, u2u) it is supposed to feed.

3. Image Feature Extraction (vision)

Impact: HIGH Description: Listing photos carry aesthetic, layout, amenity, and quality signal that text cannot capture, so vision features — CLIP or domain-tuned embeddings, room-type classification, object detection, quality scoring — are the highest-leverage extraction in a visual marketplace when done with discipline around pooling, freshness, and privacy.

4. Listing Text and Metadata Extraction (listing)

Impact: HIGH Description: Owner-supplied metadata (amenities, location, duration, pet requirements, descriptions) is the cheapest, most controllable feature source, so the quality of categorical encoding, geo-hashing, text embedding, and structured triple design determines how much downstream i2i and u2i ranking can learn without new ML infrastructure.

5. Sitter Wizard and Profile Extraction (wizard)

Impact: HIGH Description: Sitter self-declarations from the onboarding wizard are features by construction, so question ordering by information gain, multiple-choice over free text, honest skippability, and the hard-constraint-versus-soft-preference split decide whether the wizard produces usable u2i features or noise.

6. Derived Similarity and Affinity (derive)

Impact: MEDIUM-HIGH Description: Item-to-item similarity, user-to-item affinity, and user-to-user mutual fit are compositions over base features — fused modalities, trained two-tower embeddings, interpretable subscores, precomputed nearest-neighbour shelves — so the composition strategy governs whether i2i, u2i, and u2u surfaces are debuggable, servable, and actually two-sided.

7. Feature Quality and Governance (quality)

Impact: MEDIUM-HIGH Description: Features silently rot through drift, privacy leaks, training-serving skew, and definitional ambiguity, so a single feature registry, a feature store that serves both training and inference, coverage and drift alarms, and PII scrubbing at the extraction boundary keep the portfolio trustworthy over time.

8. Incremental Rollout and Value Proof (prove)

Impact: MEDIUM Description: Features must earn their place in production through online A/B tests against a feature-ablated variant, not against the previous model, so shipping one feature at a time, dedicating an exploration slice, killing non-lifting features, and retaining a feature-free baseline protect the portfolio from accumulating maintenance cost without measurable value.

Measure Coverage Before Declaring a Field a Feature

A field that is 88% null is not a feature — it is a fallback-handling problem that the model will silently average out, producing a score that is indistinguishable from the popularity baseline for most of the catalog. Before any extraction plan, run a coverage report across every candidate source field and reject anything below the coverage threshold (typically 80%) unless you are explicitly modelling a feature for a cohort and routing the other cohort to a different pipeline.

Incorrect (assumes `garden_size_sqm` is always populated):

def build_listing_features(listing: Listing) -> dict:
    return {
        "listing_id": listing.id,
        "region": listing.region_code,
        "garden_size_sqm": listing.garden_size_sqm,  # silently NaN for 88% of listings
        "num_pets": listing.num_pets,
    }

Correct (coverage audit gate before the field enters the feature plan):

COVERAGE_REPORT = run_coverage_audit(
    table="listings",
    fields=["region_code", "garden_size_sqm", "num_pets", "amenities", "cover_photo_url"],
)
# coverage_report = {"region_code": 0.99, "garden_size_sqm": 0.12, ...}

ELIGIBLE_FIELDS = {f for f, c in COVERAGE_REPORT.items() if c >= 0.80}

def build_listing_features(listing: Listing) -> dict:
    features = {"listing_id": listing.id}
    if "region_code" in ELIGIBLE_FIELDS:
        features["region"] = listing.region_code
    if "num_pets" in ELIGIBLE_FIELDS:
        features["num_pets"] = listing.num_pets
    # garden_size_sqm is excluded from the feature plan until coverage improves
    return features

Reference: Google — Rules of Machine Learning, Rule #22: Clean up features you are no longer using

Quantify Freshness Per Asset Type

A cover photo uploaded in 2019 does not reflect the listing in 2026 — the pet has changed, the kitchen is renovated, the garden is overgrown. A sitter wizard response from two years ago describes someone with fewer stays and different preferences. Before features based on these assets feed i2i or u2i scoring, measure the age distribution for each asset type, set an expiry, and decide whether stale assets are excluded, re-requested, or weighted down. A feature that silently averages 2019 and 2026 signal is worse than no feature.

Incorrect (computes embeddings over all photos regardless of upload date):

def encode_all_listing_photos() -> dict[str, np.ndarray]:
    listings = db.query("SELECT id, cover_photo_url FROM listings").all()
    return {l.id: clip_model.encode_image(fetch(l.cover_photo_url)) for l in listings}

Correct (freshness gate + ask-for-refresh bucket):

FRESHNESS_CUTOFF = timedelta(days=540)  # 18 months

def encode_listing_photos() -> tuple[dict[str, np.ndarray], list[str]]:
    listings = db.query(
        "SELECT id, cover_photo_url, cover_photo_uploaded_at FROM listings"
    ).all()

    embeddings: dict[str, np.ndarray] = {}
    ask_for_refresh: list[str] = []

    for l in listings:
        age = datetime.now(timezone.utc) - l.cover_photo_uploaded_at
        if age > FRESHNESS_CUTOFF:
            ask_for_refresh.append(l.id)  # owner nudged to re-upload; listing excluded from vision features for now
            continue
        embeddings[l.id] = clip_model.encode_image(fetch(l.cover_photo_url))

    return embeddings, ask_for_refresh

Reference: Airbnb — Amenity Detection and Beyond: New Frontiers of Computer Vision at Airbnb

Sample Every Asset Type End-to-End Before Planning Features

Schema documentation lies, migration scripts silently corrupt data, and photos stored as S3 URLs can be dead links by the time you train on them. Before committing to a feature plan, pull a random sample of 100 real instances per asset type — listing photos, listing descriptions, wizard responses — and open each one end-to-end through the same path the extractor will use. The 3-5 that fail to load reveal the ingestion bugs you would otherwise discover by training on broken data.

Incorrect (trusts the DB row count without fetching the actual asset):

def count_trainable_listings() -> int:
    return db.query("SELECT COUNT(*) FROM listings WHERE cover_photo_url IS NOT NULL").scalar()

TRAINABLE = count_trainable_listings()  # 487,219 — but 4% of URLs 404

Correct (sample end-to-end through the real fetch path):

def audit_photo_fetch(sample_size: int = 100) -> dict:
    rows = db.query(
        "SELECT listing_id, cover_photo_url FROM listings "
        "WHERE cover_photo_url IS NOT NULL ORDER BY RANDOM() LIMIT :n",
        n=sample_size,
    ).all()

    results = {"ok": 0, "404": 0, "corrupt": 0, "too_small": 0}
    for row in rows:
        resp = asset_client.fetch(row.cover_photo_url)
        if resp.status == 404:
            results["404"] += 1
        elif not is_valid_jpeg(resp.body):
            results["corrupt"] += 1
        elif image_dims(resp.body) < (200, 200):
            results["too_small"] += 1
        else:
            results["ok"] += 1
    return results

# Run before any extraction plan; a 4% 404 rate means re-hosting before training, not after.

Reference: Eugene Yan — System Design for Recommendations and Search

Separate Raw Assets from Derived Features

Deriving features in place — overwriting a photo with a smaller JPEG, replacing a description with its TF-IDF vector, discarding the wizard's free text after encoding — locks you out of ever re-extracting features with a better model. The raw input is the irreplaceable asset; the derived feature is a cheap, versionable function of it. Store raw assets immutably in object storage, store derived features as versioned columns or entries in the feature store, and treat extraction as a reproducible pipeline, not a one-time import.

Incorrect (derives features in place and discards the source):

def process_new_listing(listing: Listing) -> None:
    photo = asset_client.fetch(listing.cover_photo_url)
    embedding = clip_model.encode_image(photo)
    db.execute(
        "UPDATE listings SET cover_photo_embedding = :e WHERE id = :id",
        e=embedding.tolist(), id=listing.id,
    )
    asset_client.delete(listing.cover_photo_url)  # original lost forever

Correct (raw preserved, derived feature versioned in the feature store):

VISION_MODEL_VERSION = "clip-vit-b32-2026-q2"

def process_new_listing(listing: Listing) -> None:
    photo = asset_client.fetch(listing.cover_photo_url)  # raw stays where it is

    embedding = clip_model.encode_image(photo)
    feature_store.put(
        entity_key=listing.id,
        feature_group="listing_vision",
        values={"cover_photo_embedding": embedding.tolist()},
        version=VISION_MODEL_VERSION,
    )
    # next month, swap CLIP for a domain-tuned model; re-extract from the intact raw asset.

Reference: Feast — What is a Feature Store?

Verify Rights and Privacy Before Running Extraction

Listing photos frequently contain faces, house numbers, and car plates. Descriptions contain phone numbers and exact addresses. Wizard responses contain medical notes about pets. Extracting features from these assets without first confirming that the Terms of Service permit ML processing and that PII is stripped or hashed creates a compliance debt that is cheap to avoid at ingestion time and ruinous to unwind once embeddings are in production. The audit happens before the first CLIP.encode() call, not after.

Incorrect (pulls photos directly into a vision model):

def build_image_embeddings(listing_ids: list[str]) -> dict[str, np.ndarray]:
    embeddings = {}
    for lid in listing_ids:
        photo = asset_client.fetch(f"s3://ths-listings/{lid}/cover.jpg")
        embeddings[lid] = clip_model.encode_image(photo)  # face of the host is now in the vector
    return embeddings

Correct (ToS gate, PII scrubbing, and consent flag before extraction):

def build_image_embeddings(listing_ids: list[str]) -> dict[str, np.ndarray]:
    assert settings.ml_training_clause_version >= "2026-02", "ToS does not yet permit ML feature extraction"

    embeddings = {}
    for lid in listing_ids:
        listing = db.get(Listing, lid)
        if not listing.owner.ml_training_consent:
            continue  # explicit opt-out honoured
        photo = asset_client.fetch(listing.cover_photo_url)
        photo_scrubbed = face_blur.apply(photo)  # face detection + blur before the encoder
        embeddings[lid] = clip_model.encode_image(photo_scrubbed)
    return embeddings

Reference: Airbnb — When a Picture Is Worth More Than Words

Cache the User Embedding with a Short TTL, Not Per-Request

Recomputing the user tower on every homefeed request is wasteful — a sitter's profile and recent actions do not change between scrolls, and the user embedding is dominated by slow-changing features (wizard answers, history, preferences) with a small fast-moving contextual delta. Compute the user embedding at session start, cache it for 60-300 seconds keyed by session, and optionally update it incrementally as new actions arrive during the session. This drops per-request latency by an order of magnitude while preserving freshness.

Incorrect (rebuilds the user vector every request):

def homefeed(sitter_id: str) -> list[Listing]:
    user_features = feature_store.get_online(sitter_id)  # 50ms
    user_vector = user_tower.encode(user_features)        # 30ms
    candidates = ann_index.search(user_vector, k=200)     # 5ms
    return rank(candidates)[:24]
    # 85ms per request, mostly in feature fetch + encode

Correct (session-level cache with TTL):

USER_VECTOR_TTL_SECONDS = 180

def get_or_compute_user_vector(sitter_id: str, session_id: str) -> np.ndarray:
    cache_key = f"uvec:{session_id}:{sitter_id}"
    cached = redis.get(cache_key)
    if cached is not None:
        return np.frombuffer(cached, dtype=np.float32)

    user_features = feature_store.get_online(sitter_id)
    user_vector = user_tower.encode(user_features)
    redis.set(cache_key, user_vector.tobytes(), ex=USER_VECTOR_TTL_SECONDS)
    return user_vector

def homefeed(sitter_id: str, session_id: str) -> list[Listing]:
    user_vector = get_or_compute_user_vector(sitter_id, session_id)  # 1ms after first call
    candidates = ann_index.search(user_vector, k=200)                 # 5ms
    return rank(candidates)[:24]

Reference: DoorDash — Building a Gigascale ML Feature Store with Redis

Decompose Affinity into Interpretable Subscores

A single opaque u2i score from a two-tower model is fast to compute but impossible to debug when ranking goes wrong. Complementing it with named subscores — fit, safety, logistics, price — that are each a narrow function of features gives product, trust, and support teams a vocabulary for explaining why a listing is ranked where it is, and makes regressions traceable. The blended score is what sorts the results; the subscores are what appears in the debug panel and the why-this-recommendation explanation shown to the sitter.

Incorrect (single opaque score — untraceable):

def rank(listings: list[Listing], sitter: Sitter) -> list[Listing]:
    scored = [(l, two_tower.score(sitter, l)) for l in listings]
    return [l for l, _ in sorted(scored, key=lambda x: -x[1])]
    # when a listing appears in a wrong slot, nobody can explain why

Correct (subscores + blend + debug panel):

@dataclass
class AffinityBreakdown:
    fit: float         # pet, experience, interests
    safety: float      # verification, insurance, reviews
    logistics: float   # availability, travel feasibility
    price: float       # budget alignment (or free, for THS-style membership)
    blended: float     # final sort key
    trace: dict        # feature contributions for debug UI

def affinity_breakdown(sitter: Sitter, listing: Listing) -> AffinityBreakdown:
    fit = fit_model.score(sitter, listing)
    safety = safety_model.score(sitter, listing)
    logistics = logistics_model.score(sitter, listing)
    price = price_model.score(sitter, listing)

    blended = 0.45 * fit + 0.25 * safety + 0.20 * logistics + 0.10 * price
    trace = {
        "fit_features": fit_model.contributions(sitter, listing),
        "safety_features": safety_model.contributions(sitter, listing),
    }
    return AffinityBreakdown(fit, safety, logistics, price, blended, trace)

Reference: Netflix Research — Recommendations overview

Fuse Modalities Before Computing Item Similarity

Single-modality i2i is brittle: pure-visual similarity recommends visually-similar listings that have different pet requirements; pure-text similarity recommends listings with similar descriptions that look nothing alike; pure-metadata similarity recommends identical-category listings that don't actually match. The fix is to fuse modalities — visual CLIP embedding, description sentence-transformer embedding, and structured-feature one-hot vector — into a single item representation before building the ANN index. The simplest fusion is L2-normalised concatenation with per-modality weights learned offline against a golden i2i set.

Incorrect (single modality, picks vision and ignores everything else):

def item_vector(listing: Listing) -> np.ndarray:
    return feature_store.get(listing.id, "cover_photo_embedding")
    # visually-similar villas with totally different pet requirements are "similar"

Correct (normalised, weighted concatenation across modalities):

MODALITY_WEIGHTS = {
    "vision": 0.4,
    "text": 0.3,
    "structured": 0.3,
}  # tuned offline against a golden i2i set

def item_vector(listing: Listing) -> np.ndarray:
    vision = feature_store.get(listing.id, "listing_pooled_embedding")       # 512-dim
    text = feature_store.get(listing.id, "description_sentence_embedding")   # 384-dim
    structured = structured_feature_vector(listing)                          # 64-dim

    vision_n = vision / np.linalg.norm(vision) * MODALITY_WEIGHTS["vision"]
    text_n = text / np.linalg.norm(text) * MODALITY_WEIGHTS["text"]
    structured_n = structured / np.linalg.norm(structured) * MODALITY_WEIGHTS["structured"]

    return np.concatenate([vision_n, text_n, structured_n])
    # 960-dim fused vector fed to the ANN index

Reference: Airbnb — WIDeText: A Multimodal Deep Learning Framework

Precompute Item-to-Item Nearest Neighbours Offline

Computing cosine similarity across 500k listings at request time is a latency and compute disaster — the item shelf ("similar listings") is called on every listing page view, and recomputing neighbours on every request both burns money and bottlenecks the page. Build the i2i shelf offline as a batch job: compute pooled listing embeddings, build an ANN index (FAISS, HNSW, ScaNN, or a hosted service), and persist the top-K neighbours per listing into a key-value store keyed by listing ID. Serving then becomes a single-digit-millisecond key lookup instead of a kNN recompute.

Incorrect (recomputes neighbours at request time):

def similar_listings(listing_id: str, k: int = 12) -> list[str]:
    query_vec = feature_store.get(listing_id, "cover_photo_embedding")
    all_vecs = feature_store.scan_all("cover_photo_embedding")  # 500k items loaded every request
    sims = [(other_id, cosine(query_vec, vec)) for other_id, vec in all_vecs.items()]
    return [id for id, _ in sorted(sims, key=lambda x: -x[1])[1 : k + 1]]

Correct (precomputed offline, served from KV):

# Offline batch job (runs nightly)
def build_i2i_shelves():
    embeddings = feature_store.scan_all("listing_pooled_embedding")  # dict[id, np.ndarray]
    ids = list(embeddings.keys())
    matrix = np.stack([embeddings[i] for i in ids])

    index = faiss.IndexFlatIP(matrix.shape[1])
    index.add(matrix)

    _, neighbour_idx = index.search(matrix, k=25)  # self + 24 neighbours
    for i, listing_id in enumerate(ids):
        neighbours = [ids[j] for j in neighbour_idx[i] if ids[j] != listing_id][:24]
        i2i_store.put(listing_id, neighbours)

# Online serving
def similar_listings(listing_id: str, k: int = 12) -> list[str]:
    return i2i_store.get(listing_id)[:k]  # <5ms

Reference: Eugene Yan — Real-time Machine Learning For Recommendations

Score User-to-User Compatibility as Symmetric Mutual Fit

In a two-sided marketplace, a one-sided score ("this sitter scores 0.9 for this listing from the sitter's perspective") ignores whether the owner will accept, and produces requests that the owner rejects — both sides waste effort. The u2u compatibility score must be symmetric: min(P(owner_accepts | sitter, listing), P(sitter_requests | sitter, listing)) or the product of the two, so that high scores require both sides to say yes. Train the two sides as independent classifiers (or a joint two-sided two-tower), and always compose before ranking.

Incorrect (one-sided sitter-perspective score):

def u2u_score(sitter: Sitter, listing: Listing) -> float:
    return sitter_wants_this_listing_model.predict(sitter, listing)
    # high-scoring requests get rejected by the owner because the owner prefers someone else

Correct (symmetric fit — both sides must want the match):

def u2u_score(sitter: Sitter, listing: Listing) -> float:
    p_sitter_requests = sitter_wants_model.predict(sitter, listing)
    p_owner_accepts = owner_accepts_model.predict(listing, sitter)
    # use min (strict bottleneck) or product (geometric combination)
    return min(p_sitter_requests, p_owner_accepts)

# For ranking the owner's shortlist: rank by p_owner_accepts AND require p_sitter_requests > threshold
def shortlist_sitters_for_owner(listing: Listing, pool: list[Sitter], top_k: int = 10) -> list[Sitter]:
    scored = [
        (s, owner_accepts_model.predict(listing, s))
        for s in pool
        if sitter_wants_model.predict(s, listing) > 0.4  # feasibility gate
    ]
    return [s for s, _ in sorted(scored, key=lambda x: -x[1])[:top_k]]

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking

Use a Two-Tower Model for User-to-Item Affinity

Hand-crafted u2i scoring ("sitter pet-experience match × region overlap × availability") produces interpretable rules but hits a ceiling quickly — the weights are guesses and the interactions are missed. A two-tower model trains a user encoder and an item encoder jointly so that the dot product of their outputs approximates the acceptance probability, and the architecture naturally scales to retrieval via ANN over precomputed item vectors. Ship a two-tower as the u2i baseline once you have ~1M interaction events; hand-crafted scoring is only appropriate below that data scale or as an interpretable fallback.

Incorrect (hand-crafted scoring with fixed weights):

def score_listing_for_sitter(sitter: Sitter, listing: Listing) -> float:
    return (
        0.4 * pet_experience_match(sitter, listing)
        + 0.3 * region_overlap(sitter, listing)
        + 0.2 * availability_overlap(sitter, listing)
        + 0.1 * rating_match(sitter, listing)
    )
    # weights were guessed once and never updated; new features can only be added by hand

Correct (two-tower with trained embeddings):

class UserTower(nn.Module):
    def forward(self, user_features: dict) -> torch.Tensor:
        x = torch.cat([self.embed[f](user_features[f]) for f in self.FIELDS], dim=-1)
        return F.normalize(self.mlp(x), dim=-1)  # 128-dim user embedding

class ItemTower(nn.Module):
    def forward(self, item_features: dict) -> torch.Tensor:
        x = torch.cat([self.embed[f](item_features[f]) for f in self.FIELDS], dim=-1)
        return F.normalize(self.mlp(x), dim=-1)  # 128-dim item embedding

def train_step(batch, user_tower, item_tower, opt):
    u = user_tower(batch["user"])
    i_pos = item_tower(batch["item_pos"])
    i_neg = item_tower(batch["item_neg"])
    pos_score = (u * i_pos).sum(-1)
    neg_score = (u * i_neg).sum(-1)
    loss = -F.logsigmoid(pos_score - neg_score).mean()
    opt.zero_grad(); loss.backward(); opt.step()

# Serving: precompute item vectors offline, compute user vector per session, ANN lookup.
def score(sitter_id: str, listing_id: str) -> float:
    u = feature_store.get(sitter_id, "u2i_user_vector")
    i = feature_store.get(listing_id, "u2i_item_vector")
    return float(u @ i)

Reference: Shaped — The Two-Tower Model for Recommendation Systems: A Deep Dive

Ask What Signal a Human Uses to Make the Same Decision

A marketplace feature portfolio built by engineers guessing what matters is indistinguishable from random after 40 features. The reliable source is the people who already make the decision: interview 8-12 owners and 8-12 sitters, ask them to talk through three real listings they rejected and three they accepted, and write down every signal they name. Each named signal becomes a candidate feature with a clear hypothesis about the outcome it moves. Features that did not come from this process should justify themselves against features that did.

Incorrect (guesses what owners care about):

# engineering team whiteboard session, no owner interviews
CANDIDATE_FEATURES = [
    "listing.price",
    "listing.num_rooms",
    "listing.distance_to_city_center",
    "listing.internet_speed_mbps",  # nobody in the research asked about this
]

Correct (catalogues features from 10 real owner interviews):

# from structured interviews — each feature tagged with the owner quote that motivates it
CANDIDATE_FEATURES = {
    "has_fenced_garden": 'Owner A: "I wouldn\'t leave my dog with someone who doesn\'t have a fenced garden."',
    "sitter_has_cared_for_same_breed": 'Owner B: "I want someone who has had a labrador before."',
    "sitter_works_from_home": 'Owner C: "My dog gets anxious alone, I need someone home most of the day."',
    "review_count_from_owners_with_same_pet": 'Owner D: "5-star from a cat owner doesn\'t help me — I have two huskies."',
    "response_time_on_previous_requests": 'Owner E: "If they took 3 days to reply once, I skip them."',
}

def build_decision_features(sitter: Sitter, listing: Listing) -> dict:
    return {k: compute_feature(k, sitter, listing) for k in CANDIDATE_FEATURES}

Reference: Eugene Yan — Patterns for Personalization in Recommendations and Search

Kill Features a Popularity Baseline Already Captures

If your ship-criterion is "beat a top-N-by-completed-bookings popularity baseline", any feature whose signal is already dominated by completed-booking count will not move the needle — it correlates with what the baseline already ranks on. Before investing in a candidate feature, compute its correlation with booking count over a recent window. Features with Pearson ρ > 0.7 against booking count are almost certainly subsumed; features with ρ < 0.3 are the ones worth building. Kill the rest before they get into the store.

Incorrect (builds "review_count" as a feature without noticing it is colinear with the baseline):

feature_registry.register(
    name="listing_review_count",
    hypothesis="More reviews predict higher booking probability",
    primary_metric="booking_rate",
)
# correlation of review_count with completed_booking_count = 0.87
# the popularity baseline already ranks listings by booking count, so this feature
# adds negligible lift on top of baseline and bloats the store.

Correct (correlation screen before registration):

def screen_against_baseline(
    candidate_name: str, candidate_values: dict[str, float], booking_counts: dict[str, int]
) -> str:
    ids = list(candidate_values.keys() & booking_counts.keys())
    corr = pearsonr(
        [candidate_values[i] for i in ids],
        [booking_counts[i] for i in ids],
    ).statistic
    if abs(corr) > 0.7:
        return f"REJECT: {candidate_name} colinear with baseline (ρ={corr:.2f})"
    if abs(corr) < 0.3:
        return f"ACCEPT: {candidate_name} orthogonal to baseline (ρ={corr:.2f})"
    return f"INVESTIGATE: {candidate_name} partial overlap (ρ={corr:.2f})"

# registration is gated on this screen; reviewer comments required if correlation > 0.5.

Reference: Google — Rules of Machine Learning, Rule #1 and Rule #20

Prefer Directly Observed Features over Learned Features at Launch

The first version of any marketplace recommender should use features that the data already contains as typed columns — amenity lists, pet species, review counts, booking counts, wizard answers — before investing in embeddings, GNNs, or learned representations. Directly observed features are easy to explain, easy to validate, easy to debug, and easy to serve consistently offline and online. Learned features require training infrastructure, inference infrastructure, drift monitoring, and an ownership story. Start observed; add learned only when the observed feature portfolio has been exhausted and a specific gap is proven.

Incorrect (ships with a GNN over the booking graph as the first feature):

def build_sitter_features(sitter: Sitter) -> dict:
    return {
        "sitter_id": sitter.id,
        "graph_embedding": gnn_service.embed_node(sitter.id).tolist(),  # 256 floats, unexplainable
    }

Correct (ships with observed features; learned features deferred to v2):

def build_sitter_features(sitter: Sitter) -> dict:
    return {
        "sitter_id": sitter.id,
        "completed_stays_count": sitter.stats.completed_stays,
        "average_rating_received": sitter.stats.avg_rating,
        "review_count": sitter.stats.review_count,
        "verified_id": sitter.verification.id_verified,
        "has_senior_pet_experience": sitter.wizard.senior_pet_experience,
        "response_time_median_hours": sitter.stats.response_time_median_hours,
    }
    # v2 will add a graph embedding only after this v1 has shipped an A/B with a baseline.

Reference: Google — Rules of Machine Learning, Rule #17

Reject Features You Cannot Compute at Inference Time

A feature that is trivial to compute at training time over a warehouse batch job — "average rating over all reviews received in the last 30 days" — can be impossible to compute at inference time within a 50ms budget without a feature store, a stream processor, or a precomputation pipeline. If the serving path cannot produce the feature value for a live request, the model silently receives a zero or a stale value, and the distribution it saw at training diverges from what it sees in production. Reject any feature at the design stage whose serving story is not already wired up, or invest in the infrastructure before registering the feature.

Incorrect (training feature with no serving plan):

# training-time feature — heavy SQL join
def training_feature_avg_rating_30d(sitter_id: str) -> float:
    return db.query(
        "SELECT AVG(rating) FROM reviews "
        "WHERE sitter_id = :id AND created_at > NOW() - INTERVAL '30 days'",
        id=sitter_id,
    ).scalar()

# inference-time — the same SQL is too slow for a 50ms homefeed request
# someone patches in a cached value that is 24h stale; model now sees a different distribution online

Correct (serving-first design or hard rejection):

# Option A: feature is precomputed hourly by a batch job and served from the feature store
feature_store.put_batch(
    feature_group="sitter_stats_hourly",
    computed_at=now(),
    values=compute_avg_rating_30d_for_all_sitters(),
)

def online_serving_feature(sitter_id: str) -> float:
    return feature_store.get(entity_key=sitter_id, name="avg_rating_30d")

# Option B: the feature is rejected at registration time because serving is not wired up
feature_registry.register(
    name="avg_rating_30d",
    serving_source="feature_store.sitter_stats_hourly",  # required field, validated on registration
    serving_latency_p99_ms=8,
)

Reference: Feast — Training-Serving Skew

Start from the Decision, Not the Algorithm

Most feature engineering failures come from starting with the tools available ("we have CLIP, let's embed photos") instead of the specific decision the recommender is trying to improve ("does this owner trust this sitter enough to request a stay?"). Write the decision as a short sentence, decompose it into the 3-7 sub-judgments a human makes when answering it, and only then pick an extractor. Features that cannot trace back to a sub-judgment are almost certainly noise that will add maintenance cost without moving the primary metric.

Incorrect (tool-driven — starts from CLIP, ends in a generic embedding):

# "We have CLIP, let's use it."
def build_listing_features(listing: Listing) -> dict:
    photo = fetch(listing.cover_photo_url)
    return {
        "clip_embedding": clip_model.encode_image(photo).tolist(),
    }

Correct (decision-driven — decomposes "does this owner trust this sitter?" into sub-judgments):

# Decision: would an owner of a senior dog request this sitter?
# Sub-judgments a human makes:
#   1. Has this sitter cared for senior / medicated dogs? (experience signal)
#   2. Does this sitter's home look calm and safe for a senior dog? (vision signal)
#   3. Are there recent 5-star reviews specifically mentioning medication? (review signal)
#   4. Can this sitter commit to the timing? (availability signal)
#   5. Does the sitter's travel history overlap with this region? (geo signal)

def build_sitter_u2i_features(sitter: Sitter, listing: Listing) -> dict:
    return {
        "senior_dog_experience_count": sitter.stats.senior_dog_stays,
        "home_calmness_score": sitter.vision_features.calmness_score,  # narrow output, not raw embedding
        "medication_mentions_in_reviews": sitter.review_features.medication_mentions,
        "availability_overlaps_listing_dates": sitter.calendar.overlaps(listing.dates),
        "region_overlap_score": geo_overlap(sitter.travel_history, listing.region),
    }

Reference: Google — Rules of Machine Learning, Rule #17: Start with directly observed and reported features as opposed to learned features

Tie Every Feature to a Specific Solution and Metric

Every feature added to the store must name (a) which solution it feeds — item-to-item similarity (i2i), user-to-item affinity (u2i), or user-to-user mutual fit (u2u) — and (b) which primary metric it is hypothesised to improve (completed-booking rate, mutual-rating ≥4, request-to-acceptance rate). Features without a named solution and metric tend to be retained after the model that used them has been deleted, accumulating storage and drift risk with no owner. The solution tag is the contract that lets feature quality owners know whom to notify when a feature's coverage drops.

Incorrect (feature is added to a shared table with no downstream contract):

ALTER TABLE listing_features ADD COLUMN aesthetic_score FLOAT;
-- who uses this? what metric does it move? if CLIP is swapped, does anyone care?

Correct (feature is registered against a solution + metric + owner):

feature_registry.register(
    name="listing_aesthetic_score",
    feature_group="listing_vision",
    solutions=["i2i_similar_homes", "u2i_homefeed_ranker"],
    hypothesis="Aesthetic score correlates with sitter accept probability",
    primary_metric="completed_booking_rate_per_impression",
    owner="ml-marketplace@trustedhousesitters.com",
    dtype="float32",
    coverage_sla=0.85,
)
# registration fails if owner/metric/solution are empty; a feature without a contract does not reach production.

Reference: Google — Rules of Machine Learning, Rule #22: Clean up features you are no longer using

Declare Categorical Fields for Bounded Vocabularies

Fields with a small, enumerable vocabulary — region, property type, verification status, pet species, pet size class — should be declared as categorical features, not stuffed into a free-text description column. Categorical declaration lets the downstream model learn an embedding per value and condition ranking on exact values; free text is processed by a single text-bag tower that cannot discriminate between "terraced" and "semi-detached" meaningfully. Enumerate the vocabulary once in the schema, validate on write, and reject out-of-vocabulary values at ingestion so the feature store never sees dirty inputs.

Incorrect (free text for bounded attributes):

def build_listing_row(listing: Listing) -> dict:
    return {
        "listing_id": listing.id,
        "description": f"{listing.property_type} in {listing.region_name}, "
                       f"for {listing.pet_species} ({listing.pet_size}). "
                       f"{listing.user_description}",
        # all structure is in free text; model can't easily learn "semi-detached is popular in Leeds"
    }

Correct (structured categoricals with validated vocabularies):

PROPERTY_TYPE_VOCAB = {"detached", "semi_detached", "terraced", "apartment", "studio", "villa", "cottage"}
PET_SPECIES_VOCAB = {"dog", "cat", "rabbit", "reptile", "bird", "small_mammal", "fish"}
PET_SIZE_VOCAB = {"xsmall", "small", "medium", "large", "xlarge"}

def build_listing_row(listing: Listing) -> dict:
    assert listing.property_type in PROPERTY_TYPE_VOCAB
    assert listing.pet_species in PET_SPECIES_VOCAB
    assert listing.pet_size in PET_SIZE_VOCAB
    return {
        "listing_id": listing.id,
        "region_code": listing.region_code,           # categorical
        "property_type": listing.property_type,       # categorical
        "pet_species": listing.pet_species,           # categorical
        "pet_size": listing.pet_size,                 # categorical
        "description_text": listing.user_description, # free text only for what doesn't fit a vocabulary
    }

Reference: AWS Personalize — Items Dataset Schema Requirements

Embed Description Text with a Pretrained Sentence Encoder

Owner-written descriptions are the richest unstructured signal in the catalogue — they carry personality, calmness, formality, and specific requests that no structured field captures. A pretrained sentence encoder like all-MiniLM-L6-v2 produces 384-dim embeddings at 5x the speed of BERT-base with almost-equivalent quality on semantic similarity, turning every description into a usable feature in days without training. The embedding is what you feed downstream i2i composition and u2i ranking towers. Train a domain-specific encoder only after this v1 has shipped and the gap is measured.

Incorrect (TF-IDF over description, stale and sparse):

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(max_features=5000)
vectorizer.fit(all_descriptions)
def text_feature(desc: str) -> np.ndarray:
    return vectorizer.transform([desc]).toarray()[0]
    # sparse bag-of-words, no semantic understanding of synonyms or paraphrases

Correct (sentence-transformer embedding, 384-dim dense):

from sentence_transformers import SentenceTransformer

TEXT_ENCODER = "sentence-transformers/all-MiniLM-L6-v2"
encoder = SentenceTransformer(TEXT_ENCODER)

def text_feature(desc: str) -> np.ndarray:
    if not desc or len(desc) < 20:
        return np.zeros(384)
    vec = encoder.encode(desc, normalize_embeddings=True)
    return vec  # 384-dim, L2-normalised, ready for cosine similarity

# for non-English listings, route to the language-specific model (paraphrase-multilingual-MiniLM-L12-v2)

Reference: sentence-transformers/all-MiniLM-L6-v2 on Hugging Face

Encode Pet Requirements as Structured Triples

"Two dogs, one cat, one needs medication twice a day" as a free-text description forces every downstream model to re-parse the same sentence with its own errors. Decomposing pet requirements into a list of structured triples — (species, count, special_needs_tags) — lets the retrieval layer filter by species exactly, lets the ranker learn per-species acceptance probabilities, and lets the u2i matcher check each sitter's experience vector against each pet's specific needs. The owner still writes free text for flavour; the structured representation is what the system reasons on.

Incorrect (single free-text pet description):

def pet_feature(listing: Listing) -> str:
    return listing.pet_description
    # "Two small dogs and a senior cat who needs a pill each morning"
    # downstream models string-match on "senior" and "pill" hoping they catch it

Correct (structured triples + free text alongside):

@dataclass
class PetRecord:
    species: str            # from PET_SPECIES_VOCAB
    count: int
    size: str               # "xsmall" | "small" | "medium" | "large" | "xlarge"
    age_bucket: str         # "puppy_kitten" | "adult" | "senior"
    special_needs: list[str]  # ["medication", "anxiety", "mobility", "reactive"]

def pet_feature(listing: Listing) -> dict:
    return {
        "pet_records": [asdict(p) for p in listing.pet_records],
        "pet_description_text": listing.pet_description,  # still stored, used for vector embedding
        "total_pets": sum(p.count for p in listing.pet_records),
        "species_set": sorted({p.species for p in listing.pet_records}),
        "has_senior_pet": any(p.age_bucket == "senior" for p in listing.pet_records),
        "has_medicated_pet": any("medication" in p.special_needs for p in listing.pet_records),
    }

Reference: Airbnb — Applying Deep Learning To Airbnb Search

Extract Stay Duration Shape, Not Just Length

"14 days" is a number; "two-week stay over Christmas with flexible dates" is a cluster a sitter preference model can learn on. Extract the duration shape as multiple structured features: duration bin (weekend / short / standard / long / extended), whether it spans a holiday period, whether the dates are flexible, and the absolute day count. Sitters have preferences over the shape (weekend-only travellers, month-long retirees, flexible-schedule remote workers) that a raw day count collapses.

Incorrect (raw day count only):

def duration_feature(stay: Stay) -> int:
    return (stay.end_date - stay.start_date).days
    # a model sees "14" and cannot tell it's Christmas break vs a random two weeks in March

Correct (duration bin + shape + flexibility + holiday overlap):

DURATION_BINS = [
    ("weekend", lambda d: d <= 3),
    ("short", lambda d: 4 <= d <= 7),
    ("standard", lambda d: 8 <= d <= 14),
    ("long", lambda d: 15 <= d <= 30),
    ("extended", lambda d: d > 30),
]

def duration_feature(stay: Stay) -> dict:
    days = (stay.end_date - stay.start_date).days
    bin_label = next(label for label, pred in DURATION_BINS if pred(days))
    return {
        "duration_days": days,
        "duration_bin": bin_label,                       # categorical
        "spans_public_holiday": spans_holiday(stay),
        "spans_school_holiday": spans_school_holiday(stay, stay.region_code),
        "dates_are_flexible": stay.flexible,
        "start_weekday": stay.start_date.strftime("%A").lower(),
    }

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking

Hash Geo to Hierarchies, Not Raw Lat/Lon

Raw (latitude, longitude) pairs are two continuous floats — the model has no native way to learn "this area is popular with cat sitters" or "this neighbourhood has a high acceptance rate". Hashing the location into a spatial index (H3, S2, or geohash) at multiple resolutions produces categorical features the model can embed per hash cell — coarse cells capture city-level effects, fine cells capture street-level effects, and an ablation study can tell you which resolution moves the metric. Privacy benefits are a bonus: publishing an H3-resolution-9 cell instead of an exact address protects the owner from scraping.

Incorrect (raw lat/lon floats):

def geo_feature(listing: Listing) -> dict:
    return {
        "latitude": listing.lat,   # float
        "longitude": listing.lon,  # float
        # model treats these as arbitrary numbers; no neighbourhood learning
    }

Correct (H3 cells at multiple resolutions):

import h3

H3_RESOLUTIONS = [5, 7, 9]
# r5 ≈ city  (252 km² area, ~8.5 km edge)
# r7 ≈ neighbourhood (5.16 km² area, ~1.22 km edge)
# r9 ≈ street (0.1 km² area, ~174 m edge)

def geo_feature(listing: Listing) -> dict:
    return {
        f"h3_r{res}": h3.latlng_to_cell(listing.lat, listing.lon, res)
        for res in H3_RESOLUTIONS
    }
    # {"h3_r5": "85283473fffffff", "h3_r7": "872830828ffffff", "h3_r9": "89283082837ffff"}
    # model learns embeddings per cell at each resolution; neighbourhood effects emerge naturally

Reference: Uber — H3 Hexagonal Hierarchical Spatial Index

Encode Amenity Lists as Multi-Hot Vectors, Not Free-Text Strings

Owners select amenities from a checklist of ~50 options (wifi, fenced garden, washing machine, pet-friendly yard, wheelchair-accessible, etc.) — the checklist is a fixed vocabulary, so the feature should be a multi-hot vector, not a comma-joined string. Multi-hot encoding gives the ranker a feature per amenity it can learn a weight for (sitters with large dogs care about fenced gardens), enables efficient AND/OR retrieval at candidate generation time, and makes amenity-based rules debuggable. A comma-joined string forces the downstream model to re-split the string at every inference and hopes the tokenization is consistent.

Incorrect (joined string that requires re-tokenization downstream):

def amenity_feature(listing: Listing) -> str:
    return ",".join(listing.amenities)  # "wifi,fenced_garden,washing_machine,cat_flap"
    # downstream model string-splits and text-hashes at every inference, inconsistent across clients

Correct (fixed-vocabulary multi-hot):

AMENITY_VOCAB = [
    "wifi", "fenced_garden", "washing_machine", "dryer", "dishwasher",
    "cat_flap", "pet_door", "garage", "driveway", "fireplace",
    "heating_central", "air_conditioning", "wheelchair_accessible",
    # ...complete, versioned vocabulary
]
AMENITY_INDEX = {a: i for i, a in enumerate(AMENITY_VOCAB)}

def amenity_feature(listing: Listing) -> list[int]:
    vector = [0] * len(AMENITY_VOCAB)
    for a in listing.amenities:
        if a in AMENITY_INDEX:
            vector[AMENITY_INDEX[a]] = 1
        # silently-ignored unknowns are a red flag — add an assert if strict
    return vector

# retrieval: candidate = amenities.contains_all(required_amenities)
# ranking: learned weight per position in the vector

Reference: AWS Personalize — Items Dataset Schema Requirements

Discovery Playbook: From Raw Asset to Shipped Recsys Feature

This playbook walks through the end-to-end workflow for discovering, extracting, composing, and shipping a new recommender feature in a two-sided trust marketplace. It composes the rules from every category into a seven-step process that starts with an asset audit and a decision decomposition and ends with an ablation A/B test against a feature-ablated baseline.

Use this playbook when:

• The task is "how do we extract more signal from our photos / metadata / wizard?"
• A sibling skill (marketplace-personalisation, marketplace-search-recsys-planning)

has identified a feature gap and the question is what to build

• Planning the next quarter of feature-engineering work
• Diagnosing a feature portfolio that has grown quickly without a discipline

Skip to the individual rules when a specific implementation question arises mid-stream (e.g., "which text encoder should I use?" — go straight to `listing-embed-description-with-pretrained-sentence-encoder`).

Summary

Total: 6-10 weeks from "we have raw assets" to "a shipped feature with proven online lift." Every step has an exit criterion — do not proceed to the next step until the current step passes.

Step 1 — Audit the Raw Assets

Goal: an honest, numeric picture of every asset type you might extract features from.

Run the audit against every source table and object store that could feed features: the listings table, the listing_photos bucket, the sitter_profiles table, the wizard responses store. For each one, produce three numbers: coverage per field (`audit-measure-coverage-before-modelling`), end-to-end fetchability of a random sample (`audit-sample-every-asset-type-end-to-end`), and freshness distribution (`audit-quantify-freshness-per-asset`).

In parallel, confirm rights and privacy (`audit-verify-rights-and-privacy-before-extraction`) and separate raw storage from derived storage (`audit-separate-raw-assets-from-derived-features`) before the first extractor runs.

Exit criterion: a one-page audit document for every asset type, listing coverage (≥80% gate), fetch success rate (≥95% gate), freshness median and tail, ToS status, consent flag status, and which fields are excluded from the feature plan with reason.

Step 2 — Decompose the Decision

Goal: write the specific decision this feature is meant to help the recommender make.

Do not start from CLIP or two-tower or anything else (`firstp-start-from-the-decision-not-the-algorithm`). Start from a sentence like "would an owner of a senior dog request this sitter?" or "which two listings are similar enough that a sitter who booked one would book the other?" Decompose the decision into 3-7 sub-judgments a human would make to answer it.

Interview 8-12 real owners and 8-12 real sitters and ask them to talk through three listings they accepted and three they rejected (`firstp-ask-what-signal-a-human-uses`). Every named signal is a candidate feature with a quote attached.

Exit criterion: a written decision statement, a list of 3-7 sub-judgments with human quotes backing each, and a shortlist of 5-15 candidate features.

Step 3 — Pick One Candidate Feature

Goal: one feature, one solution, one metric, one owner, one hypothesis.

From the shortlist, pick the single highest-leverage feature. Tie it to a specific downstream solution — i2i similarity shelf, u2i homefeed ranker, or u2u matchmaking — and a specific primary metric (`firstp-tie-every-feature-to-a-specific-solution`). Run a correlation screen against the current popularity baseline to kill features that are subsumed by booking count (`firstp-kill-features-a-popularity-baseline-already-captures`). Prefer directly observed features over learned ones unless a learned feature is specifically justified (`firstp-prefer-directly-observed-over-learned`). Reject any feature whose serving story is not already wired up (`firstp-reject-features-you-cannot-serve-at-inference`).

Exit criterion: one feature picked, one-page RFC listing owner, solution, metric, hypothesis, serving source, correlation against baseline, and the specific sub-judgment it is meant to improve.

Step 4 — Prototype the Extractor

Goal: working extractor code on a 5k-item sample, within the serving budget.

Route to the appropriate extraction category based on the source:

• Images: start with zero-shot CLIP

(`vision-use-clip-for-zero-shot-listing-embeddings`), detect room type before amenities (`vision-detect-room-types-before-detecting-amenities`), extract quality separately from content (`vision-quantify-image-quality-separately-from-content`), count objects per class (`vision-extract-per-object-counts-not-just-presence`), and pool across the listing's full photo set (`vision-pool-embeddings-across-a-listings-photo-set`).

• Listing text and metadata: declare categorical fields

(`listing-declare-categorical-fields-for-bounded-vocabularies`), multi-hot amenities (`listing-multi-hot-encode-amenity-lists`), hash geo (`listing-hash-geo-to-hierarchies-not-raw-lat-lon`), embed descriptions with Sentence-Transformers (`listing-embed-description-with-pretrained-sentence-encoder`), and extract duration shape and pet triples (`listing-extract-stay-duration-shape-not-just-length`, `listing-encode-pet-requirements-as-structured-triples`).

• Sitter wizard: restructure questions by information gain

(`wizard-order-questions-by-information-gain`), convert free text to multiple choice (`wizard-prefer-multiple-choice-over-free-text`), log skips (`wizard-make-skips-genuine-and-log-them`), capture experience numerically (`wizard-capture-experience-as-counts-and-dates`), and separate hard constraints from soft preferences (`wizard-separate-hard-constraints-from-soft-preferences`).

Benchmark latency and memory on the prototype before declaring it viable.

Exit criterion: the extractor produces feature values for a 5k-item sample, the p99 latency is within the serving budget, and the output passes a spot-check on 20 hand-picked items.

Step 5 — Compose Into the Target Solution

Goal: the feature is used by its named consumer (i2i, u2i, or u2u).

Route by consumer:

• i2i: fuse modalities

(`derive-fuse-modalities-before-item-similarity`) and precompute the shelf offline (`derive-precompute-i2i-nearest-neighbours-offline`).

• u2i: add the feature to the item tower or user tower of a two-tower model

(`derive-use-two-tower-for-user-item-affinity`), decompose into interpretable subscores (`derive-decompose-affinity-into-interpretable-subscores`), and cache the user vector (`derive-cache-user-embedding-with-short-ttl`).

• u2u: wire both sides of the mutual-fit score

(`derive-score-u2u-as-symmetric-mutual-fit`) so that no request is generated where the owner would reject.

Exit criterion: an end-to-end offline pipeline produces ranked outputs for a representative set of queries or users, and the outputs are hand-reviewable.

Step 6 — Register, Gate, and Productionise

Goal: the feature exists in a single registry, is gated on coverage and drift, and is served by a single store for both training and inference.

Register the feature in the feature registry with owner, metric, solution, and serving path (`quality-version-feature-definitions-in-one-registry`). Point training and inference at the same feature store (`quality-serve-training-and-inference-from-one-store`). Wire coverage and drift alarms (`quality-gate-features-on-coverage-and-drift`). Scrub PII at the extraction boundary (`quality-scrub-pii-before-features-leave-secure-zone`). Freeze the feature schema per model version (`quality-freeze-feature-schemas-per-model-version`).

Exit criterion: the feature is queryable by training and serving code through the same interface, coverage and PSI alarms are firing normally, and schema hash is committed to the next model artifact.

Step 7 — Ablation A/B and Decide

Goal: a statistically significant online decision about whether the feature earns its place.

Train two models with identical hyperparameters, one with the feature and one without it (`prove-measure-lift-against-feature-ablated-variant`). Ship the feature-included variant as treatment, the feature-excluded variant as control — not the previous production model as control. Run exactly one feature per experiment (`prove-ship-one-feature-at-a-time`). Reserve a 3-5% exploration slice if the offline metric was close to tie (`prove-dedicate-random-exploration-slice-to-new-features`), retain the permanent feature-free baseline slice (`prove-retain-feature-free-baseline-permanently`), and put the feature on the next quarterly kill review (`prove-kill-features-that-dont-earn-maintenance`).

Exit criterion: a shipped decision (ship, kill, iterate) written into the decisions log with reason, lift, confidence interval, segment breakdown, and the next step for the feature.

After the Cycle

Successful features graduate into permanent production and feed the downstream skills:

• The marketplace-personalisation skill's recipe selection and ranking uses

the new features via the feature store.

• The marketplace-search-recsys-planning skill's retrieval layer uses them for

candidate generation and rescoring.

• The marketplace-pre-member-personalisation skill's pre-member ranking can

reference them once the anonymous session has enough signal to match.

Killed features are archived and the registry entry is removed; the lessons learned are added to `../../gotchas.md` so the next discovery cycle starts smarter.

Dedicate a Random Exploration Slice to New Features

A new feature that looks weak on the offline golden set can win online — because the offline set is a frozen snapshot of past behaviour and cannot reward a feature that unlocks new matches the system has never made. Reserve 3-5% of traffic as a permanent exploration slice where the current best model is replaced by candidate models that include features whose offline numbers are close-to-tied. The slice produces unbiased training data for the next model version and catches features that offline evaluation would reject.

Incorrect (features that don't win offline are never tried online):

# ablation on golden set shows +0.2% NDCG, p = 0.08 → "not worth A/B-ing"
# the feature is shelved forever; online behaviour change is never measured

Correct (exploration slice captures close calls):

EXPLORATION_SLICE_PCT = 0.04  # 4% of homefeed requests

def route_request(sitter_id: str) -> Model:
    session_hash = hash(f"{sitter_id}:{today()}") % 1000
    if session_hash < EXPLORATION_SLICE_PCT * 1000:
        return pick_exploration_candidate()  # rotates through offline-close-to-tied candidates
    return production_model()

def pick_exploration_candidate() -> Model:
    # currently rotating: model_v14_base, model_v14_with_description_embed, model_v14_with_h3_geo
    candidates = registry.models_tagged("exploration_candidate_active")
    return random.choice(candidates)

# log every exploration impression with candidate_id + outcome;
# each candidate graduates to full A/B when its exploration lift clears the noise floor.

Reference: DoorDash — Homepage recommendation with exploitation and exploration

Kill Features That Do Not Earn Their Maintenance Cost

Features accumulate cost long after they stop earning lift: storage, serving latency, drift monitoring, schema migration risk, cognitive load during debugging. Any feature whose ablation test shows a lift below the maintenance cost threshold — typically anything under a 0.5% statistically-significant improvement on the primary metric — should be killed, its registry entry archived, its storage freed, and its computation dag entries removed. This is unpopular (someone spent a quarter building it) but necessary: a 60-feature portfolio with 15 useful features is strictly worse than a 15-feature portfolio.

Incorrect (keeps every feature ever shipped because nobody owns deletion):

# feature store has 140 features
# 40 were disabled two years ago but never deleted; their drift alarms still fire
# every debug session starts by asking "is this feature still used?"

Correct (quarterly kill review based on attribution):

@dataclass
class FeatureAudit:
    name: str
    last_trained_in_model: datetime
    attributed_lift_pct: float  # from its ablation A/B
    monthly_maintenance_hours: float

def quarterly_kill_review(features: list[FeatureAudit]) -> list[str]:
    to_kill = []
    for f in features:
        age = datetime.now() - f.last_trained_in_model
        if age > timedelta(days=180) and f.attributed_lift_pct < 0.5:
            to_kill.append(f.name)
        if f.monthly_maintenance_hours > 2 and f.attributed_lift_pct < 0.5:
            to_kill.append(f.name)
    return to_kill

def archive_feature(name: str):
    feature_registry.archive(name)
    feature_store.stop_computing(name)
    feature_store.delete_monitoring(name)
    ann_index.rebuild_without(name)

Reference: Google — Rules of Machine Learning, Rule #22: Clean up features you are no longer using

Measure Lift Against a Feature-Ablated Variant, Not the Old Model

A/B testing a new feature by comparing "old model" vs "new model with the feature" confounds the feature's contribution with any incidental changes in the model version — different hyperparameters, different training data window, different code path. The correct comparison is "new model with the feature" vs "new model without the feature", trained with identical hyperparameters on identical data, varying only the presence of the feature. This is the ablation variant, and it is the only evidence that isolates the feature's specific contribution.

Incorrect (compares new model against old; cannot isolate the feature's effect):

# control: model_v13 (from March, trained on February data)
# treatment: model_v14 (from April, trained on March data, adds the new feature)
# result: +3% lift — but was it the feature, the fresh data, or the new hyperparameters?

Correct (ablation variant: same model, feature masked):

# both variants trained this week with identical hyperparameters, identical training window
FEATURE_UNDER_TEST = "listing_pooled_embedding"

control_features = ALL_V14_FEATURES - {FEATURE_UNDER_TEST}
treatment_features = ALL_V14_FEATURES

control_model = train_model(feature_set=control_features, seed=42, **v14_hparams)
treatment_model = train_model(feature_set=treatment_features, seed=42, **v14_hparams)

run_ab_test(
    experiment="listing_pooled_embedding_ablation",
    control=control_model,
    treatment=treatment_model,
    # any lift is attributable to the feature, not to training data or hyperparameter drift
)

Reference: Kohavi — Trustworthy Online Controlled Experiments

Retain a Feature-Free Baseline Permanently

A popularity baseline (top-N listings by completed bookings in the region over the last 30 days) uses zero learned features and should remain a small permanent traffic slice even after the ML model has been winning for a year. Its purpose is not to beat the ML model — it will not — but to act as a drift anchor: if the gap between the ML model and the baseline shrinks below a threshold, something has gone wrong (drift, coverage drop, broken feature). The baseline is the canary, and retaining it is the cheapest feature-portfolio insurance you can buy.

Incorrect (baseline retired as soon as ML proves itself):

# experiment "ml_v1 vs popularity" finished with +4% lift, popularity variant deleted
# 6 months later, a feature drop silently regresses the ML model to baseline-level performance
# nobody notices because there is no comparison point

Correct (permanent 2% baseline slice with an alarm on the gap):

BASELINE_TRAFFIC_PCT = 0.02
GAP_ALARM_THRESHOLD_PCT = 1.5  # if ML only beats baseline by <1.5%, page the team

def route(sitter_id: str) -> Model:
    bucket = hash(sitter_id) % 100
    if bucket < int(BASELINE_TRAFFIC_PCT * 100):
        return popularity_baseline_model()
    return current_ml_model()

def daily_gap_check() -> None:
    ml_rate = online_metrics.booking_rate(model="ml_v14", window="24h")
    base_rate = online_metrics.booking_rate(model="popularity_baseline", window="24h")
    gap_pct = 100 * (ml_rate - base_rate) / base_rate
    if gap_pct < GAP_ALARM_THRESHOLD_PCT:
        alert(f"ML-vs-baseline gap collapsed to {gap_pct:.1f}%", severity="page")

Reference: Google — Rules of Machine Learning, Rule #1: Don't be afraid to launch a product without machine learning

Ship One Feature at a Time in the First Year

Shipping three features together in a single release and seeing a 2% lift tells you nothing about which of the three moved the metric — and the one that regressed is hidden by the two that helped. In the first year of a feature portfolio, release exactly one feature per A/B test: one new feature goes into the treatment, the control has the model without it, and the ship/kill decision is per feature. Once the portfolio is mature and the team has credibility, bundled releases become defensible; before that, bundling is the fastest way to accumulate features that nobody can defend.

Incorrect (three features ship together):

# experiment ths_homefeed_ml_v2
# treatment: +amenity_multihot, +pet_description_embedding, +sitter_experience_count
# control: previous model
# result: +1.8% booking rate, p < 0.05 → "ship"
# but which feature actually helped? and which one regressed a subsegment?

Correct (three separate experiments, serial or parallel with disjoint populations):

EXPERIMENTS = [
    {
        "name": "ths_homefeed_amenity_multihot",
        "treatment": "model_v14_with_amenity_multihot",
        "control": "model_v14_without_amenity_multihot",
    },
    {
        "name": "ths_homefeed_pet_description_embedding",
        "treatment": "model_v14_with_pet_desc_embed",
        "control": "model_v14_without_pet_desc_embed",
    },
    {
        "name": "ths_homefeed_sitter_experience_count",
        "treatment": "model_v14_with_experience_count",
        "control": "model_v14_without_experience_count",
    },
]
# ship decision made per experiment; losers are killed without blocking the winners

Reference: Google — Rules of Machine Learning, Rule #16: Plan to launch and iterate

Freeze Feature Schemas per Model Version

Changing a feature's dtype, adding a new categorical value, or renaming a field mid-flight breaks the training-serving contract silently — the serving path keeps working on old inputs while the next training run sees the new schema and learns different weights. Freeze the feature schema per deployed model version: record the schema hash at training time, store it with the model artifact, and refuse to deploy a model whose schema hash does not match the current feature store state. Schema changes get a new model version; they never retroactively modify an existing one.

Incorrect (schema changes ripple into deployed models):

# deployed model v14 was trained on amenity_vocab_v1 (50 items)
# someone adds a new amenity "ev_charger" — feature store now produces 51-dim multi-hot
# model v14 receives 51-dim input, silently truncates or errors
AMENITY_VOCAB.append("ev_charger")  # no version bump

Correct (schema is pinned to the model version):

@dataclass(frozen=True)
class FeatureSchema:
    name: str
    dtype: str
    vocab_hash: str | None = None  # for categorical/multi-hot
    version: str = "v1"

AMENITY_SCHEMA_V1 = FeatureSchema(
    name="listing_amenities",
    dtype="multi_hot_50",
    vocab_hash="a1b2c3d4",  # sha1 of sorted(vocab)
    version="v1",
)

def train_model(features: dict[str, FeatureSchema]) -> Model:
    model = train(...)
    model.schema_hashes = {name: schema.vocab_hash for name, schema in features.items()}
    return model

def deploy_model(model: Model) -> None:
    for name, expected_hash in model.schema_hashes.items():
        current_hash = feature_registry.get(name).vocab_hash
        if current_hash != expected_hash:
            raise DeployError(
                f"{name} schema changed ({expected_hash} → {current_hash}); train a new model version"
            )
    deploy_to_production(model)

Reference: Uber — Evolving Michelangelo Model Representation for Flexibility at Scale

Gate Every Feature on Coverage and Drift Alarms

A feature that starts at 95% coverage can silently degrade to 62% over a quarter as an upstream API changes, a migration is deployed, or a wizard question is retired — and the model's online metrics drift a week later when the gap has widened enough to matter. Every feature in production needs two monitors: a coverage alarm (population with non-null values should stay within ±3% of the baseline) and a drift alarm (population-stability index against a frozen reference window should stay under a threshold). Both are cheap to compute and catch regressions before they land in booking rate.

Incorrect (no monitoring; find out from customer complaints):

# feature is deployed with no coverage or drift alarms
feature_store.put_batch(feature_group="listing_vision", values=compute_vision_features())
# two weeks later, CLIP-ingester bug drops coverage from 96% to 61%, nobody notices

Correct (coverage + PSI checks wired into the deploy pipeline):

def coverage(values: list[float | None]) -> float:
    return sum(1 for v in values if v is not None) / len(values)

def population_stability_index(ref: list[float], current: list[float], bins: int = 10) -> float:
    quantiles = np.quantile(ref, np.linspace(0, 1, bins + 1))
    ref_dist, _ = np.histogram(ref, bins=quantiles)
    cur_dist, _ = np.histogram(current, bins=quantiles)
    ref_p = ref_dist / ref_dist.sum()
    cur_p = cur_dist / cur_dist.sum()
    eps = 1e-6
    return float(np.sum((cur_p - ref_p) * np.log((cur_p + eps) / (ref_p + eps))))

def post_deploy_check(feature_name: str, reference_window: str, current_window: str) -> None:
    ref = feature_store.query_window(feature_name, reference_window)
    cur = feature_store.query_window(feature_name, current_window)

    cov = coverage(cur)
    assert cov >= 0.80, f"{feature_name} coverage {cov:.2f} below 0.80 floor"

    psi = population_stability_index([v for v in ref if v is not None], [v for v in cur if v is not None])
    if psi > 0.25:
        alert(f"{feature_name} PSI={psi:.2f}", severity="page")
    elif psi > 0.1:
        alert(f"{feature_name} PSI={psi:.2f}", severity="warn")

Reference: Great Expectations — Why Data Quality is Key to Successful MLOps

Scrub PII Before Features Leave the Secure Zone

Image embeddings from photos containing faces, text embeddings from descriptions containing phone numbers, and user embeddings that memorise unique rare features are all privacy risks — an attacker with query access to the ANN index can recover that a specific person exists in the training data. PII scrubbing must happen at the extraction boundary, not at the serving boundary: blur faces before CLIP encoding, regex-scrub phone numbers and postcodes before the text encoder sees them, and use differential privacy or k-anonymity on per-user features that might be unique. Once the embedding contains PII, removing it later is effectively impossible.

Incorrect (PII scrubbing deferred until after embedding):

def embed_description(text: str) -> np.ndarray:
    return text_encoder.encode(text)
    # "Call me on +44 7911 123456 if you need me" embeds the number into a 384-dim vector
    # later attempts to "redact" the phone number from the embedding are impossible

Correct (PII scrubbed at the boundary, before extraction):

PII_PATTERNS = [
    (r"\+?\d[\d\s().-]{7,}\d", "[PHONE]"),
    (r"[\w.+-]+@[\w-]+\.[\w.-]+", "[EMAIL]"),
    (r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b", "[POSTCODE]"),  # UK postcode
    (r"\b\d{1,5}\s+[A-Z][a-z]+\s+(Street|Road|Avenue|Lane)\b", "[ADDRESS]"),
]

def scrub_pii(text: str) -> str:
    for pattern, replacement in PII_PATTERNS:
        text = re.sub(pattern, replacement, text)
    return text

def embed_description(text: str) -> np.ndarray:
    scrubbed = scrub_pii(text)
    return text_encoder.encode(scrubbed)

def embed_photo(photo_bytes: bytes) -> np.ndarray:
    blurred = face_blur.apply(photo_bytes)  # face detection + gaussian blur on face regions
    return image_encoder.encode(blurred)

Reference: Airbnb — When a Picture Is Worth More Than Words