Opensearch Function Scoring Algorithms

Marketplace-Research OpenSearch Function Scoring Best Practices

A reference distillation of research-backed algorithms for ranking in two-sided marketplaces (Airbnb, Uber Eats, DoorDash, Etsy, eBay, Booking.com) implemented on OpenSearch or Elasticsearch. Contains 56 rules across 9 categories, prioritised by cascade effect in the search ranking pipeline. Each rule explains the WHY (the cascade or the bias it corrects), shows incorrect-vs-correct code (OpenSearch JSON queries, Painless scripts, Python pre-processing, evaluation methodology), and links to the canonical source — KDD/SIGIR/WSDM papers, the OpenSearch documentation, and the engineering blogs of the marketplaces that proved these patterns at scale.

When to Apply

Reach for this skill when:

• Designing a new marketplace search system on OpenSearch or Elasticsearch from scratch
• Tuning function_score / rank_feature / script_score queries that aren't moving the needle
• Setting up hybrid retrieval (BM25 + dense vectors) with Reciprocal Rank Fusion
• Choosing between HNSW and IVF for billion-scale ANN indexes
• Adding personalization via listing/user embeddings or two-tower architectures
• Correcting position bias in click logs before retraining an LTR model
• Designing exposure-fairness or new-listing cold-start exposure allocation
• Composing decay functions (gauss / exp / linear) over geo + date + freshness
• Diversifying the top window with MMR, DPP, or per-host caps
• Debugging "why does my top-10 show 8 listings from one host?" or "why does ranking favor popular incumbents?"
• Building offline evaluation infrastructure — graded judgment sets, NDCG@k pipelines, ablation studies, regression query suites
• Designing A/B tests for ranking changes — MDE / power / sample-size pre-computation, CUPED variance reduction, online-offline correlation calibration
• Attributing lift to specific scoring components — "did my new bias-correction help, or was it the embeddings, or both?"

The rules apply to any OpenSearch/Elasticsearch-backed marketplace search regardless of vertical — accommodation, food delivery, restaurants, services, jobs, secondhand goods, real estate. Triggers include "marketplace ranking", "search relevance", "function_score", "rank_feature", "script_score", "kNN", "hybrid search", "RRF", "learning to rank", "embedding-based retrieval", "two-tower", "position bias", "MMR", "supply fairness", "Pareto multi-objective", "NDCG", "judgment set", "ablation study", "CUPED", "A/B sample size", "ranking eval", and "why are my search results bad".

The Search Ranking Lifecycle

Categories are derived from the marketplace search ranking pipeline. Earlier stages cascade — a miss in recall (stage 1) cannot be repaired by any downstream boost, and a wrong base relevance multiplies through every functional score:

Query → [1] Recall → [2] Base Relevance → [3] Quality Signals → [4] Personalization
      → [5] Geo/Time Decay → [6] Marketplace Balance → [7] Diversity Re-rank → Results
                                                            ↑
                                          [8] Bias Correction (applied across all stages
                                                       and into training)
                                                            ↑
                                          [9] Evaluation & Measurement (the meta-layer:
                                                       judgment sets, NDCG, ablation, A/B
                                                       sizing, CUPED — without these you
                                                       can't tell if any rule helped)

Rule Categories by Priority

Quick Reference

1. Candidate Retrieval & Recall (CRITICAL)

• `recall-hybrid-rrf` — Use Hybrid BM25 + kNN with Reciprocal Rank Fusion
• `recall-two-tower-ebr` — Use Two-Tower Architecture for Embedding-Based Retrieval
• `recall-prefilter-knn` — Apply Pre-Filter to kNN with Hard Constraints
• `recall-hnsw-vs-ivf` — Choose HNSW for Latency, IVF for Memory at Scale
• `recall-multi-stage` — Split Retrieval into Cheap Recall and Expensive Re-rank
• `recall-query-expansion` — Apply Synonym Expansion at Index Time for Recall, Query Time for Precision

2. Base Relevance & Field Scoring (CRITICAL)

• `rel-bm25f-field-weights` — Tune BM25F Field Weights Before k1/b
• `rel-multi-match-strategy` — Pick multi_match Type by Query Shape, Not by Default
• `rel-bm25-k1-b-tuning` — Tune BM25 k1 and b Per-Field for Short Marketplace Documents
• `rel-listwise-loss` — Prefer Listwise (LambdaMART) over Pairwise (RankNet) LTR Loss
• `rel-script-score-over-function-score` — Use script_score Query, Not function_score, for Composition
• `rel-rescore-over-bool-should` — Use rescore Phase for Heavy Scoring, Not bool/should at Retrieval
• `rel-avoid-boost-inflation` — Avoid Field-Boost Inflation Above ~10x

3. Quality Signals & Confidence Bounds (HIGH)

• `qual-wilson-lower-bound` — Sort by Wilson Lower Bound, Not Average Rating
• `qual-bayesian-average` — Use Bayesian Average for Star Ratings with Low Sample Sizes
• `qual-rank-feature-saturation` — Saturate Popularity Counts with rank_feature.saturation
• `qual-rank-feature-sigmoid` — Apply Sigmoid Modifier for Bounded Ratio Signals
• `qual-log1p-vs-saturation` — Choose log1p over Saturation for Long-Tail Signal Preservation
• `qual-completeness-score` — Score Listing Completeness as a Quality Signal

4. Personalization & Embeddings (HIGH)

• `pers-listing-embeddings` — Train Listing Embeddings from Booking-Session Co-occurrence
• `pers-type-embeddings-cold-start` — Use Type Embeddings for Cold-Start Users and Listings
• `pers-real-time-session-vector` — Update Session Vector in Real-Time from Click Events
• `pers-multi-modal-embeddings` — Use Multi-Modal Embeddings (Text + Image) for Recall
• `pers-cross-encoder-rerank` — Apply Cross-Encoder Re-rank on Top-50 for Personalization
• `pers-tower-split-offline-online` — Split Item Tower Offline, Query Tower Online
• `pers-contextual-features` — Inject Contextual Features into script_score

5. Spatial & Temporal Decay (HIGH)

• `decay-gauss-geo` — Use Gauss Decay for Geo Distance, Not Linear
• `decay-exp-freshness` — Use Exp Decay for Time Freshness, Gauss for Date Proximity
• `decay-scale-calibration` — Calibrate Decay Scale to the 0.5-Score Distance Target
• `decay-offset-noise` — Add Offset to Decay Functions for Noisy Sparse Fields
• `decay-multi-field-composition` — Compose Multi-Field Decay with Explicit Weights

6. Two-Sided Marketplace Balance (HIGH)

• `market-conversion-weighted-ranking` — Weight Ranking by Conversion Rate, Not Click-Through Rate
• `market-cold-start-exploration` — Boost Cold-Start Listings with Bounded Exposure Allocation
• `market-supply-fairness-lorenz` — Monitor Supply-Side Fairness with Lorenz/Gini Metrics
• `market-host-quality-signals` — Separate Host-Quality and Listing-Quality Signals
• `market-inventory-health` — Penalize Listings with Low Inventory Health
• `market-pareto-multi-objective` — Optimize Multi-Objective Ranking with Pareto-Aware Weights
• `market-price-relevance` — Score Price Relevance with Soft Bands, Not Hard Filters

7. Bias Correction & Online Learning (HIGH)

• `bias-position-ips` — Correct Position Bias with Inverse Propensity Scoring
• `bias-click-models` — Estimate Click Propensities with PBM, Cascade, or DBN
• `bias-thompson-sampling` — Explore Ranking Alternatives with Thompson Sampling
• `bias-counterfactual-eval` — Validate Ranking Changes with Counterfactual Evaluation
• `bias-interleaved-evaluation` — Use Interleaved Evaluation for Low-Traffic Ranking Comparisons
• `bias-popularity-debiasing` — Subsample Popular Items in Embedding Training Negatives

8. Evaluation & Measurement (HIGH)

• `eval-graded-judgment-set` — Build a Graded Judgment Set for Offline Evaluation
• `eval-ndcg-primary-metric` — Use NDCG@k as the Primary Offline Ranking Metric
• `eval-online-offline-correlation` — Validate Online-Offline Metric Correlation Before Trusting Offline Scores
• `eval-ablation-attribution` — Run Ablation Studies to Attribute Lift to Specific Components
• `eval-ab-sample-size-mde` — Calculate A/B Sample Size from MDE Before Running
• `eval-cuped-variance-reduction` — Apply CUPED to Halve A/B Sample Size with Pre-Experiment Covariates
• `eval-regression-query-suite` — Maintain a Regression Query Suite for Silent Quality Drops

9. Diversity & Re-ranking (MEDIUM-HIGH)

• `div-mmr-rerank` — Apply MMR Rerank for Top-Window Diversity
• `div-max-per-host` — Cap Impressions Per Host with Max-Per-Group Constraint
• `div-category-diversity` — Diversify Categories Hierarchically in the Top Window
• `div-dpp-quality-diversity` — Use Determinantal Point Processes for Joint Quality and Diversity
• `div-window-penalty` — Apply Window-Based Diversity Penalty in Rescore

How to Use

For a focused question ("which decay function for geo distance?"), jump directly to the relevant rule (decay-gauss-geo) — each rule is self-contained with the WHY, OpenSearch query/Painless code, and the canonical source citation.

For a full ranking system review, work the categories top-to-bottom. The cascade ordering is real: get recall right first (no boost recovers a missed candidate), then base relevance (it's the multiplicand of every functional score), then quality / personalization / decay / marketplace balance / bias correction in that order. Diversity is the last re-rank step over a well-ordered top window.

For correcting bias before retraining, start with bias-position-ips and bias-click-models — applying IPS to position-confounded click data is the single highest-leverage change for any marketplace that retrains LTR models on logged clicks.

For testing multiple algorithms together and validating empirically, start with eval-graded-judgment-set (build the foundation), eval-ndcg-primary-metric (pick the metric), then eval-ablation-attribution (attribute lift to specific components). Pair with eval-online-offline-correlation to verify your offline metric predicts online behavior, eval-ab-sample-size-mde + eval-cuped-variance-reduction for disciplined A/B testing, and eval-regression-query-suite to catch silent quality drops on named queries.

For research-citing a design decision, every rule ends with the canonical reference — KDD/SIGIR/WSDM papers, the relevant engineering blog (Airbnb, Pinterest, DoorDash, Etsy, Just Eat Takeaway, Thumbtack), or the OpenSearch documentation page.

Read section definitions for the cascade-impact rationale behind the category ordering, or the rule template when adding a new rule.

Reference Files

OpenSearch Function Scoring for Two-Sided Marketplaces

Version 0.2.0 Marketplace-Research May 2026

Note: Agent/LLM-facing table of contents for the OpenSearch Function Scoring for Two-Sided Marketplaces rule set; entry point for AI agents maintaining, generating, refactoring, or evaluating OpenSearch / Elasticsearch ranking code for marketplace search. Humans may also find it useful, but guidance here is optimized for automation and consistency by AI-assisted workflows.

---

Abstract

Comprehensive guide to research-backed scoring algorithms AND empirical-evaluation methodology for two-sided marketplace search on OpenSearch or Elasticsearch. Contains 56 rules across 9 categories, ordered by cascade effect in the search ranking pipeline — from candidate retrieval (where a miss cannot be recovered downstream) through base relevance, quality signals, personalization, decay, marketplace balance, bias correction, evaluation & measurement, and diversity. Each rule explains the underlying mechanism, shows incorrect-vs-correct code (OpenSearch JSON, Painless, Python pre-processing, evaluation pipelines), and cites a canonical source — KDD/SIGIR/WSDM/CIKM papers (Airbnb, Pinterest, Google, Microsoft), the OpenSearch documentation, and the engineering blogs of marketplaces that proved these patterns at scale (Airbnb, DoorDash, Etsy, Pinterest, Thumbtack, Just Eat Takeaway, Booking, Netflix). The evaluation category explicitly covers how to test multiple algorithms together — judgment sets, NDCG, ablation studies, A/B sample-size planning, CUPED variance reduction, online-offline correlation calibration, and regression query suites. Suitable as the source of truth for AI agents implementing, reviewing, OR evaluating marketplace ranking code.

---

Table of Contents

1. Candidate Retrieval & Recall — CRITICAL

• 1.1 Apply Pre-Filter to kNN with Hard Constraints — CRITICAL (prevents empty result sets with strict filters)
• 1.2 Apply Synonym Expansion at Index Time for Recall, Query Time for Precision — MEDIUM-HIGH (prevents O(synonyms × terms) query blowup)
• 1.3 Choose HNSW for Latency, IVF for Memory at Scale — HIGH (3-10x memory savings with IVF beyond 100M vectors)
• 1.4 Split Retrieval into Cheap Recall and Expensive Re-rank — HIGH (10-100x cost reduction vs single-stage)
• 1.5 Use Hybrid BM25 + kNN with Reciprocal Rank Fusion — CRITICAL (8-15% NDCG@10 lift over either alone)
• 1.6 Use Two-Tower Architecture for Embedding-Based Retrieval — CRITICAL (enables sub-100ms recall at billion-item scale)

2. Base Relevance & Field Scoring — CRITICAL

• 2.1 Avoid Field-Boost Inflation Above ~10x — MEDIUM-HIGH (prevents single-field dominance collapse)
• 2.2 Pick multi_match Type by Query Shape, Not by Default — HIGH (10-25% relevance shift between types)
• 2.3 Prefer Listwise (LambdaMART) over Pairwise (RankNet) LTR Loss — HIGH (3-8% NDCG@10 gain on graded relevance sets)
• 2.4 Tune BM25 k1 and b Per-Field for Short Marketplace Documents — HIGH (5-15% NDCG@10 lift on title fields)
• 2.5 Tune BM25F Field Weights Before k1/b — CRITICAL (15-30% NDCG gain over single-field BM25)
• 2.6 Use rescore Phase for Heavy Scoring, Not bool/should at Retrieval — HIGH (5-20x latency reduction with same ranking quality)
• 2.7 Use script_score Query, Not function_score, for Composition — HIGH (2-5x faster, supports caching and modern features)

3. Quality Signals & Confidence Bounds — HIGH

• 3.1 Apply Sigmoid Modifier for Bounded Ratio Signals — MEDIUM-HIGH (prevents flat-line at high ratios)
• 3.2 Choose log1p over Saturation for Long-Tail Signal Preservation — MEDIUM-HIGH (preserves head-vs-super-head differentiation)
• 3.3 Saturate Popularity Counts with rank_feature.saturation — HIGH (prevents popularity-blowout where 10x reviews = 10x score)
• 3.4 Score Listing Completeness as a Quality Signal — MEDIUM (5-10% conversion lift on listings nudged to complete)
• 3.5 Sort by Wilson Lower Bound, Not Average Rating — HIGH (prevents 1-rating-of-5-stars beating 1000-ratings-of-4.8)
• 3.6 Use Bayesian Average for Star Ratings with Low Sample Sizes — HIGH (prevents new-listing cold-start rating distortion)

4. Personalization & Embeddings — HIGH

• 4.1 Apply Cross-Encoder Re-rank on Top-50 for Personalization — HIGH (5-10% NDCG@10 lift on top window)
• 4.2 Inject Contextual Features into script_score — MEDIUM-HIGH (2-5% conversion lift from device/time/location context)
• 4.3 Split Item Tower Offline, Query Tower Online — HIGH (enables sub-100ms query latency at billion-item scale)
• 4.4 Train Listing Embeddings from Booking-Session Co-occurrence — HIGH (21% NDCG@10 lift in Airbnb production (KDD 2018))
• 4.5 Update Session Vector in Real-Time from Click Events — HIGH (3-5% session conversion lift vs static user vector)
• 4.6 Use Multi-Modal Embeddings (Text + Image) for Recall — MEDIUM-HIGH (7-12% incremental recall over text-only embeddings)
• 4.7 Use Type Embeddings for Cold-Start Users and Listings — HIGH (lifts cold-start ranking quality 12-18% NDCG)

5. Spatial & Temporal Decay — HIGH

• 5.1 Add Offset to Decay Functions for Noisy Sparse Fields — MEDIUM (prevents micro-distance ranking instability)
• 5.2 Calibrate Decay Scale to the 0.5-Score Distance Target — MEDIUM-HIGH (prevents over- or under-penalty cliffs)
• 5.3 Compose Multi-Field Decay with Explicit Weights — MEDIUM-HIGH (prevents one decay dimension from dominating)
• 5.4 Use Exp Decay for Time Freshness, Gauss for Date Proximity — HIGH (prevents symmetric falloff on directional time)
• 5.5 Use Gauss Decay for Geo Distance, Not Linear — HIGH (prevents linear over-penalty within walking distance)

6. Two-Sided Marketplace Balance — HIGH

• 6.1 Boost Cold-Start Listings with Bounded Exposure Allocation — HIGH (enables supply growth without ranking instability)
• 6.2 Monitor Supply-Side Fairness with Lorenz/Gini Metrics — HIGH (prevents winner-take-all supply collapse)
• 6.3 Optimize Multi-Objective Ranking with Pareto-Aware Weights — HIGH (explicit Pareto frontier > implicit single objective)
• 6.4 Penalize Listings with Low Inventory Health — MEDIUM-HIGH (prevents user dead-ends on unavailable inventory)
• 6.5 Score Price Relevance with Soft Bands, Not Hard Filters — HIGH (prevents zero-result pages from tight budgets)
• 6.6 Separate Host-Quality and Listing-Quality Signals — MEDIUM-HIGH (prevents host-good-listing-bad confusion)
• 6.7 Weight Ranking by Conversion Rate, Not Click-Through Rate — HIGH (5-15% conversion lift vs CTR-only ranking)

7. Bias Correction & Online Learning — HIGH

• 7.1 Correct Position Bias with Inverse Propensity Scoring — HIGH (prevents 5-10× position-from-relevance confound)
• 7.2 Estimate Click Propensities with PBM, Cascade, or DBN — HIGH (enables IPS without randomization experiments)
• 7.3 Explore Ranking Alternatives with Thompson Sampling — HIGH (95% of greedy gain with proven exploration)
• 7.4 Subsample Popular Items in Embedding Training Negatives — MEDIUM-HIGH (prevents head-item embedding collapse)
• 7.5 Use Interleaved Evaluation for Low-Traffic Ranking Comparisons — MEDIUM-HIGH (10-100x more statistical power than A/B at low traffic)
• 7.6 Validate Ranking Changes with Counterfactual Evaluation — MEDIUM-HIGH (80% of A/B-test signal without exposing users)

8. Evaluation & Measurement — HIGH

• 8.1 Apply CUPED to Halve A/B Sample Size with Pre-Experiment Covariates — HIGH (40-60% variance reduction, 2x test throughput)
• 8.2 Build a Graded Judgment Set for Offline Evaluation — HIGH (enables all offline ranking metrics)
• 8.3 Calculate A/B Sample Size from MDE Before Running — HIGH (prevents 20-30% false positive rate from peeking)
• 8.4 Maintain a Regression Query Suite for Silent Quality Drops — MEDIUM (prevents tail/edge-case degradation while average is flat)
• 8.5 Run Ablation Studies to Attribute Lift to Specific Components — HIGH (prevents bundled-change blame attribution failure)
• 8.6 Use NDCG@k as the Primary Offline Ranking Metric — HIGH (prevents metric-mismatch with multi-grade relevance)
• 8.7 Validate Online-Offline Metric Correlation Before Trusting Offline Scores — HIGH (prevents shipping rankers that improve NDCG but hurt conversion)

9. Diversity & Re-ranking — MEDIUM-HIGH

• 9.1 Apply MMR Rerank for Top-Window Diversity — MEDIUM-HIGH (3-7% session-level engagement lift)
• 9.2 Apply Window-Based Diversity Penalty in Rescore — MEDIUM (preserves rank stability across sessions)
• 9.3 Cap Impressions Per Host with Max-Per-Group Constraint — MEDIUM-HIGH (prevents single-host page domination)
• 9.4 Diversify Categories Hierarchically in the Top Window — MEDIUM-HIGH (4-8% category-coverage lift in top-10)
• 9.5 Use Determinantal Point Processes for Joint Quality and Diversity — MEDIUM (1-3% engagement lift over MMR on high-stakes pages)

---

References

1. https://docs.opensearch.org/latest/query-dsl/compound/function-score/ 2. https://docs.opensearch.org/latest/query-dsl/specialized/script-score/ 3. https://docs.opensearch.org/latest/query-dsl/specialized/rank-feature/ 4. https://docs.opensearch.org/latest/search-plugins/knn/ 5. https://docs.opensearch.org/latest/search-plugins/ltr/ 6. https://docs.opensearch.org/latest/query-dsl/full-text/combined-fields/ 7. https://docs.opensearch.org/latest/vector-search/specialized-operations/vector-search-mmr/ 8. https://opensearch.org/blog/introducing-reciprocal-rank-fusion-hybrid-search/ 9. https://dl.acm.org/doi/10.1145/3219819.3219885 10. https://arxiv.org/pdf/2601.06873 11. https://arxiv.org/pdf/2210.07774 12. https://airbnb.tech/uncategorized/embedding-based-retrieval-for-airbnb-search/ 13. https://airbnb.tech/infrastructure/academic-publications-airbnb-tech-2025-year-in-review/ 14. https://medium.com/airbnb-engineering/listing-embeddings-for-similar-listing-recommendations-and-real-time-personalization-in-search-601172f7603e 15. https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf 16. https://www.cs.cmu.edu/~jgc/publication/The_Use_MMR_Diversity_Based_LTMIR_1998.pdf 17. https://www.cs.cornell.edu/people/tj/publications/joachims_etal_17a.pdf 18. https://research.google/pubs/pub46485/ 19. https://arxiv.org/abs/1802.07281 20. https://www.evanmiller.org/how-not-to-sort-by-average-rating.html 21. https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf 22. https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/ 23. https://arxiv.org/abs/1207.6083 24. https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf 25. https://arxiv.org/html/2404.16260v1 26. https://medium.com/pinterest-engineering/pinnersage-multi-modal-user-embedding-framework-for-recommendations-at-pinterest-bfd116b49475 27. https://medium.com/pinterest-engineering/pinsage-a-new-graph-convolutional-neural-network-for-web-scale-recommender-systems-88795a107f48 28. https://careersatdoordash.com/blog/doordash-kdd-llm-assisted-personalization-framework/ 29. https://arxiv.org/pdf/2402.02626 30. https://arxiv.org/pdf/2206.11720 31. https://medium.com/justeattakeaway-tech/inverse-propensity-score-based-offline-estimator-for-deterministic-ranking-lists-using-position-89ce866c27dd 32. https://www.elastic.co/blog/practical-bm25-part-3-considerations-for-picking-b-and-k1-in-elasticsearch 33. https://openreview.net/pdf?id=uPWdkoZHgba 34. https://www.cs.cornell.edu/people/tj/publications/radlinski_etal_08a.pdf 35. https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality 36. https://www.jstor.org/stable/2276774 37. https://dl.acm.org/doi/10.1145/582415.582418 38. https://trec.nist.gov/pubs/trec16/appendices/measures.pdf 39. https://www.evidentlyai.com/ranking-metrics/ndcg-metric 40. https://www.shaped.ai/blog/ndcg-evaluating-ranking-quality-with-graded-relevance 41. https://exp-platform.com/Documents/2013-02-OnlineControlledExperimentsAtLargeScale.pdf 42. https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf 43. https://docs.growthbook.io/statistics/cuped 44. https://experimentguide.com/ 45. https://docs.geteppo.com/statistics/sample-size-calculator/mde/ 46. https://en.wikipedia.org/wiki/Ablation_(artificial_intelligence)) 47. https://capitalone.com/tech/machine-learning/xai-ablation-study

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

Rule Title in Imperative Mood

Brief explanation (1-3 sentences) of WHY this matters in a marketplace ranking context. Focus on the cascade effect — what goes wrong downstream when this rule is violated, and which property (recall, base relevance, fairness, bias) is being protected. Cite the underlying mechanism or paper at the level of intuition; details go below.

Incorrect (concrete description of the wrong pattern):

{
  "query": {
    "bad_example_query": "..."
  }
}

Brief one-line annotation about what's wrong above (e.g., "applies popularity boost to all 200k matches").

Correct (concrete description of the right pattern):

{
  "query": {
    "good_example_query": "..."
  }
}

Brief one-line annotation about why this works (e.g., "rescore phase applies popularity only to top-500").

Optional sections (include when applicable):

Why this matters at marketplace scale: Deeper explanation tying the rule to two-sided dynamics, scale economics, or training-data hygiene.

Calibration / Tuning: Table or recipe for picking parameters.

When NOT to use this pattern: Important exceptions (e.g., "don't apply MMR diversity to specific-intent queries").

Warning (gotcha): A subtle failure mode worth calling out.

Reference: Primary source title · Secondary source title

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

Categories appear in impact order (CRITICAL → MEDIUM-HIGH), but the search ranking pipeline runs in a different order: Query → Recall → Base Relevance → Quality → Personalization → Decay → Marketplace Balance → Diversity → (Bias Correction and Evaluation are meta concerns applied across all stages). A miss at recall cannot be recovered downstream; a wrong base relevance multiplies every functional score; uncorrected bias poisons every future model trained on click logs; and without evaluation infrastructure you cannot tell whether any of the other rules helped.

---

1. Candidate Retrieval & Recall (recall)

Impact: CRITICAL Description: Recall gates everything. A listing missing from the candidate set cannot be ranked, no matter how clever the scoring. Hybrid retrieval (BM25 + kNN with RRF), two-tower embedding-based retrieval, and ANN index selection (HNSW vs IVF) belong here.

2. Base Relevance & Field Scoring (rel)

Impact: CRITICAL Description: The base relevance score is the multiplicand for every downstream function. BM25F field weighting, combined_fields, multi_match strategy selection, and LTR loss-function choice (pairwise vs listwise) determine whether every later boost amplifies signal or noise.

3. Quality Signals & Confidence Bounds (qual)

Impact: HIGH Description: Item-intrinsic quality signals computable from the listing alone — ratings, review counts, photo quality, completeness scores. Raw averages and unsmoothed counts are systematically biased toward low-sample items. Wilson Lower Bound, Bayesian average, rank_feature saturation/sigmoid, and log1p normalization fix this.

4. Personalization & Embeddings (pers)

Impact: HIGH Description: Per-user differentiation via listing/user embeddings, two-tower architectures, real-time session vectors, and cross-encoder re-ranking. The offline/online tower split (precompute item embeddings, score query tower at request time) is what makes embedding-based retrieval feasible at marketplace scale.

5. Spatial & Temporal Decay (decay)

Impact: HIGH Description: Geographic and time-based relevance via Gauss, exp, and linear decay functions. Origin, scale, decay, and offset tuning determines whether "near" means 1km or 100km, and whether "fresh" means hours or weeks. Multi-field decay composition (geo × date × seasonality) is standard for accommodation/delivery marketplaces.

6. Two-Sided Marketplace Balance (market)

Impact: HIGH Description: Signals that exist only because of two-sided dynamics — conversion rate, host acceptance rate, cancellation rate, supply scarcity, cold-start exposure, inventory health, and multi-objective Pareto balance. This is what distinguishes marketplace ranking from general web search; getting it wrong starves supply or burns demand.

7. Bias Correction & Online Learning (bias)

Impact: HIGH Description: Position bias, popularity bias, and cold-start bias all enter through implicit feedback (clicks, bookings). Training on raw click logs without Inverse Propensity Scoring (Joachims et al. 2017) compounds these biases on every retrain. Click models (PBM/cascade/DBN), counterfactual evaluation, and bounded exploration (Thompson Sampling, epsilon-greedy) are required for sustainable online learning.

8. Evaluation & Measurement (eval)

Impact: HIGH Description: Infrastructure for empirically validating that ranking changes actually work. Graded judgment sets, NDCG@k as the primary offline metric, online-offline correlation checks, ablation studies for component attribution, A/B test sample-size planning (MDE × power × variance), CUPED variance reduction, and regression query suites. Without measurement infrastructure you cannot tell whether any of the 49 algorithmic rules in the other categories actually helped — and you can't safely test combinations.

9. Diversity & Re-ranking (div)

Impact: MEDIUM-HIGH Description: Post-rank reordering to prevent homogeneous result lists. MMR (Carbonell & Goldstein 1998), Determinantal Point Processes, max-per-host constraints, and hierarchical category diversity. Lower impact than upstream stages because it operates on an already-ranked set, but materially affects engagement on the top window.

Estimate Click Propensities with PBM, Cascade, or DBN

IPS requires per-position propensities — the probability a user examined position i. Click models infer these from observed click patterns without needing the (costly) randomization experiments described in bias-position-ips. Three click models cover most marketplace scenarios: PBM (Position-Based Model — clicks independent across positions); Cascade — user scans top-to-bottom and stops at first click; DBN (Dynamic Bayesian Network) — adds the satisfaction event (user stops after click if satisfied). Pick by your traffic pattern.

Incorrect (assume uniform propensity across positions — naive average):

# Naive — treat every position equally
position_propensity = {i: 1.0 for i in range(1, 51)}

This is what no bias correction looks like.

Correct (PBM with EM — start here for most marketplaces):

# Position-Based Model (PBM)
#   Assumption: Click iff (Examined at position) AND (Item is Relevant)
#   P(click | q, item, pos) = P(examined | pos) × P(relevant | q, item)
#   EM:
#     E-step: attribute clicks to examination vs relevance given current params
#     M-step: re-estimate examination prior per position

from pyclick.click_models.PBM import PBM

sessions = parse_marketplace_sessions(click_log)  # list of [(query, item, position, clicked)]
pbm = PBM()
pbm.train(sessions)

# Export propensity table for IPS training
position_propensity = {
    p: pbm.params[PBM.param_names.exam][p - 1]
    for p in range(1, 51)
}

Alternative — Cascade Model:

Assumption: User scans top-to-bottom, clicks if relevant, stops at first click
  P(click_i = 1) = relevance_i × Π_{j<i} (1 - relevance_j)

Use when: users click at most once per session (specific transactional searches). Less suited to browse-heavy marketplaces.

Alternative — DBN (Dynamic Bayesian Network):

Adds a Satisfaction event after click:
  P(continue | clicked) = 1 - σ
  σ is per-item — captures how often a click leads to a "good" outcome

Use when: you have both clicks AND conversion signal — DBN models both jointly. Highest-capacity option for marketplace data where you want propensities and per-item satisfaction together.

Validation: Hold out a slice of randomized-exposure traffic (1-2% of queries with shuffled top-K) and compare PBM-derived propensities against the empirical position-CTR from the randomized slice. They should track within ~10%.

When PBM fails: If your marketplace has strong rank-dependent layout (e.g., images get larger at the top), PBM under-estimates position-1 propensity. Use a multi-modal model (incorporates layout features) or DBN with attention weights as a per-position covariate.

Reference: Chuklin, Markov, de Rijke — Click Models for Web Search (book, 2015) · pyClick library · Wang et al. — Position Bias Estimation (WSDM 2018)

Validate Ranking Changes with Counterfactual Evaluation

Every A/B test costs days of wall-clock, statistical-power budget, and risks user exposure to a bad ranker. Counterfactual (off-policy) evaluation answers "how would a new policy have performed on past logged data?" without running the new policy live. Using IPS as the off-policy estimator, you can pre-screen 5-10 candidate rankers offline, ship only the top 1-2 to A/B test. Airbnb (KDD 2025 "Harnessing the Power of Interleaving and Counterfactual Evaluation") reports this saves ~80% of A/B-test calendar time.

Incorrect (every ranking change ships to A/B test — slow, risky):

# Direct path: idea → ship to 5% A/B → wait 14 days → analyze
new_ranker = train_v2()
ab_test.launch(treatment=new_ranker, control=current_ranker, traffic_pct=5)
# 14 days later: lift signal often within noise band — wasted cycle if it was a bad idea

Correct (counterfactual filter before A/B test):

# Step 1: Estimate counterfactual reward of new_ranker on logged data
def ips_estimator(new_ranker, logged_data, propensity_table):
    """
    logged_data: list of (query, ranking_shown, item_clicked, position_clicked,
                          logging_policy_score)
    new_ranker: function (query, candidates) -> ranking
    """
    total_reward = 0.0
    total_weight = 0.0
    for record in logged_data:
        new_ranking = new_ranker(record.query, record.candidates)
        new_position = position_of(record.item_clicked, new_ranking)
        if new_position is None or new_position > 50:
            continue  # item not in new policy's top-K

        # IPS weight: prob under new policy / prob under logging policy
        p_new = soft_position_prob(new_position)
        p_log = propensity_table.get(record.position_clicked, 0.05)
        weight = min(p_new / p_log, 50.0)  # clip to avoid huge weights

        total_reward += weight * record.reward  # 1 for conversion, 0 otherwise
        total_weight += weight

    return total_reward / max(total_weight, 1.0)  # estimated reward per impression

# Step 2: Rank candidate rankers by counterfactual estimate
candidates_to_test = [ranker_v2_a, ranker_v2_b, ranker_v2_c, ranker_v2_d, ranker_v2_e]
estimates = [(r, ips_estimator(r, logged_data, propensities)) for r in candidates_to_test]

# Step 3: A/B only the top 1-2
top_two = sorted(estimates, key=lambda x: -x[1])[:2]
for ranker, est in top_two:
    print(f"Counterfactual conversion estimate: {est:.4f}")
    ab_test.launch(treatment=ranker, ...)

Variance reduction with Self-Normalized IPS (SNIPS):

Raw IPS has high variance when propensities are extreme. SNIPS normalizes by the sum of weights:

SNIPS = (Σ w_i × reward_i) / (Σ w_i)

Use SNIPS instead of raw IPS in production — substantially lower variance, slightly biased but accept the trade-off.

Doubly-Robust (DR) estimator — even lower variance:

Train a separate per-item reward predictor r̂(query, item). DR estimator:

DR = E_new[r̂] + (1/N) Σ (1/p_log_i) × (r_i - r̂_i) × 1[item shown by new policy]

If r̂ is decent, the residual is small and variance drops further. Standard pattern for marketplace counterfactual eval.

Calibration check before trusting offline estimates:

Once a quarter, run a deliberate A/B and compare its result to your counterfactual estimate for the same change. If the offline estimate and the online result diverge by >30%, your propensity model is wrong — fix that before trusting more offline estimates.

The marketplace habit: Treat counterfactual evaluation as a triage step, not a substitute for A/B. Goal: of every 10 ideas, kill 8 offline, A/B test the top 2. Saves engineering and user-exposure budget enormously.

Reference: Joachims, Swaminathan, Schnabel — Unbiased Learning-to-Rank with Biased Feedback (WSDM 2017) · Airbnb — Harnessing Interleaving and Counterfactual Evaluation (KDD 2025) · Just Eat Takeaway — IPS Offline Estimator

Use Interleaved Evaluation for Low-Traffic Ranking Comparisons

Classic A/B test at 1% traffic on a low-volume query (10 queries/day) needs months to reach statistical significance. Interleaving — show users a merged ranking from both rankers, attribute clicks to whichever ranker contributed each item — is 10-100× higher statistical efficiency per impression because every user contributes to both arms simultaneously. Team-Draft Interleaving (Radlinski et al., CIKM 2008) is the canonical implementation and what Airbnb uses for ranking experimentation on low-traffic verticals.

Incorrect (classic A/B on low-traffic vertical — months to statistical power):

# 1% traffic on a vertical that gets 50 queries/day = 0.5 queries/day in treatment
# Need ~1000 conversions to detect 5% effect → months of waiting
ab_test.launch(treatment=ranker_v2, control=ranker_v1, traffic_pct=1)

Correct (Team-Draft Interleaving — every query contributes to both arms):

import random

def team_draft_interleave(ranking_a, ranking_b, k=10):
    """
    Build a merged top-K ranking where each slot is drawn from one of the two rankers,
    alternating who picks first (the "team captain" coin flip).
    """
    merged = []
    used = set()
    pool_a = list(ranking_a)
    pool_b = list(ranking_b)
    a_picks = []
    b_picks = []
    a_first = random.random() < 0.5  # coin flip for first pick

    while len(merged) < k and (pool_a or pool_b):
        if (len(a_picks) <= len(b_picks)) if a_first else (len(a_picks) < len(b_picks)):
            picker = "A"; pool = pool_a; picks = a_picks
        else:
            picker = "B"; pool = pool_b; picks = b_picks
        # Pop next unused from the picker's pool
        while pool and pool[0].id in used:
            pool.pop(0)
        if not pool:
            continue
        item = pool.pop(0)
        merged.append((item, picker))
        used.add(item.id)
        picks.append(item.id)

    return merged  # [(item, attributed_to), ...]


# Online: serve merged ranking, log which "team" each clicked item belongs to
ranking_a = ranker_v1.rank(query, candidates)
ranking_b = ranker_v2.rank(query, candidates)
merged = team_draft_interleave(ranking_a, ranking_b, k=10)
serve_to_user(merged)

# After clicks logged: tally credit per ranker
def credit_clicks(impression_log):
    a_wins = b_wins = 0
    for impression in impression_log:
        a_clicks = sum(1 for click in impression.clicks if click.attribution == "A")
        b_clicks = sum(1 for click in impression.clicks if click.attribution == "B")
        if a_clicks > b_clicks: a_wins += 1
        elif b_clicks > a_clicks: b_wins += 1
        # Ties (including zero clicks) don't count
    return a_wins, b_wins

# Binomial test for significance — much higher power than A/B

Why interleaving is so much more efficient:

A/B test: each impression contributes to ONE arm (Var = pq/N per arm)
Interleaving: each impression contributes to comparison directly (paired observation)
              variance is dominated by within-impression noise, not between-user noise

Result: ~10-100x sample-size efficiency

Trade-offs:

Use interleaving when: comparing two rankers head-to-head, low-traffic vertical, need fast iteration. Use A/B when: measuring absolute lift on long-term metrics, learning effects (habit formation), business KPIs.

Combine with counterfactual evaluation: Counterfactual triage offline → interleaving live for fast preference signal → A/B confirmation on top variant. That's the full Airbnb playbook.

Reference: Radlinski, Kurup, Joachims — Team-Draft Interleaving (CIKM 2008) · Chapelle et al. — Large-scale validation and analysis of interleaved search evaluation (TIST 2012) · Airbnb — Interleaving + Counterfactual (KDD 2025)

Subsample Popular Items in Embedding Training Negatives

Embedding models trained with naive uniform negative sampling treat every item as equally likely to be the "negative" — but popular items get sampled as negatives often, pushing them away from queries even when they're relevant. The result: head items collapse to mediocre embeddings; long-tail items get pulled around by noise. Word2Vec's authors (Mikolov 2013) noted this and proposed subsampling frequent words; the equivalent fix for marketplace item embeddings is sampling negatives proportional to f^(3/4) (or similar sub-linear scaling) and explicitly excluding the in-batch positives.

Incorrect (uniform negative sampling — head items collapse):

def get_negatives(positive_item_id, k=5):
    """Uniform — every item equally likely to be a negative"""
    return random.sample(all_item_ids - {positive_item_id}, k)

A popular item gets sampled as a negative ~10× more often than it should, because it co-occurs with 10× more queries. Its embedding drifts away from many queries even when it's actually relevant to them.

Correct (frequency-adjusted negative sampling, à la Word2Vec):

import numpy as np

# Pre-compute sampling probabilities ∝ frequency^(3/4)
item_freq = compute_item_frequencies(training_data)  # {item_id: count}
freq_arr = np.array([item_freq[i] for i in item_ids])
sample_prob = freq_arr ** 0.75
sample_prob /= sample_prob.sum()

def get_negatives_adjusted(positive_item_id, k=5):
    """Sample ∝ freq^0.75 — heavy items down-weighted from raw frequency"""
    return np.random.choice(item_ids, size=k, replace=False, p=sample_prob).tolist()

The 0.75 exponent comes from Mikolov et al.'s Word2Vec — empirically the best trade-off between sampling rare items often enough and not over-sampling head items.

Equivalent: importance-weighted contrastive loss:

If you can't change the sampler, weight the contrastive loss by 1 / freq^0.75 for each negative:

def contrastive_loss(query_vec, pos_vec, neg_vecs, neg_freqs):
    pos_score = query_vec @ pos_vec
    neg_scores = query_vec @ neg_vecs.T  # shape (k,)

    # Down-weight popular-item negatives
    weights = 1.0 / (neg_freqs ** 0.75)
    weights /= weights.sum()  # normalize

    loss = -torch.log_softmax(
        torch.cat([pos_score.unsqueeze(0), weights * neg_scores]),
        dim=0
    )[0]
    return loss

Validation: After training, plot per-item average cosine similarity to a random query batch. Without debiasing, head items show systematically lower similarity (over-pushed-away). With debiasing, the distribution flattens.

The deeper bias: Popularity bias affects more than negative sampling. It shows up in: 1. Click logs — head items get clicked more from exposure, not relevance (see bias-position-ips). 2. Conversion rates — head items convert more from familiarity, not match quality. 3. User similarity — users who click head items look "similar" to each other because head items are everywhere.

Apply IPS for click bias (bias-position-ips); apply frequency-adjusted sampling for embedding bias (this rule). They're complementary, not redundant.

For marketplaces with long-tail strategy: Down-weighting popularity in embeddings is a prerequisite for surfacing long-tail inventory. Without it, your retrieval system can't physically generate long-tail candidates because their embeddings are too far from query embeddings.

Reference: Mikolov et al. — Distributed Representations of Words and Phrases (NIPS 2013) · Recsys Popularity Bias survey (RecSys 2021)

Correct Position Bias with Inverse Propensity Scoring

A click on the #1 result doesn't mean that result is better — it means the user saw it first. Position bias is the largest systematic bias in implicit feedback: by some estimates, the #1 position receives 5-10× the clicks of position #5 purely from position, independent of relevance. Training an LTR model on raw click data without correction reinforces whatever was previously ranked at the top, creating a feedback loop that drifts away from true relevance. Inverse Propensity Scoring (Joachims et al., WSDM 2017) corrects this by weighting each observed click by 1 / propensity, where propensity is the probability the user examined that position.

The IPS framework:

Raw training signal:        L_naive = Σ click_i × loss(rank_i)

IPS-corrected signal:       L_ips   = Σ (click_i / p_i) × loss(rank_i)

  where p_i = P(examined | shown at position i)

Items at position 1 have p₁ ≈ 1; items at position 10 have p₁₀ ≈ 0.2. Dividing the click signal by propensity inflates the importance of clicks that occurred despite low examination probability, removing the position-from-relevance confound.

Incorrect (training LTR on raw click logs — bias compounds):

# Raw click as training label — top-of-page gets 5-10x weight from position alone
training_examples = [
    (query, listing, listing.shown_position, listing.was_clicked)
    for impression in click_log
]

# This trains "what was at the top got clicked," not "what was relevant got clicked"
ltr.fit(training_examples)

Correct (IPS-weighted training):

# 1. Estimate propensity p_i — e.g., from a position-bias model (see bias-click-models)
position_propensity = {1: 1.00, 2: 0.65, 3: 0.45, 4: 0.32, 5: 0.24,
                       6: 0.19, 7: 0.16, 8: 0.14, 9: 0.12, 10: 0.10}

# 2. Re-weight training examples by 1 / propensity
training_examples = []
for impression in click_log:
    p = position_propensity[impression.shown_position]
    if impression.was_clicked:
        weight = 1.0 / max(p, 0.05)  # clip propensity to avoid huge weights
        training_examples.append((impression.query, impression.listing, weight, label=1))

ltr.fit(training_examples)  # weighted loss

Propensity-clipping: Without a floor on p, a click at position 50 (p ≈ 0.02) gets weight 50× — a single click dominates the gradient. Clip propensities at max(p, 0.05) or use Self-Normalized IPS.

Cold-start the propensity model: Initially, you have no propensities. Two options: 1. Randomized exposure (RandPair): For 1-5% of queries, shuffle positions; observe clicks; fit a position-bias model. Joachims et al. recommend this when you can tolerate the UX hit. 2. Result-randomization-free estimation: EM-based estimation à la Wang et al. (WSDM 2018) — fit the propensity and the relevance model jointly.

Validating IPS is working: Track offline NDCG against a held-out judged set before and after IPS. If NDCG goes down, your propensity estimates are wrong; check for clipping, missing positions in propensity table, or a popularity-bias confound (see bias-popularity-debiasing).

Why this isn't optional for marketplaces: Position bias in marketplace click logs is documented to drift models monotonically toward popular-incumbent items (Thumbtack 2024, Just Eat Takeaway 2023). Uncorrected, your model trains itself into a self-reinforcing loop where the top of yesterday's results trains today's model to keep them at the top.

Reference: Joachims et al. — Unbiased Learning-to-Rank with Biased Feedback (WSDM 2017) · Wang et al. — Position Bias Estimation (WSDM 2018) · Thumbtack — Position Bias in Features (arXiv 2402.02626)

Explore Ranking Alternatives with Thompson Sampling

A purely greedy ranker (always show the highest-predicted-relevance result) never learns whether other items might convert better — it just exploits its current estimate. Thompson Sampling provides Bayes-optimal exploration: maintain a posterior distribution over each item's relevance, sample from each posterior at scoring time, and rank by sampled values. Items with tight posteriors (lots of data) get nearly-deterministic scores; items with wide posteriors (little data) get randomized scores that occasionally win, gathering data. Asymptotically optimal regret, simpler than UCB, and trivially parallelizable.

Incorrect (pure greedy — never explores; never learns about under-served items):

# Always show by current point estimate of conversion rate
def rank(candidates):
    return sorted(candidates, key=lambda c: c.estimated_conv_rate, reverse=True)

Correct (Thompson Sampling — sample from Beta posterior, rank by sample):

import numpy as np

def thompson_rank(candidates):
    # Each candidate has alpha = bookings + 1, beta = (impressions - bookings) + 1
    # (Beta prior with one pseudo-success and one pseudo-failure)
    sampled_scores = []
    for c in candidates:
        alpha = c.bookings_30d + 1
        beta_param = (c.impressions_30d - c.bookings_30d) + 1
        sample = np.random.beta(alpha, beta_param)
        sampled_scores.append((sample, c))

    return [c for _, c in sorted(sampled_scores, key=lambda x: -x[0])]

Apply in OpenSearch via `script_score` with a randomization seed:

OpenSearch doesn't natively have Beta sampling in Painless, so push the sampled scores via a pre-query step:

# At request time:
sampled_scores = {c.id: float(np.random.beta(c.bookings+1, c.impressions-c.bookings+1))
                  for c in candidates}

opensearch_query = {
    "query": {
        "function_score": {
            "query": {"match": {"city": query.city}},
            "functions": [{
                "script_score": {
                    "script": {
                        "source": "params.ts_scores.containsKey(doc['_id'].value) ? params.ts_scores[doc['_id'].value] : 0.5",
                        "params": {"ts_scores": sampled_scores}
                    }
                }
            }],
            "boost_mode": "multiply"
        }
    }
}

Why Beta is the right posterior for conversion: Conversion is a Bernoulli trial (booked / not booked) with a Beta prior — conjugate, so the posterior update is just α += bookings; β += non-bookings. Closed-form, computationally trivial.

Discounted Thompson Sampling for non-stationary marketplaces:

# Apply exponential decay to old observations — keeps posterior current as preferences shift
def update_with_decay(c, click_event, gamma=0.999):
    c.alpha = gamma * c.alpha + (1 if click_event.booked else 0)
    c.beta_param = gamma * c.beta_param + (0 if click_event.booked else 1)

gamma=0.999 per day gives effective horizon of ~1000 days; gamma=0.95 gives ~20 days (responsive to fast-changing demand).

Calibrate exploration vs exploitation via prior strength:

Stronger prior → less exploration on new items (they don't immediately get sampled high). Use the category mean as the prior mean, prior strength = your shrinkage parameter m.

Don't apply Thompson Sampling at the head of every query: Apply only to the exploration slot (5-10% of impressions) — see market-cold-start-exploration. Doing it on every result randomizes the user experience to an unacceptable degree.

Reference: Russo, Van Roy et al. — A Tutorial on Thompson Sampling (Foundations and Trends 2018) · Chapelle & Li — An Empirical Evaluation of Thompson Sampling (NIPS 2011)

Use Exp Decay for Time Freshness, Gauss for Date Proximity

Time-based decay has two distinct shapes depending on the question. Exp decay (exp(-λ × age)) is "older is monotonically worse, fast at first then slowly" — right for freshness (new listings, recent updates, recent reviews). Gauss decay around a date origin is "match this date, with a tolerance window" — right for event-date or availability matching. Mixing them up gives bizarre results: exp around an event date treats "1 day before" the same as "1 day after"; gauss for freshness treats "yesterday" same as "tomorrow."

Incorrect (gauss decay on listing age — symmetric falloff doesn't model freshness):

{
  "query": {
    "function_score": {
      "query": { "match_all": {} },
      "gauss": {
        "listed_at": {
          "origin": "now",
          "scale": "14d",
          "decay": 0.5
        }
      }
    }
  }
}

Gauss is symmetric around origin — listed_at in the future (impossible for real data, but possible with clock skew or test data) scores same as past. More importantly, it suggests "30 days old" is much worse than "14 days old," which over-penalizes anything older than your window.

Correct (exp decay for freshness — monotonic, smooth long-tail):

{
  "query": {
    "function_score": {
      "query": { "match_all": {} },
      "exp": {
        "listed_at": {
          "origin": "now",
          "offset": "3d",
          "scale": "30d",
          "decay": 0.5
        }
      },
      "boost_mode": "multiply"
    }
  }
}

So: brand-new listings within 3 days score 1.0, 33-day-old listings score 0.5, year-old listings score ~0.02 — gradual, monotonic.

Correct (gauss decay for date-proximity matching — symmetric tolerance):

{
  "query": {
    "function_score": {
      "query": { "match_all": {} },
      "gauss": {
        "event_date": {
          "origin": "2026-08-15",
          "scale": "7d",
          "decay": 0.5
        }
      }
    }
  }
}

An event on 2026-08-08 or 2026-08-22 both score 0.5 — symmetric around the user's target date, which is what you want for "find an event around this date."

Decision table for time-based decay:

Combining freshness with text relevance: Use boost_mode: multiply so freshness modulates relevance rather than overwhelming it. With boost_mode: sum, very old listings can ride to the top on text relevance alone.

Anti-pattern — using `now`/`d` units as filter: Don't use decay as a substitute for filtering out stale records. If anything older than 90 days is irrelevant, filter it out with range before scoring; let decay shape ranking within the relevant window.

Reference: OpenSearch decay functions

Use Gauss Decay for Geo Distance, Not Linear

Linear distance decay penalizes distance proportionally — a place 2km away scores half as much as one at 1km, and 4km scores zero. That's wrong for how users perceive distance: there's a "close enough" plateau where everything within the user's mental radius feels equivalent, then a sharp falloff. Gaussian decay models this directly: bell-shaped, plateau near origin, steep falloff after the scale distance. It's the default geo-decay function for Airbnb-style accommodation search, Uber Eats, DoorDash, Yelp.

Incorrect (linear decay — 2km is twice as bad as 1km):

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "coffee" } },
      "linear": {
        "location": {
          "origin": { "lat": 38.71, "lon": -9.13 },
          "scale": "5km",
          "decay": 0.5
        }
      }
    }
  }
}

Correct (gauss decay — plateau near origin, falloff after scale):

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "coffee" } },
      "gauss": {
        "location": {
          "origin": { "lat": 38.71, "lon": -9.13 },
          "offset": "500m",
          "scale": "2km",
          "decay": 0.5
        }
      }
    }
  }
}

Parameter semantics:

origin: the user's location
offset: distance at which decay starts (plateau within offset)
scale:  distance beyond offset where decay reaches `decay` value
decay:  score at (origin + offset + scale) — typically 0.5

So with offset: 500m, scale: 2km, decay: 0.5:

• 0-500m → score = 1.0 (full)
• 2.5km → score = 0.5
• 5km → score ≈ 0.06 (near zero)

Calibrating by domain:

Compose with text relevance via `multiply`:

{
  "query": {
    "function_score": {
      "query": { "match": { "name": "coffee" } },
      "gauss": {
        "location": { "origin": "38.71,-9.13", "offset": "200m", "scale": "1km", "decay": 0.5 }
      },
      "boost_mode": "multiply"
    }
  }
}

multiply means the geo signal modulates the text relevance — a far-away perfect-match still loses to a near-by good-match.

Why not Euclidean / Haversine directly: The raw distance is unbounded and not interpretable as a relevance multiplier. The decay functions map distance into [0,1] with a domain-meaningful shape.

Reference: OpenSearch decay functions · Elastic — Decay functions for relevance

Compose Multi-Field Decay with Explicit Weights

Marketplace queries are multi-dimensional: an Airbnb search has a location AND dates; a DoorDash search has a location AND time-of-day demand. Each dimension needs its own decay function. Default composition (multiply) silently lets one dimension dominate — a far-but-perfect-date listing gets crushed by distance, a near-but-wrong-date one survives. Explicit weights let you express "distance and date are equally important," "distance matters 2× more than date," etc.

Incorrect (multiple decays without explicit weights — one signal dominates):

{
  "query": {
    "function_score": {
      "query": { "match": { "type": "apartment" } },
      "functions": [
        {
          "gauss": {
            "location": { "origin": "38.71,-9.13", "scale": "5km", "decay": 0.5 }
          }
        },
        {
          "gauss": {
            "available_until": { "origin": "2026-08-15", "scale": "7d", "decay": 0.5 }
          }
        }
      ],
      "score_mode": "multiply",
      "boost_mode": "multiply"
    }
  }
}

A perfectly available listing 50km away: geo ≈ 0.0001, date = 1.0; product = 0.0001. Effectively excluded. A nearby listing available 60d off: geo = 0.99, date ≈ 0.001; product = 0.001. Also excluded.

Both should be in results with different rankings, but multiply zeros them.

Correct (`weighted_sum` of geometric components — weights explicit):

{
  "query": {
    "function_score": {
      "query": { "match": { "type": "apartment" } },
      "functions": [
        {
          "weight": 0.6,
          "gauss": {
            "location": { "origin": "38.71,-9.13", "offset": "1km", "scale": "4km", "decay": 0.5 }
          }
        },
        {
          "weight": 0.4,
          "gauss": {
            "available_until": { "origin": "2026-08-15", "offset": "1d", "scale": "6d", "decay": 0.5 }
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

score_mode: sum with weights gives a convex combination of decays — neither dimension can zero the other out, and the weights are interpretable as "60% distance, 40% date."

Or use `script_score` for explicit Painless formula:

{
  "query": {
    "script_score": {
      "query": { "match": { "type": "apartment" } },
      "script": {
        "source": """
          double distKm = doc['location'].arcDistance(params.lat, params.lon) / 1000;
          double distScore = Math.exp( -0.5 * Math.pow(Math.max(0, distKm - 1.0) / 4.0, 2) );

          long dayDiff = Math.abs(ChronoUnit.DAYS.between(
              doc['available_until'].value.toInstant(),
              ZonedDateTime.parse(params.target_date).toInstant()
          ));
          double dateScore = Math.exp( -0.5 * Math.pow(Math.max(0, dayDiff - 1) / 6.0, 2) );

          return _score * (0.6 * distScore + 0.4 * dateScore);
        """,
        "params": { "lat": 38.71, "lon": -9.13, "target_date": "2026-08-15T00:00:00Z" }
      }
    }
  }
}

Calibrating weights: Weights should sum to 1.0 for interpretability. Tune them on a graded judgment set or via online A/B testing — no closed-form solution exists. Starting points by query archetype: city break = 70% date / 30% location; specific neighborhood = 60% location / 40% date; flexible-date getaway = 80% location / 20% date.

When `multiply` is right anyway: If both dimensions are hard requirements (must be near AND must be available), then multiply correctly zeros out failures. Use it deliberately, not by default.

Reference: OpenSearch function_score score_mode · Painless date helpers

Add Offset to Decay Functions for Noisy Sparse Fields

Without offset, the decay function starts decaying immediately from origin, making it sensitive to micro-differences — a listing 50m away scores higher than one 60m away. For most marketplace ranking, that distinction is noise: GPS accuracy is ~10-20m and users don't care about meters. The offset parameter establishes a plateau within which all items score 1.0, removing this micro-instability and concentrating decay's discriminative power on meaningful distances.

Incorrect (no offset — micro-distance noise drives ranking):

{
  "gauss": {
    "location": {
      "origin": "38.71,-9.13",
      "scale": "2km",
      "decay": 0.5
    }
  }
}

A restaurant at 80m scores 0.999; one at 200m scores 0.997. Real differences below GPS accuracy now affect rank order — pure noise amplification.

Correct (offset creates a meaningful plateau):

{
  "gauss": {
    "location": {
      "origin": "38.71,-9.13",
      "offset": "300m",
      "scale": "1.7km",
      "decay": 0.5
    }
  }
}

Everything within 300m scores 1.0 (no micro-distinctions inside walking distance); beyond that, decay engages.

Offset = your signal noise floor:

Why this matters for ranking stability: Without offset, every refresh of a user's location (which moves ~10m as they walk) reshuffles results. With offset, ranking is stable within the noise floor — the user perceives consistent results, not flickering.

Pairing with `decay`: Offset doesn't replace decay; they work together. Offset says "no penalty inside this radius," decay says "what penalty applies at the edge of scale." Tune them independently.

Anti-pattern — using offset as a filter substitute: If you want hard "no results beyond 10km," use a geo_distance filter, not a decay with huge offset. Offset is for soft within-plateau leniency, not hard cutoffs.

Reference: OpenSearch decay function parameters · GPS accuracy reference (US.gov)

Calibrate Decay Scale to the 0.5-Score Distance Target

scale is the most-misset decay parameter because its meaning is non-obvious — it's "the distance from (origin + offset) at which the function returns decay" (default 0.5). Set it without thinking about that and you get either a wall ("anything beyond 1km is dead") or a flat noise floor ("everything within 100km looks the same"). The right calibration is empirical: pick the distance/duration at which you want the score to halve, set scale to that minus offset.

Incorrect (scale chosen by gut feel — falloff happens nowhere useful):

{
  "gauss": {
    "location": {
      "origin": "38.71,-9.13",
      "scale": "100km",
      "decay": 0.5
    }
  }
}

For city food-delivery, scale: 100km means a 50km-away restaurant still scores ~0.85 — distance signal is nearly absent.

Correct (scale = the "half-score distance" minus offset, derived from data):

Step 1 — Look at the historical click/conversion data:

SELECT distance_km_bucket, COUNT(*) as clicks, AVG(converted) as conv_rate
FROM search_events
WHERE category = 'food_delivery'
GROUP BY distance_km_bucket
ORDER BY distance_km_bucket;

Suppose conversion halves at 2km from origin.

Step 2 — Set scale so the function halves at that distance:

{
  "gauss": {
    "location": {
      "origin": "38.71,-9.13",
      "offset": "200m",
      "scale": "1.8km",
      "decay": 0.5
    }
  }
}

Now the decay function's 0.5 point matches the observed conversion 0.5 point.

The math:

Gauss:    s(d) = exp( -ln(decay) / scale^2 * max(0, d - offset)^2 )
Exp:      s(d) = exp(  ln(decay) / scale      * max(0, d - offset)   )
Linear:   s(d) = max(0, 1 - max(0, d - offset) / scale * (1 - decay))

Solving for scale at a target half-distance d_half (with decay = 0.5):
  Gauss:  scale = (d_half - offset) / sqrt(1)  = d_half - offset
  Exp:    scale = (d_half - offset)            (same — at decay=0.5)
  Linear: scale = (d_half - offset) * 2        (linear hits 0.5 at half-scale)

So for Gauss/Exp with decay=0.5, scale = d_half - offset directly.

Calibration heuristics by domain:

Validation: Plot the resulting decay curve against historical conversion-by-distance. If they don't track, recalibrate scale or switch the curve shape.

Common mistake — copying values across domains: A scale: 5km calibrated for restaurant delivery is wrong for hotel search. The "half-distance" is fundamentally different.

Reference: OpenSearch decay function parameters

Diversify Categories Hierarchically in the Top Window

Marketplaces have a taxonomy (apartments / houses / rooms; restaurants / cafes / bars; clothing / shoes / accessories) — and search relevance often clusters within one branch even for broad queries. A user searching "lisbon stay" who only sees apartments misses the option to consider hotels or villas. Hierarchical category diversification ensures the top window spans the relevant subtree of the taxonomy, not a single leaf. Airbnb's "Learning to Rank Diversely" (2022) formalizes this as a constrained re-rank.

Incorrect (no category diversification — top-10 is all apartments):

{
  "size": 10,
  "query": { "match": { "city": "lisbon" } }
}

Correct (re-rank with per-category quota in top-K):

def hierarchical_diversify(ranked, taxonomy, top_k=10):
    """
    taxonomy: {category_id: parent_id} mapping
    Ensures top_k spans at least 3 distinct top-level categories if possible.
    """
    target_top_categories = 3
    top_seen = set()
    result = []
    overflow = []

    for item in ranked:
        top_cat = root_of(item.category_id, taxonomy)

        if len(result) < target_top_categories:
            # First N slots: enforce distinct top-level categories
            if top_cat not in top_seen:
                result.append(item)
                top_seen.add(top_cat)
            else:
                overflow.append(item)
        else:
            # After diversity quota met: fill by relevance from full pool
            if len(result) < top_k:
                result.append(item)

        if len(result) >= top_k:
            break

    # Fill remaining slots with overflow if needed
    for item in overflow:
        if len(result) >= top_k:
            break
        if item not in result:
            result.append(item)

    return result

def root_of(category_id, taxonomy):
    while taxonomy.get(category_id):
        category_id = taxonomy[category_id]
    return category_id

Use a sliding-window quota for longer pages:

def windowed_diversity(ranked, window=10, max_per_top_cat_per_window=4):
    result = []
    for batch_start in range(0, len(ranked), window):
        window_items = ranked[batch_start : batch_start + window]
        per_cat_count = collections.Counter()
        diversified = []
        overflow = []
        for item in window_items:
            top_cat = root_of(item.category_id, taxonomy)
            if per_cat_count[top_cat] < max_per_top_cat_per_window:
                diversified.append(item)
                per_cat_count[top_cat] += 1
            else:
                overflow.append(item)
        result.extend(diversified + overflow)
    return result

Why hierarchical, not flat? A flat "max-per-category" treats apartments_studio and apartments_1br as different categories — but the user perceives both as "apartments." Hierarchical rolls up to top-level taxonomy nodes, matching user mental model.

Calibration:

Don't over-diversify: If a query is genuinely category-specific ("studio apartment lisbon"), forcing the top-10 to include 3 different categories surfaces irrelevant results. Detect category specificity from the query (NER on accommodation types) and reduce the diversity target proportionally.

Coupling with MMR: Use hierarchical category diversity as a hard quota on the top-3 slots, then apply MMR with λ=0.7 on the remaining slots for soft within-category diversity.

Reference: Airbnb — Learning to Rank Diversely (arXiv 2210.07774) · Diversity in Recommender Systems Survey (Castells et al., 2022)

Use Determinantal Point Processes for Joint Quality and Diversity

MMR is a greedy heuristic; Determinantal Point Processes (DPPs — Kulesza & Taskar 2012) are the principled probabilistic framework for diverse subset selection. A DPP defines a probability distribution over subsets where the probability is proportional to the determinant of a kernel matrix that encodes both per-item quality and pairwise similarity. The result: subsets that simultaneously maximize quality AND minimize within-subset similarity, with a single tunable trade-off and clean theoretical properties (the only sampler distribution that's both repulsive and tractable).

The DPP construction:

Kernel: L_ij = q_i × s_ij × q_j

  where:
    q_i  = quality of item i (e.g., relevance score, conversion rate)
    s_ij = similarity between items i and j (cosine over embeddings, ∈ [0,1])

Probability of subset Y ⊆ [N]:
    P(Y) ∝ det(L_Y)

  Maximizes when items are individually high-quality (large diagonal)
  AND pairwise diverse (small off-diagonal, making det large)

Incorrect (MMR greedy — locally optimal, can miss good subsets):

# Greedy MMR picks one item at a time — may get stuck in local optima
selected = mmr_rerank(candidates, query_vec, lambda_=0.5, top_k=10)

Correct (DPP MAP via greedy submodular approximation):

import numpy as np

def dpp_greedy(candidates, query_vec, top_k=10):
    """
    Greedy MAP inference for DPP — same time complexity as MMR but uses
    proper DPP kernel structure. Within 1-1/e of optimum (submodular guarantee).
    """
    # 1. Build feature vectors and quality scores
    feats = np.array([c.embedding for c in candidates])
    feats = feats / np.linalg.norm(feats, axis=1, keepdims=True)  # L2-normalize
    quality = np.array([(c.relevance + c.shrunken_conv_rate) / 2.0 for c in candidates])

    # 2. Construct DPP kernel: L_ij = q_i × s_ij × q_j
    sim = feats @ feats.T  # cosine similarity (since normalized)
    L = (quality[:, None] * sim) * quality[None, :]

    # 3. Greedy MAP — pick items maximizing log-det incrementally
    selected = []
    remaining = list(range(len(candidates)))
    while len(selected) < top_k and remaining:
        best_i = best_gain = None
        for i in remaining:
            sub_idx = selected + [i]
            try:
                gain = np.linalg.slogdet(L[np.ix_(sub_idx, sub_idx)])[1]
            except np.linalg.LinAlgError:
                continue
            if best_gain is None or gain > best_gain:
                best_gain, best_i = gain, i
        if best_i is None:
            break
        selected.append(best_i)
        remaining.remove(best_i)

    return [candidates[i] for i in selected]

When DPP beats MMR (and when it doesn't):

Computational note: DPP greedy is O(N × k²) for top-k from N candidates due to the determinant updates. For top-10 from top-500, that's ~50k ops — fast. For top-50 from top-5000, ~12.5M ops — getting expensive; switch to MMR or apply DPP only on the top-100 candidates.

Implementation tip: Use Cholesky updates rather than full determinant re-computation; reduces inner loop from O(k³) to O(k²).

Library options: dppy (Python), or implement directly — the kernel is just quality⊙sim⊙quality.

Reference: Kulesza & Taskar — Determinantal Point Processes for Machine Learning (Foundations and Trends 2012) · Chen et al. — Fast Greedy MAP Inference for DPP (NIPS 2018)

Cap Impressions Per Host with Max-Per-Group Constraint

A prolific host with 50 listings in Lisbon can occupy 8 of the top-10 slots — relevance-correct but UX-terrible. The user sees one host's brand five times and feels the marketplace is small. OpenSearch's collapse query collapses results by a field (e.g., host_id) keeping only the top N per group; a per-page hard cap (2-3 listings per host max) is the standard marketplace fix. Airbnb has documented this pattern in their diversity research.

Incorrect (no per-host cap — single host dominates top page):

{
  "size": 20,
  "query": { "match": { "city": "lisbon" } },
  "sort": [{ "_score": "desc" }]
}

Top-20 might contain 8 listings from one host who happens to have well-optimized titles.

Correct (collapse by host_id, top-2 per host):

{
  "size": 20,
  "query": { "match": { "city": "lisbon" } },
  "collapse": {
    "field": "host_id",
    "inner_hits": {
      "name": "more_from_host",
      "size": 1,
      "sort": [{ "_score": "desc" }]
    }
  }
}

This returns top-20 distinct hosts (one listing per host) with inner_hits carrying the second-best from each host for "more from this host" UI surfacing.

For top-2 per host (not top-1), use script-based re-ranking:

def per_host_cap(ranked, max_per_host=2):
    host_count = collections.Counter()
    capped = []
    overflow = []
    for item in ranked:
        if host_count[item.host_id] < max_per_host:
            capped.append(item)
            host_count[item.host_id] += 1
        else:
            overflow.append(item)
    # Append overflow at the bottom to fill page if needed
    return capped + overflow[:max(0, len(ranked) - len(capped))]

candidates = opensearch.search(...)
final_page = per_host_cap(candidates, max_per_host=2)

Why a hard cap is the right tool here (and MMR isn't enough): MMR uses similarity in embedding space to deduplicate — but two listings from the same host can have very different embeddings (different photos, different titles, different prices) while still being from the same host. The "same host" signal is structural, not semantic; you need a structural constraint to enforce it.

Tune by surface:

Combine with category diversity (`div-category-diversity`): Cap per host AND per category in the top window. Both are independent dimensions of "user perceives the marketplace as diverse."

Don't collapse on listing_id by accident: collapse: {field: "listing_id"} collapses identical listings into one result — useful for de-duplicating but unrelated to per-host diversity.

When NOT to cap: When the user has explicitly searched for a specific host (host:"Some Inn Lisbon"), suspend the cap — they want to see that host's full catalog.

Reference: OpenSearch collapse · Airbnb — Learning to Rank Diversely (arXiv 2210.07774)

Apply MMR Rerank for Top-Window Diversity

A pure relevance-ranked top-10 often shows the same neighborhood five times, the same price tier seven times, the same host twice. Users perceive this as "the same listing rendered in slightly different ways" and disengage. MMR (Maximal Marginal Relevance — Carbonell & Goldstein, SIGIR 1998) is a post-rank re-ranker that greedily picks the next item by trading relevance against similarity to already-picked items. It's the simplest, most-cited diversity algorithm; OpenSearch supports it natively for vector search since 3.3 (enable the mmr_over_sample_factory and mmr_rerank_factory system-generated processors via cluster.search.enabled_system_generated_factories before using the ext.mmr block on a knn or neural top-level query).

The MMR objective:

MMR = arg max  [ λ × Rel(item, query) − (1 − λ) × max Sim(item, selected) ]
       item                                          j∈selected

  λ = 1.0 → pure relevance (no diversity)
  λ = 0.5 → balanced (typical starting point)
  λ = 0.0 → pure diversity (no relevance — bad)

Incorrect (no diversity — top-10 is 7 lofts in Bairro Alto):

{
  "size": 10,
  "query": {
    "function_score": {
      "query": { "match": { "city": "lisbon" } },
      "functions": [ /* relevance signals */ ]
    }
  }
}

Correct (OpenSearch native MMR re-ranker on vector search):

POST /listings/_search
{
  "size": 50,
  "query": {
    "knn": {
      "embedding": {
        "vector": [/* query vector */],
        "k": 50
      }
    }
  },
  "ext": {
    "mmr": {
      "candidates": 50,
      "diversity": 0.5,
      "vector_field": "embedding"
    }
  }
}

Manual MMR re-rank in Python (when you need custom similarity):

import numpy as np

def mmr_rerank(candidates, query_vec, lambda_=0.5, top_k=10):
    """Greedy MMR — pick next item by relevance minus max-similarity-to-picked."""
    candidate_vecs = np.array([c.embedding for c in candidates])
    rel_scores = candidate_vecs @ query_vec
    selected = []
    remaining = list(range(len(candidates)))

    for _ in range(top_k):
        if not remaining:
            break
        best_i = best_score = None
        for i in remaining:
            sim_to_selected = (
                max(candidate_vecs[i] @ candidate_vecs[s] for s in selected)
                if selected else 0.0
            )
            mmr = lambda_ * rel_scores[i] - (1 - lambda_) * sim_to_selected
            if best_score is None or mmr > best_score:
                best_score, best_i = mmr, i
        selected.append(best_i)
        remaining.remove(best_i)

    return [candidates[i] for i in selected]

Calibrating `λ`:

Apply MMR only to the top window (top-50 → top-10), not the whole result set:

Re-ranking 5000 candidates with MMR's O(k × N) loop is expensive and pointless — diversity only matters in what the user sees. Apply MMR in the rescore phase on the top-50, output top-10.

Why this beats post-hoc filtering for diversity: "Show me one listing per neighborhood, ranked by relevance" sounds equivalent but loses information — it discards perfectly-relevant items just for being from a popular neighborhood. MMR penalizes redundancy continuously rather than thresholding, preserving rank order while spreading attribute coverage.

Reference: Carbonell & Goldstein — The Use of MMR, Diversity-Based Reranking (SIGIR 1998) · OpenSearch MMR vector search (3.3+) · OpenSearch blog — Improving vector search diversity through native MMR

Apply Window-Based Diversity Penalty in Rescore

Global re-rankers (MMR, DPP) reshuffle the entire top-K — which can cause significant rank-position changes that confuse repeat users ("where did that listing I clicked yesterday go?"). A window-based penalty applies diversity only to the visible window the user is scrolling through, leaving lower-ranked positions stable. The pattern: as you build the top-K, penalize an item's score by its similarity to items already placed within the last w positions; once a similar item exits the window, the penalty drops off.

The window-penalty algorithm:

For position i from 1 to top_k:
    candidate_score(c) = base_score(c) - α × max{ sim(c, j) : j ∈ already_placed[i-w : i] }
    place item with highest candidate_score
    advance i

α controls the penalty strength; w controls how "local" the diversity is.

Incorrect (global MMR reshuffles ranking — breaks repeat-visit memory):

# Re-ranks all top-50 with single global trade-off
final = mmr_rerank(candidates, query_vec, lambda_=0.5, top_k=50)

A listing that ranked #4 yesterday might rank #19 today purely from MMR diversity vs whoever else is in the top window — user thinks the marketplace is "unstable."

Correct (window-based penalty — only adjacent positions interact):

import numpy as np

def window_diverse_rerank(candidates, query_vec, top_k=50, window=5, alpha=0.3):
    """
    Place items into positions 1..top_k in order. At each position,
    penalize candidates by similarity to items in the last `window` placed positions.
    """
    feats = np.array([c.embedding for c in candidates])
    feats = feats / np.linalg.norm(feats, axis=1, keepdims=True)
    base_scores = np.array([c.base_score for c in candidates])

    placed = []
    remaining = list(range(len(candidates)))

    for pos in range(top_k):
        if not remaining:
            break
        # Compute window-local penalty
        window_set = placed[-window:]
        if window_set:
            sim_to_window = feats[remaining] @ feats[window_set].T  # shape (R, W)
            penalty = sim_to_window.max(axis=1)
        else:
            penalty = np.zeros(len(remaining))

        adjusted = base_scores[remaining] - alpha * penalty
        winner_local = int(np.argmax(adjusted))
        winner_global = remaining[winner_local]
        placed.append(winner_global)
        remaining.pop(winner_local)

    return [candidates[i] for i in placed]

Why this preserves rank stability better than MMR: Items far from the current placement window don't influence each other — so adding/removing a single item in the candidate set causes only local reshuffles, not global. Users perceive consistent rankings session-over-session.

Parameter calibration:

Combine with hard caps: Window penalty for soft within-window diversity; hard per-host cap (div-max-per-host) for structural diversity. They address different problems: window-penalty stops near-duplicates from clumping; max-per-host stops a single host from dominating regardless of similarity.

When to NOT diversify at all: Specific intent queries ("Hotel Marriott Lisbon"), filter-narrow queries (already 4 results — diversity is moot), repeat queries from the same user (stability > diversity).

Reference: Carbonell & Goldstein — The Use of MMR (SIGIR 1998) · Castells, Hurley, Vargas — Novelty and Diversity in Recommender Systems (book chapter)

Calculate A/B Sample Size from MDE Before Running

"Run until significance" inflates false positive rate to ~25-30% via repeated peeking. The disciplined approach: pre-compute the sample size your test needs from three inputs — your Minimum Detectable Effect (MDE), the desired statistical power (usually 0.8), and your baseline metric's variance. Run until that sample size is reached, then evaluate once. This is the difference between "this ranker looks better" and "this ranker is +1.2% better with 95% confidence, p<0.01."

Incorrect (peek and stop — false positive rate inflates):

# Run the test, check p-value daily, stop as soon as p<0.05
def run_ab_until_significance(treatment, control, max_days=30):
    for day in range(1, max_days + 1):
        data = collect_data_for(day)
        p = ttest(data.treatment, data.control).pvalue
        if p < 0.05:
            return {"day": day, "p": p, "result": "ship"}  # 25-30% of these are false positives
    return {"result": "inconclusive"}

Correct (pre-compute n from MDE × power × variance, run until n, evaluate once):

import math
from scipy.stats import norm

def sample_size_per_arm(baseline_rate, mde_relative, power=0.80, alpha=0.05):
    """
    Sample size per arm for two-proportion z-test.

    baseline_rate:  observed metric (e.g., conversion rate 0.024)
    mde_relative:   smallest effect you care about as fraction of baseline
                    (e.g., 0.05 = "detect a 5% relative lift")
    power:          1 - β (probability of detecting the effect when it exists)
    alpha:          significance threshold (Type I error rate)
    """
    p1 = baseline_rate
    p2 = baseline_rate * (1 + mde_relative)
    pbar = (p1 + p2) / 2

    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta  = norm.ppf(power)

    n = (
        (z_alpha * math.sqrt(2 * pbar * (1 - pbar))
         + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
        / (p2 - p1) ** 2
    )
    return math.ceil(n)

# Example: marketplace with 2.4% conversion, want to detect 5% relative lift
n = sample_size_per_arm(baseline_rate=0.024, mde_relative=0.05)
# → n = 282,000 per arm → 564,000 sessions total
# At 50k sessions/day per arm: need ~6 days of pure traffic

The fundamental relationship:

n ∝ σ² / MDE²

  Halving the MDE quadruples the required sample size.
  Doubling the baseline variance doubles the required sample size.
  Going from 80% to 90% power roughly +30% sample size.

MDE calibration by traffic level:

Lower the bar with two techniques (in order of cost):

1. CUPED (see eval-cuped-variance-reduction) — 40-60% variance reduction → halve required sample size. Cheap, just statistical adjustment. 2. Interleaving (see bias-interleaved-evaluation) — 10-100× more sample-efficient per impression. More implementation cost, but transformative for low-traffic verticals.

Sequential testing (if you really must peek):

If iteration speed matters more than purity, use sequential tests with corrected p-value thresholds (e.g., O'Brien-Fleming, Pocock) instead of repeated naive t-tests. Modern A/B platforms (Eppo, GrowthBook, Statsig) provide these out of the box. Never peek without sequential corrections.

Always combine with sample-ratio mismatch (SRM) check:

def srm_check(treatment_n, control_n, alpha=0.001):
    """Sanity-check that random assignment produced the expected ratio.
       If it didn't, the experiment is invalid regardless of the result."""
    from scipy.stats import chisquare
    chi, p = chisquare([treatment_n, control_n])
    if p < alpha:
        raise SRMError(
            f"Assignment imbalance detected (p={p}). "
            f"Possible causes: bot traffic, redirect loss, logging bug."
        )

A test that looks significant but failed SRM is broken — don't ship.

Why pre-computation matters: It anchors you to a decision rule before seeing data. Without it, you'll find a way to convince yourself any result is "interesting." Sample-size discipline is the difference between A/B testing as science and A/B testing as confirmation bias.

Reference: Microsoft — Online Controlled Experiments at Large Scale (KDD 2013) · Kohavi, Tang, Xu — Trustworthy Online Controlled Experiments (Cambridge book, 2020) · Eppo MDE calculator

Run Ablation Studies to Attribute Lift to Specific Components

A marketplace ranking stack stacks many rules at once — BM25F, Wilson Lower Bound, listing embeddings, gauss decay, conversion-weighted scoring, IPS-corrected LTR, MMR diversity. When the new system as a whole is +0.04 NDCG@10, which of those moved the needle? Ablation studies answer this: turn each component off (or replace it with a no-op), measure the NDCG drop, and attribute the lift proportionally. Without ablation, you maintain a fragile stack of "everything matters" with no insight into which components are doing the work and which are dead weight.

Incorrect (one big A/B test, no idea which component contributed):

# Old ranker vs new ranker (everything changed at once)
# Result: +0.038 NDCG@10, +1.2% conversion
# Insight: zero. We don't know if the listing embeddings did it or the bias correction
#          or the new decay function or the host fairness reweighting.

Correct (additive ablation — measure marginal contribution of each component):

COMPONENTS = [
    ("baseline_bm25", build_bm25_only),
    ("+ bm25f_field_weights", add_bm25f),
    ("+ wilson_rating_signal", add_wilson),
    ("+ listing_embeddings", add_embeddings),
    ("+ gauss_geo_decay", add_geo_decay),
    ("+ conversion_weighted_rerank", add_conv_rerank),
    ("+ ips_position_correction", add_ips),
    ("+ mmr_diversity", add_mmr),
]

def additive_ablation(judgment_set, k=10):
    """Build up the stack one component at a time, measure NDCG after each."""
    results = []
    cumulative = None
    for name, mutator in COMPONENTS:
        cumulative = mutator(cumulative)
        ndcg = mean_ndcg_at_k(cumulative, judgment_set, k=k)
        results.append({"step": name, "ndcg": ndcg})
    return results

# Output:
# baseline_bm25                       NDCG@10 = 0.512
# + bm25f_field_weights               NDCG@10 = 0.541   (+0.029)
# + wilson_rating_signal              NDCG@10 = 0.548   (+0.007)
# + listing_embeddings                NDCG@10 = 0.583   (+0.035)  ← big win
# + gauss_geo_decay                   NDCG@10 = 0.594   (+0.011)
# + conversion_weighted_rerank        NDCG@10 = 0.612   (+0.018)
# + ips_position_correction           NDCG@10 = 0.620   (+0.008)
# + mmr_diversity                     NDCG@10 = 0.619   (-0.001)  ← negligible / negative

Now you know: listing embeddings + BM25F + conversion-weighted scoring did most of the work; MMR diversity is actually slightly hurting NDCG (which is expected — it trades NDCG for engagement, so the right next step is to measure session engagement separately).

Use leave-one-out for "is this component still pulling its weight" checks:

def leave_one_out_ablation(full_stack, components, judgment_set, k=10):
    """For each component, build a stack without it and measure NDCG drop."""
    baseline = mean_ndcg_at_k(full_stack, judgment_set, k=k)
    results = []
    for name in components:
        ablated = remove_component(full_stack, name)
        ndcg = mean_ndcg_at_k(ablated, judgment_set, k=k)
        results.append({"removed": name, "ndcg": ndcg, "drop": baseline - ndcg})
    return sorted(results, key=lambda x: -x["drop"])

# Output:
# Removed bm25f_field_weights         NDCG@10 = 0.591  drop=0.029
# Removed listing_embeddings          NDCG@10 = 0.585  drop=0.035  ← still load-bearing
# Removed conversion_weighted_rerank  NDCG@10 = 0.602  drop=0.018
# Removed mmr_diversity               NDCG@10 = 0.620  drop=0.000  ← consider removing

Order-dependence trap: Additive ablation measures marginal contribution at each step — later additions look smaller because earlier ones already captured related signal. For component attribution in the final system, prefer leave-one-out (also called "subtractive ablation"), which is order-independent.

Run ablations against multiple metrics: NDCG@10, NDCG@5, per-stratum NDCG (head/torso/tail), and online metric proxies. A component might be NDCG-neutral but improve tail NDCG by 0.05 — visible only in stratified ablation.

Use ablations to feed Pareto multi-objective decisions: If two components have similar NDCG impact but very different latency cost, the cheaper one wins. Cross-reference with market-pareto-multi-objective.

When ablation finds a component contributes ~0: 1. Don't immediately remove it — it might be redundant given other components but load-bearing without them. 2. Run a "minimum stack" experiment — remove the suspect plus 1-2 of its plausible substitutes; if quality stays the same, it's truly removable. 3. If it stays, document it as kept-for-defense-in-depth, not because it pulls weight.

Reference: Sculley et al. — Hidden Technical Debt in Machine Learning Systems (NIPS 2015) · Wikipedia — Ablation (Artificial Intelligence)) · Capital One Tech — Ablation Studies for ML

Apply CUPED to Halve A/B Sample Size with Pre-Experiment Covariates

A user's metric in an A/B test (conversion, bookings, sessions) is dominated by who that user is, not by the treatment. A power user converts 5× more than a casual user — that variance overwhelms the small treatment effect. CUPED (Controlled-experiment Using Pre-Existing Data, Microsoft 2013) adjusts each user's post-experiment metric by their pre-experiment metric, removing the persistent individual-level variation and exposing the treatment effect. Result: typical 40-60% variance reduction — equivalent to running with 2× the traffic, at zero infrastructure cost. Deployed at Netflix, Microsoft, Booking, Airbnb, Uber, DoorDash, LinkedIn, TripAdvisor.

Incorrect (raw difference-in-means — most variance is from individual differences):

import numpy as np
from scipy.stats import ttest_ind

# Treatment vs control on bookings_per_user — high variance, slow to power
treatment = np.array([user.bookings_post for user in treatment_users])
control   = np.array([user.bookings_post for user in control_users])

ate = treatment.mean() - control.mean()  # average treatment effect
stat, pvalue = ttest_ind(treatment, control)

The variance of treatment.mean() - control.mean() is dominated by user-level variation that has nothing to do with the experiment.

Correct (CUPED — adjust post-metric by pre-metric covariate):

import numpy as np
from scipy.stats import ttest_ind

# Y = post-experiment metric (e.g., bookings during the test window)
# X = pre-experiment metric for the SAME user (e.g., bookings in the 28d before)
all_users = treatment_users + control_users
Y = np.array([u.bookings_post for u in all_users])
X = np.array([u.bookings_pre  for u in all_users])

# 1. Compute theta from pooled data — Cov(X, Y) / Var(X)
theta = np.cov(X, Y, ddof=1)[0, 1] / np.var(X, ddof=1)

# 2. Adjust each Y by removing the X-predicted portion
#    Subtract X.mean() so the adjusted metric still has the same mean as Y
Y_cuped = Y - theta * (X - X.mean())

# 3. Run the t-test on the adjusted metric
treat_idx   = [i for i, u in enumerate(all_users) if u.in_treatment]
control_idx = [i for i, u in enumerate(all_users) if not u.in_treatment]
ate_cuped = Y_cuped[treat_idx].mean() - Y_cuped[control_idx].mean()
stat, pvalue = ttest_ind(Y_cuped[treat_idx], Y_cuped[control_idx])

The variance reduction:

Var(Y_cuped) = Var(Y) × (1 - ρ²)

  ρ = correlation(X, Y) across users

  Typical ρ for behavioral metrics:
    bookings_pre vs bookings_post:    ρ ≈ 0.6-0.8  → 36-64% variance reduction
    sessions_pre vs sessions_post:    ρ ≈ 0.7-0.9  → 49-81% variance reduction
    conversion_pre vs conversion_post: ρ ≈ 0.3-0.5  → 9-25% variance reduction

Picking the covariate: The best X is the SAME metric measured in a comparable pre-experiment window. Longer pre-window = higher ρ = more reduction; common choice is 28-56 days.

Picking the pre-window length:

Trap — pre-period must end BEFORE experiment starts:

# WRONG — pre-window overlaps with the experiment
X = user.bookings_in_last_28_days  # includes treatment days for in-experiment users

# RIGHT — pre-window strictly before experiment start
X = user.bookings_in_28_days_before_experiment_start

If pre and post overlap, you can introduce treatment leakage into the covariate and invalidate the analysis.

For new users with no pre-period data: Either drop them from the CUPED analysis (acceptable if they're a small minority) or use a population-level default (X = population mean) — this gives zero variance reduction for those users but doesn't break anything.

Combining CUPED with stratified analysis:

# Apply CUPED within strata for better attribution
for stratum in ["high_value_user", "medium_value_user", "low_value_user"]:
    users = [u for u in all_users if u.value_tier == stratum]
    apply_cuped_and_test(users)

Validation: After applying CUPED, compute the empirical variance of Y_cuped and compare to Y. The ratio Var(Y_cuped) / Var(Y) should match (1 - ρ²) within ~5%. If it doesn't, the covariate isn't well-correlated with the outcome (use a different covariate) or the computation has a bug.

Modern A/B platforms have CUPED built in: Eppo, GrowthBook, Statsig all support CUPED with a checkbox. If you're using these, just enable it on your primary metric; if not, the implementation above is ~10 lines.

Reference: Deng, Xu, Kohavi, Walker — Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (WSDM 2013) · GrowthBook CUPED documentation · Matteo Courthoud — Understanding CUPED

Build a Graded Judgment Set for Offline Evaluation

Every offline ranking metric (NDCG, MAP, MRR, Precision@k) requires a "ground truth" — a graded judgment set of (query, item, relevance_grade) tuples that says "for this query, this item is grade 0/1/2/3/4." Without one, you can't measure whether any of your 49 ranking rules actually improved ranking quality. The set is built once and refreshed quarterly; it's the foundation that turns "I think this ranker is better" into "this ranker is +0.043 NDCG@10 better, p<0.01." Cranfield-style methodology (used by TREC since 1992) is the canonical approach.

Incorrect (no judgment set — every ranking change is a guess):

# "I think the new ranker is better — let me just ship it to A/B"
# No offline confidence; every change risks user exposure
def evaluate_ranker(ranker):
    return None  # ¯\_(ツ)_/¯

Correct (graded judgment set with stratified query sampling):

# 1. Sample queries stratified by head/torso/tail (search-volume buckets)
queries = sample_stratified_queries(
    log_source="search_log_30d",
    n_per_stratum={"head": 200, "torso": 200, "tail": 200},
    strata_def={"head": "rank <= 100", "torso": "100 < rank <= 10000", "tail": "rank > 10000"}
)

# 2. For each query, get top-N from a baseline ranker (avoid annotator effort
#    on irrelevant items by limiting to ranker-recall)
candidate_pairs = []
for q in queries:
    top_50 = baseline_ranker.search(q, k=50)
    candidate_pairs.extend([(q, item) for item in top_50])

# 3. Annotate with 5-grade scale (Cranfield-standard)
GRADE_GUIDELINE = """
0 = Off-topic (different intent)
1 = Related but not what user wanted
2 = On-topic, acceptable result
3 = Strong match (user likely satisfied)
4 = Perfect match (exemplar of intent)
"""

# 4. Multiple annotators per pair to measure inter-annotator agreement (Cohen's κ)
#    Reject the set if κ < 0.6 — grading guidelines need refinement

# 5. Store as JSONL — one record per (query, item, grade) with metadata
import json
with open("judgment_set_v1.jsonl", "w") as f:
    for q, item, grade in annotations:
        f.write(json.dumps({
            "query": q.text,
            "query_stratum": q.stratum,        # head/torso/tail
            "query_intent": q.intent,           # navigational/transactional/exploratory
            "item_id": item.id,
            "grade": grade,                     # 0-4
            "annotator_id": annotator.id,
            "annotated_at": "2026-05-17"
        }) + "\n")

Sizing the judgment set:

Stratification matters more than sheer size: A 600-query set with proper head/torso/tail/intent stratification beats a 6000-query set of random head queries. Tail and edge-case queries are where rankers fail most distinctively.

Refresh cadence: Quarterly. The catalogue evolves, user intents drift, new query patterns emerge. Stale judgment sets lie about ranking quality.

Pair with online-offline correlation: A judgment set is only useful if it predicts online behavior — see eval-online-offline-correlation for the validation step.

Tooling: TREC Eval is the canonical scorer; Sproutwords and Snorkel help with weak-supervision augmentation if pure human annotation isn't feasible.

Reference: TREC — Common Evaluation Measures · Cranfield methodology overview (Voorhees, NIST) · Shaped.ai — NDCG and graded relevance

Choose HNSW for Latency, IVF for Memory at Scale

HNSW (Hierarchical Navigable Small Worlds) gives sub-10ms p99 latency and the highest recall at small-to-medium scale (<100M vectors) but uses ~1.5-2× the vector data in graph links — RAM cost is brutal at billion scale. IVF (Inverted File) clusters vectors and probes only the nearest centroids; memory footprint is roughly the raw vector size but recall and latency depend heavily on nprobe. Airbnb (Abdool et al. 2025) evaluated both and chose IVF for their listing index based on speed/quality tradeoff at their scale.

Incorrect (HNSW at 500M-vector scale with default parameters — OOM):

PUT /listings_500m
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 128,
        "method": {
          "name": "hnsw",
          "engine": "lucene",
          "parameters": { "ef_construction": 512, "m": 48 }
        }
      }
    }
  }
}

At 500M vectors × 128 dims × 4 bytes = 256GB raw; HNSW with m=48 adds ~50% in graph links = 384GB working set per shard replica.

Correct (IVF for billion-scale with controlled probe budget):

PUT /listings_500m
{
  "settings": { "index.knn": true, "number_of_shards": 16 },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 128,
        "method": {
          "name": "ivf",
          "engine": "faiss",
          "parameters": { "nlist": 4096, "nprobes": 32 }
        }
      }
    }
  }
}

Sizing guidance:

Tune `nprobes` empirically: Higher = better recall, more latency. Start at sqrt(nlist) and adjust based on offline recall@k evaluation against a gold set.

Reference: OpenSearch k-NN engines · Embedding-Based Retrieval for Airbnb Search