Marketplace Search Recsys Planning

Marketplace Engineering Two-Sided Search and Recsys Planning Best Practices

Comprehensive planning, design and diagnostic guide for search and recommendation systems in two-sided trust marketplaces. Covers OpenSearch index, query and ranking patterns, the methodology for planning retrieval work, the handoff points to recommendation-specific tooling, and the instrumentation and dashboard layer that turns measurement into ongoing decision making. Contains 57 rules across 10 categories ordered by cascade impact, plus two playbooks (plan a new system from scratch, diagnose an existing one) and explicit living-artefact conventions (decisions log, golden set, gotchas).

When to Apply

Reference this skill when:

• Planning a new marketplace retrieval project from scratch
• Reviewing an existing retrieval system that feels stale, unfair, or unpersonalised
• Designing the OpenSearch index mapping, analyzers, or query DSL
• Choosing retrieval primitives per product surface (search, recs, hybrid, curated)
• Deciding which search quality metrics to track and dashboard
• Running the weekly search-quality review ritual
• Diagnosing a silent regression in ranking, coverage, or zero-result rate
• Deciding when a retrieval problem is actually a personalisation problem

This skill is the precursor to marketplace-personalisation. Start here for planning and search work; hand off to the personalisation skill when the diagnosed bottleneck is impression tracking, feedback-loop bias, or AWS Personalize-specific design.

Living Context

This skill treats the system as evolving. Three living artefacts carry context across sessions, releases, and team changes — read them before making suggestions, update them after every shipped change:

• `gotchas.md` (in this skill folder) — append-only diagnostic lessons. Every gotcha

has a date and a short description of what surprised the team and how it was resolved.

• Decisions log (maintained in the product repo, typically decisions/*.md) —

every ranking change, schema tweak, and synonym edit recorded with its hypothesis, offline and online evidence, ship criterion, outcome, and rollback path. See rule `plan-maintain-a-decisions-log`.

• Golden query set (frozen per eval cycle, committed to the product repo) — the

reference set of queries against which every ranking change is offline-evaluated before an online test. See rule `plan-version-the-golden-set`.

Rule Categories

Categories are ordered by cascade impact on the retrieval lifecycle: intent misunderstanding poisons architecture; wrong architecture poisons index; wrong index poisons retrieval forever until a reindex; every downstream layer inherits the upstream error.

Quick Reference

1. Problem Framing and User Intent (CRITICAL)

• `intent-map-queries-to-intent-classes` — classify before retrieving
• `intent-separate-known-item-from-discovery` — different failure modes, different strategies
• `intent-audit-live-query-logs-first` — design from real data, not imagined data
• `intent-distinguish-transactional-from-exploratory` — precision vs diversity
• `intent-reject-one-search-for-everything` — per-surface query shapes
• `intent-treat-no-search-as-first-class-choice` — curated is a legitimate answer

2. Surface Taxonomy and Architecture (CRITICAL)

• `arch-map-surface-to-retrieval-primitive` — a single-source-of-truth routing table
• `arch-split-candidate-generation-from-ranking` — two-stage pipelines
• `arch-design-zero-result-fallback` — declare fallback owner per surface
• `arch-design-for-cold-start-from-day-one` — cold start is permanent, not bootstrap
• `arch-avoid-mono-stack-retrieval` — diversify primary dependencies
• `arch-route-surfaces-deliberately` — every routing decision recorded

3. Index Design and Mapping (HIGH)

• `index-design-mappings-conservatively` — reindex is expensive
• `index-use-keyword-and-text-as-multi-fields` — full-text plus exact match
• `index-match-index-and-query-time-analyzers` — tokens must agree
• `index-use-language-analyzers-for-language-fields` — language-aware stemming
• `index-separate-searchable-from-display-fields` — index only what you search
• `index-use-index-templates-for-consistency` — prevent mapping drift
• `index-stream-listing-updates-via-cdc` — freshness in seconds, not hours

4. Planning and Improvement Methodology (HIGH)

• `plan-audit-before-you-build` — instrumentation gate on kick-off
• `plan-build-golden-query-set-first` — the first artefact, not the last
• `plan-find-bottleneck-before-optimising` — theory of constraints
• `plan-maintain-a-decisions-log` — living context across team changes
• `plan-version-the-golden-set` — frozen per eval cycle
• `plan-handoff-to-personalisation-skill` — recognise the boundary

5. Query Understanding (MEDIUM-HIGH)

• `query-normalise-before-anything-else` — canonical string in
• `query-use-language-analyzers-for-stemming` — double-digit recall wins
• `query-curate-synonyms-by-domain` — domain vocabulary not thesaurus
• `query-use-fuzzy-matching-for-typos` — 10-15% of queries have typos
• `query-classify-before-routing` — single-pass classifier
• `query-build-autocomplete-on-separate-index` — latency isolation

6. Retrieval Strategy (MEDIUM-HIGH)

• `retrieve-use-filter-clauses-for-exact-matches` — filter cache wins
• `retrieve-use-bool-structure-deliberately` — must vs should vs filter
• `retrieve-run-expensive-signals-in-rescore` — rescore window limits cost
• `retrieve-combine-bm25-and-knn-via-hybrid-search` — lexical plus semantic
• `retrieve-paginate-with-search-after` — constant-cost deep pagination
• `retrieve-choose-embedding-model-deliberately` — re-embedding is expensive

7. Relevance and Ranking (MEDIUM-HIGH)

• `rank-tune-bm25-parameters-last` — upstream levers first
• `rank-use-function-score-for-business-signals` — explicit named functions
• `rank-deploy-ltr-only-after-golden-set-exists` — supervised learning needs labels
• `rank-apply-diversity-at-rank-time` — after scoring, not before
• `rank-normalise-scores-across-retrieval-primitives` — comparable scales

8. Search and Recommender Blending (MEDIUM)

• `blend-use-search-alone-for-specific-intent` — precision queries
• `blend-combine-search-and-personalisation-scores` — normalised weighted sum
• `blend-keep-hybrid-blending-explainable` — traceable results
• `blend-never-return-zero-results` — guaranteed cascade to non-empty

9. Measurement and Experimentation (MEDIUM)

• `measure-define-session-success-per-surface` — one definition per surface
• `measure-track-ndcg-mrr-zero-result-rate` — three metrics for one picture
• `measure-track-reformulation-rate-as-failure-signal` — cheapest failure metric
• `measure-use-click-models-for-implicit-judgments` — scale beyond human judges
• `measure-run-interleaving-as-cheap-ab-proxy` — 10x less sample needed

10. Instrumentation, Dashboards and Decision Triggers (MEDIUM)

• `monitor-log-every-query-with-full-context` — structured replayable events
• `monitor-scrub-pii-from-query-logs` — redact before warehouse ingestion
• `monitor-build-search-health-dashboard` — threshold lines, colour bands
• `monitor-alert-on-decision-triggers` — quality metrics, not error rates
• `monitor-track-ranking-stability-churn` — RBO churn as leading indicator
• `monitor-run-weekly-search-quality-review` — calendar-driven ritual

Planning and Improving

Two playbooks compose the rules into end-to-end workflows:

• `references/playbooks/planning.md` — Plan a new marketplace retrieval system from scratch. Nine-step workflow from intent audit through the first A/B-tested online lift, with explicit exit criteria per step.
• `references/playbooks/improving.md` — Diagnose and improve an existing retrieval system. Decision tree that walks through telemetry, index freshness, coverage, baseline gap, cold start, segment regressions, and algorithm iteration in that order, with hand-off points to marketplace-personalisation when the bottleneck is personalisation-specific.

Read the playbooks first when the task is "design a new search and recommender project" or "this retrieval system needs to get better". Read individual rules when a specific question arises during implementation or review.

How to Use

• Read `references/_sections.md` for category structure and cascade rationale.
• Read `gotchas.md` for diagnostic lessons accumulated from prior incidents.
• Read `references/playbooks/planning.md` to plan a new system.
• Read `references/playbooks/improving.md` to diagnose an existing one.
• Read individual rule files when a specific task matches the rule title.
• Use `assets/templates/_template.md` to author new rules as the skill grows.

Related Skills

• `marketplace-personalisation` — The companion skill covering AWS Personalize implementation, impression tracking, schema design, two-sided matching, feedback loops, and the personalisation-specific diagnostic playbook. Hand off to this skill when the diagnostic identifies a personalisation-specific bottleneck.

Reference Files

Two-Sided Search and Recsys Planning

Version 0.1.0 Marketplace Engineering April 2026

Note:

This document is mainly for agents and LLMs to follow when maintaining,

generating, or refactoring codebases. Humans may also find it useful,

but guidance here is optimized for automation and consistency by AI-assisted workflows.

---

Abstract

Planning, design and diagnostic guide for search and recommendation systems in two-sided trust marketplaces built on OpenSearch. Contains 57 rules across 10 categories ordered by cascade impact on the retrieval lifecycle — from user-intent framing and product-surface architecture through OpenSearch index and query design, CDC ingestion, embedding-model selection, retrieval strategy, ranking, search-plus-recs blending, measurement, PII scrubbing and the instrumentation-and-dashboard layer that turns measurement into ongoing decision making. Includes two playbooks for planning a new retrieval system from scratch and diagnosing an existing one, plus explicit living-artefact conventions (decisions log, golden set, gotchas) so context accumulates across sessions, releases, and team changes. Functions as the precursor to the companion marketplace-personalisation skill with an explicit hand-off rule.

---

Table of Contents

1. Problem Framing and User Intent — CRITICAL

• 1.1 Audit Live Query Logs Before Designing — CRITICAL (prevents designing for imagined users)
• 1.2 Distinguish Transactional from Exploratory Intent — CRITICAL (prevents conversion loss on transactional sessions)
• 1.3 Map Queries to Intent Classes Before Touching Retrieval — CRITICAL (prevents retrieval-strategy mismatch with user goal)
• 1.4 Reject the One-Search-For-Everything Temptation — CRITICAL (prevents system-wide compromise)
• 1.5 Separate Known-Item Search from Discovery — CRITICAL (prevents recall loss on known-item queries)
• 1.6 Treat No-Search as a First-Class Choice — CRITICAL (prevents forcing retrieval where browse is correct)

2. Surface Taxonomy and Architecture — CRITICAL

• 2.1 Avoid Mono-Stack Retrieval — CRITICAL (prevents single-point-of-failure in retrieval)
• 2.2 Declare a Fallback Owner per Surface at Architecture Time — CRITICAL (prevents fallback gaps on new surfaces)
• 2.3 Design for Cold Start from Day One — CRITICAL (prevents new-listing discovery failure)
• 2.4 Map Each Surface to a Retrieval Primitive Deliberately — CRITICAL (prevents architectural drift across surfaces)
• 2.5 Route Surfaces to Search, Recs, or Hybrid Deliberately — CRITICAL (prevents ad-hoc routing drift)
• 2.6 Split Candidate Generation from Ranking — CRITICAL (enables independent tuning of retrieval and ranking)

3. Index Design and Mapping — HIGH

• 3.1 Design Mappings Conservatively Because Reindex Is Expensive — HIGH (avoids full reindex downtime)
• 3.2 Match Index-Time and Query-Time Analyzers — HIGH (prevents tokenisation mismatch at query time)
• 3.3 Separate Searchable Fields from Display Fields — HIGH (reduces index storage and query cost)
• 3.4 Stream Listing Updates via CDC, Not Periodic Full Re-Import — HIGH (reduces index staleness from hours to seconds)
• 3.5 Use Index Templates to Enforce Consistency — HIGH (prevents mapping drift across indices)
• 3.6 Use keyword and text as Multi-Fields — HIGH (enables exact match and full-text on one field)
• 3.7 Use Language Analyzers for Language-Sensitive Fields — HIGH (enables language-aware stemming and stopwords)

4. Planning and Improvement Methodology — HIGH

• 4.1 Audit Before You Build — Gate Work on Instrumentation Readiness — HIGH (prevents building on broken telemetry)
• 4.2 Build a Golden Query Set as the First Artefact — HIGH (enables offline regression detection)
• 4.3 Find the Bottleneck Before Optimising — HIGH (prevents work on non-bottleneck layers)
• 4.4 Freeze and Version the Golden Set per Evaluation Cycle — HIGH (enables comparable evaluations across releases)
• 4.5 Hand Off to the Personalisation Skill When the Bottleneck Is Personalisation — HIGH (prevents duplicated planning effort)
• 4.6 Maintain a Decisions Log as Living Context — HIGH (prevents lost context across team changes)

5. Query Understanding — MEDIUM-HIGH

• 5.1 Build Autocomplete on a Separate Index — MEDIUM-HIGH (prevents autocomplete latency from blocking main search)
• 5.2 Classify Queries Before Routing — MEDIUM-HIGH (enables intent-aware routing)
• 5.3 Curate Synonyms by Domain Intent — MEDIUM-HIGH (enables domain-specific recall)
• 5.4 Normalise Queries Before Anything Else — MEDIUM-HIGH (prevents unicode and whitespace misses)
• 5.5 Use Fuzzy Matching for Typo Tolerance — MEDIUM-HIGH (prevents recall loss on typos)
• 5.6 Use Language Analyzers for Stemming and Stopwords — MEDIUM-HIGH (enables stemming and stopword removal)

6. Retrieval Strategy — MEDIUM-HIGH

• 6.1 Choose the Embedding Model Deliberately Before Hybrid Search — MEDIUM-HIGH (avoids full re-embedding on model change)
• 6.2 Combine BM25 and KNN via Hybrid Search — MEDIUM-HIGH (enables semantic plus lexical recall)
• 6.3 Paginate with search_after for Deep Result Sets — MEDIUM-HIGH (prevents deep-pagination memory cost)
• 6.4 Run Expensive Signals in rescore — MEDIUM-HIGH (reduces scoring cost on full candidate set)
• 6.5 Use bool Structure Deliberately — MEDIUM-HIGH (prevents ambiguous clause semantics)
• 6.6 Use filter Clauses for Exact Matches — MEDIUM-HIGH (enables query result caching)

7. Relevance and Ranking — MEDIUM-HIGH

• 7.1 Apply Diversity at Rank Time, Not Retrieval — MEDIUM-HIGH (preserves retrieval recall for diversity)
• 7.2 Deploy Learning to Rank Only After Golden Set and Judgments Exist — MEDIUM-HIGH (prevents premature LTR complexity)
• 7.3 Normalise Scores Across Retrieval Primitives — MEDIUM-HIGH (enables comparable hybrid ranking)
• 7.4 Tune BM25 Parameters Last, Not First — MEDIUM-HIGH (prevents premature micro-optimisation)
• 7.5 Use function_score for Business Signals — MEDIUM-HIGH (enables explainable business ranking)

8. Search and Recommender Blending — MEDIUM

• 8.1 Combine Search and Personalisation Scores with Normalised Weights — MEDIUM (enables comparable hybrid ranking)
• 8.2 Keep Hybrid Blending Explainable — MEDIUM (enables blending debugging and tuning)
• 8.3 Never Return Zero Results — MEDIUM (prevents dead-end sessions)
• 8.4 Use Search Alone When Intent Is Specific — MEDIUM (prevents noise on precision-oriented queries)

9. Measurement and Experimentation — MEDIUM

• 9.1 Define Session Success per Surface — MEDIUM (enables surface-specific measurement)
• 9.2 Run Interleaving as a Cheap A/B Proxy — MEDIUM (reduces experiment sample-size cost)
• 9.3 Track NDCG, MRR and Zero-Result Rate — MEDIUM (enables ranking-quality measurement)
• 9.4 Track Reformulation Rate as a Failure Signal — MEDIUM (enables implicit query-failure detection)
• 9.5 Use Click Models for Implicit Relevance Judgments — MEDIUM (enables scalable judgment collection)

10. Instrumentation, Dashboards and Decision Triggers — MEDIUM

• 10.1 Alert on Decision-Triggering Metrics, Not Just Error Rates — MEDIUM (enables early quality regression detection)
• 10.2 Build a Search Health Dashboard with Threshold Lines — MEDIUM (enables at-a-glance quality monitoring)
• 10.3 Log Every Query with Full Context for Counterfactual Replay — MEDIUM (enables post-hoc query debugging)
• 10.4 Run a Weekly Search-Quality Review Ritual — MEDIUM (enables calendar-driven decision making)
• 10.5 Scrub PII from Query Logs Before Warehouse Ingestion — MEDIUM (prevents GDPR exposure in analytics)
• 10.6 Track Ranking Stability as a Churn Metric — MEDIUM (enables leading-indicator detection)

---

References

1. https://docs.opensearch.org/latest/query-dsl/compound/bool/ 2. https://docs.opensearch.org/latest/query-dsl/query-filter-context/ 3. https://docs.opensearch.org/latest/query-dsl/rescore/ 4. https://docs.opensearch.org/latest/analyzers/ 5. https://docs.opensearch.org/latest/analyzers/custom-analyzer/ 6. https://docs.opensearch.org/latest/analyzers/language-analyzers/index/ 7. https://docs.opensearch.org/latest/analyzers/language-analyzers/english/ 8. https://docs.opensearch.org/latest/vector-search/ai-search/hybrid-search/index/ 9. https://opensearch.org/blog/building-effective-hybrid-search-in-opensearch-techniques-and-best-practices/ 10. https://opensearch.org/blog/multilingual-search/ 11. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/learning-to-rank.html 12. https://aws.amazon.com/blogs/big-data/hybrid-search-with-amazon-opensearch-service/ 13. https://www.manning.com/books/relevant-search 14. https://opensourceconnections.com/blog/2019/12/11/what-is-a-relevant-search-result/ 15. https://eugeneyan.com/writing/recsys-llm/ 16. https://www.kdd.org/kdd2018/accepted-papers/view/real-time-personalization-using-embeddings-for-search-ranking-at-airbnb 17. https://pubsonline.informs.org/doi/10.1287/mksc.2022.0238 18. https://www.pinecone.io/learn/offline-evaluation/ 19. https://developers.google.com/machine-learning/guides/rules-of-ml 20. https://careersatdoordash.com/blog/homepage-recommendation-with-exploitation-and-exploration/ 21. https://docs.opensearch.org/latest/field-types/ 22. https://docs.opensearch.org/latest/search-plugins/searching-data/paginate/ 23. https://sre.google/sre-book/embracing-risk/ 24. https://sbert.net/examples/sentence_transformer/domain_adaptation/README.html 25. https://eugeneyan.com/writing/system-design-for-discovery/ 26. https://lantern.splunk.com/Security/UCE/Foundational_Visibility/Compliance/Detecting_Personally_Identifiable_Information_(PII)_in_log_data_for_GDPR_compliance_in_log_data_for_GDPR_compliance)

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

{Action-Oriented Rule Title in Title Case}

{One-to-three sentences explaining WHY this rule matters. Focus on the cascade effect — what goes wrong when the rule is not followed, and how the problem propagates to every downstream stage. This is the most important part of the rule; LLMs generalise better from understood reasoning than from dictation. Do not say "always do X" — explain what happens when you do not, in concrete terms the model can internalise.}

Incorrect ({specific failure mode in 3-6 words}):

```{language}

Production-realistic bad code — not a strawman.

Keep it under 20 lines. Use realistic marketplace names like seeker, provider,

listing, booking, request_id, trust_score — never foo, bar, MyComponent, doSomething.

**Correct ({specific benefit in 3-6 words}):**

Production-realistic good code that differs minimally from the incorrect version —

only the key insight changes. Same variable names, same structure, different

behaviour where it matters. Under 20 lines.

Reference: [{Authoritative Source Title}]({https://url-to-primary-source})

## Authoring Checklist

Before saving, verify:

- [ ] Frontmatter has `title`, `impact`, `impactDescription`, `tags`
- [ ] First tag matches the category prefix from `_sections.md`
- [ ] Impact level is consistent with sibling rules in the same category
- [ ] Impact description is quantified (contains a number, verb like `prevents` / `reduces` / `enables`, or `O(n)` notation)
- [ ] The H2 title matches the frontmatter title exactly
- [ ] Both code blocks have a language specifier (```json, ```python, ```typescript)
- [ ] Both annotations `(failure mode)` and `(benefit)` are specific, not `(bad)` or `(good)`
- [ ] No vague language: `might want`, `perhaps`, `maybe`, `it is recommended`
- [ ] No marketing language: `powerful`, `magic`, `seamless`, `blazing fast`
- [ ] No generic names: `foo`, `bar`, `MyComponent`, `doSomething`, `processData`
- [ ] Reference is from OpenSearch maintainers, canonical search text, peer-reviewed research, or production engineering blog
- [ ] Rule validates with `node scripts/validate-skill.js {skill-dir}` with no new errors

Gotchas

Living log of diagnostic lessons accumulated from marketplace search and recsys work. Every entry is dated, describes a concrete surprise or failure mode, and records the resolution. Append new entries at the top. Never delete entries — older lessons still carry context even when the specific code they refer to has changed.

Format

### Short descriptive title of the surprise
Date: YYYY-MM-DD
Context: what the team was working on
Symptom: what the observable failure mode was
Root cause: what was actually wrong
Resolution: how the team fixed it
Lesson: the generalisable takeaway

---

Example seed: analyzer mismatch caused silent zero-result spike

Date: 2026-04-11 Context: Worked example illustrating the gotchas convention — replace when a real one is captured. Symptom: Zero-result rate jumped from 4% to 14% over a weekend on queries containing common plurals ("sitters", "walks", "stays"). Root cause: A well-intentioned mapping change dropped the english analyzer from the title field, reverting to the standard analyzer which does not stem. Plurals stopped matching singular-form listing titles. Resolution: Reverted the mapping change, scheduled a reindex, and added a post-release RBO churn check that would have caught the silent drift within an hour of deploy (see monitor-track-ranking-stability-churn). Lesson: Analyzer changes on text fields are effectively ranking changes and belong in the decisions log with a golden-set offline evaluation before deploy.

Marketplace Search and Recsys Planning Skill

Planning, design and diagnostic best-practices skill for search and recommendation systems in two-sided trust marketplaces built on OpenSearch. Functions as the precursor to the companion marketplace-personalisation skill.

Overview

This skill is a distillation of authoritative guidance from OpenSearch documentation, canonical search-relevance engineering texts (Turnbull's Relevant Search), academic work on two-sided marketplace recommendation, Google's Rules of Machine Learning, and production engineering blogs from two-sided marketplace companies. It contains 57 rules across 10 categories, ordered by cascade impact on the retrieval lifecycle, plus two playbooks and explicit living-artefact conventions for evolving context.

The skill treats the marketplace system as an evolving artefact — a gotchas log, a decisions log, and a versioned golden set carry context across sessions, releases, and team changes.

Structure

marketplace-search-recsys-planning/
├── SKILL.md                    # Entry point with category index and quick reference
├── AGENTS.md                   # Compiled navigation document (built by script)
├── metadata.json               # Version, discipline, authoritative references
├── README.md                   # This file
├── gotchas.md                  # Living diagnostic lessons (append-only)
├── references/
│   ├── _sections.md            # Category definitions and impact ordering
│   ├── intent-*.md             # Problem Framing and User Intent (6 rules)
│   ├── arch-*.md               # Surface Taxonomy and Architecture (6 rules)
│   ├── index-*.md              # Index Design and Mapping (7 rules)
│   ├── plan-*.md               # Planning and Improvement Methodology (6 rules)
│   ├── query-*.md              # Query Understanding (6 rules)
│   ├── retrieve-*.md           # Retrieval Strategy (6 rules)
│   ├── rank-*.md               # Relevance and Ranking (5 rules)
│   ├── blend-*.md              # Search and Recommender Blending (4 rules)
│   ├── measure-*.md            # Measurement and Experimentation (5 rules)
│   ├── monitor-*.md            # Instrumentation, Dashboards and Decision Triggers (6 rules)
│   └── playbooks/
│       ├── planning.md         # Plan a new retrieval system from scratch
│       └── improving.md        # Diagnose and improve an existing one
└── assets/
    └── templates/
        └── _template.md        # Template for authoring new rules

Getting Started

From the repo root, install plugin dependencies and run the skill validator:

pnpm install
pnpm build
pnpm validate

The validator runs structural and substance checks against the skill:

node scripts/validate-skill.js skills/.experimental/marketplace-search-recsys-planning

Build the compiled navigation document:

node scripts/build-agents-md.js skills/.experimental/marketplace-search-recsys-planning

Creating a New Rule

Rules go in references/ with a filename of the form {prefix}-{slug}.md, where {prefix} matches an existing category in references/_sections.md. Copy assets/templates/_template.md as a starting point and fill in the frontmatter and body.

A rule must include:

• YAML frontmatter: title, impact, impactDescription, tags (first tag is the prefix)
• One-to-three sentence explanation of why the rule matters and its cascade effect
• An **Incorrect (specific description):** code block with production-realistic code
• A **Correct (specific description):** code block that differs minimally from the incorrect
• A Reference: line linking to an authoritative source (OpenSearch docs, canonical text, or engineering blog)

Run pnpm validate after adding or editing rules.

Rule File Structure

Each rule has a strict structure enforced by the validator:

---
title: Use Filter Clauses for Exact Matches
impact: MEDIUM-HIGH
impactDescription: enables query result caching
tags: retrieve, filter, caching
---

## Use Filter Clauses for Exact Matches

Explanation paragraph — why the rule matters, what goes wrong without it,
and how the cascade effect plays out downstream.

**Incorrect (concrete failure mode):**

{ "production-realistic bad example": "here" }

**Correct (concrete solution):**

{ "production-realistic good example": "here" }

Reference: [OpenSearch Documentation — Query and Filter Context](https://docs.opensearch.org/latest/query-dsl/query-filter-context/)

File Naming Convention

• Skill directory: kebab-case matching the skill name (marketplace-search-recsys-planning)
• Rule files: {category-prefix}-{slug}.md with kebab-case slugs (intent-map-queries-to-intent-classes.md)
• Playbook files: references/playbooks/{name}.md
• Templates: assets/templates/_template.md (underscore prefix excludes from rule listings)
• Category prefixes are 3-8 lowercase letters and defined once in _sections.md

Impact Levels

Categories and rules use six impact levels ordered from highest to lowest cascade impact:

Target distribution for a 40-60 rule distillation: 2-3 CRITICAL categories, 2-4 HIGH, the rest MEDIUM or lower. This skill has 2 CRITICAL, 2 HIGH, 3 MEDIUM-HIGH and 3 MEDIUM categories with an evenly spread four-tier rule distribution.

Scripts

The dev-skill plugin provides two scripts used by this skill:

• scripts/validate-skill.js — runs structural and substance validation (required before shipping)
• scripts/build-agents-md.js — compiles the navigation document (never write AGENTS.md manually)

Example invocations:

node scripts/validate-skill.js skills/.experimental/marketplace-search-recsys-planning
node scripts/validate-skill.js skills/.experimental/marketplace-search-recsys-planning --sections-only
node scripts/build-agents-md.js skills/.experimental/marketplace-search-recsys-planning

Contributing

• New rules must follow the structure above and pass pnpm validate with zero errors
• Every rule must have incorrect and correct examples with specific annotations
• References must be from OpenSearch maintainers, canonical search texts, peer-reviewed research, or engineering blogs with data
• Avoid hedging language (might, perhaps, it is recommended) — use imperative form
• Quantify impact where possible (10-15%, 200ms, prevents stale closures, O(n) to O(1))
• Playbooks in references/playbooks/ compose rules into end-to-end workflows
• Update gotchas.md when a new diagnostic lesson is learned
• Never edit AGENTS.md manually — it is regenerated by build-agents-md.js

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

Categories are ordered by cascade impact on the retrieval lifecycle of a two-sided marketplace. Intent misunderstanding poisons architecture; wrong architecture poisons index shape; wrong index shape poisons retrieval forever until a reindex; and every downstream layer inherits the upstream error. Planning and monitoring are meta-layers that observe the cascade and drive the iteration cycle — they are deliberately positioned to feed back into the upstream layers as the system evolves.

---

1. Problem Framing and User Intent (intent)

Impact: CRITICAL Description: Misunderstanding what users are actually doing — searching for a specific result, browsing for inspiration, or asking the system to suggest — poisons every downstream decision and produces a system that is technically correct and operationally wrong.

2. Surface Taxonomy and Architecture (arch)

Impact: CRITICAL Description: Mapping product surfaces to the right retrieval primitive (search, recommender, hybrid) determines the shape of the entire system, and choosing the wrong primitive per surface forces expensive architectural rework months later.

3. Index Design and Mapping (index)

Impact: HIGH Description: OpenSearch mappings are effectively immutable without a full reindex, so field types, analyzers and multi-field layouts are lifetime commitments that constrain every query and ranking choice that follows.

4. Planning and Improvement Methodology (plan)

Impact: HIGH Description: Retrieval work without a structured plan, a golden query set, a decisions log and a bottleneck analysis drifts into algorithm tuning when the real problem is instrumentation or coverage — applying theory of constraints prevents months of wasted effort.

5. Query Understanding (query)

Impact: MEDIUM-HIGH Description: Query parsing, normalization, analyzer choice, synonym management, typo tolerance and intent classification turn raw user input into a structured retrieval request, and each layer directly affects recall, precision and reformulation rate.

6. Retrieval Strategy (retrieve)

Impact: MEDIUM-HIGH Description: OpenSearch query DSL structure — filter versus must clauses, bool composition, hybrid BM25 plus KNN, rescoring — governs how candidates are generated from the index, and each stage trades off latency, relevance and cache efficiency.

7. Relevance and Ranking (rank)

Impact: MEDIUM-HIGH Description: Scoring candidates by BM25 parameters, function_score business signals, rescoring pipelines, and Learning to Rank models determines the ordering users actually see, but only if the upstream retrieval and index layers are correct.

8. Search and Recommender Blending (blend)

Impact: MEDIUM Description: Deciding when to use search alone, recommendations alone, or a blended response — with explicit normalization and zero-result fallbacks — protects the marketplace from dead-end sessions and keeps cold-start cohorts productive.

9. Measurement and Experimentation (measure)

Impact: MEDIUM Description: Defining the right metrics — NDCG, MRR, zero-result rate, session success, reformulation rate — and the right experimentation primitives — golden sets, offline judgments, interleaving, online A/B tests — turns "does it feel better" into "did it actually improve".

10. Instrumentation, Dashboards and Decision Triggers (monitor)

Impact: MEDIUM Description: Raw query logs, decision-triggering dashboards, threshold alerts, ranking churn tracking and a weekly quality review ritual are the instrumentation that converts measurement into ongoing decision-making, and without them the team operates on intuition instead of evidence.

Avoid Mono-Stack Retrieval

Putting every surface behind a single OpenSearch cluster with a single query template creates a brittle system: one cluster outage, one schema migration error, one noisy-neighbour shard and the entire marketplace goes blank. Diversifying retrieval — lexical search on OpenSearch, a recommender on AWS Personalize, a curated content store for editorial, a simple popularity fallback — means no single dependency can take down the whole experience. The cost is operational complexity; the benefit is a system that degrades gracefully instead of failing catastrophically.

Incorrect (all surfaces behind the same OpenSearch cluster with no fallback):

def unified_retrieval(surface: str, seeker: Seeker, query: SearchQuery | None) -> list[Listing]:
    body = build_body_for(surface, seeker, query)
    response = opensearch.search(index="listings", body=body)
    return hydrate(response["hits"]["hits"])

Correct (each surface has a primary plus a degraded-mode fallback):

def homefeed(seeker: Seeker) -> list[Listing]:
    try:
        return personalize_recommender(seeker, limit=24)
    except PersonalizeUnavailable:
        logger.warning("personalize_down, falling back to opensearch popularity")
        return opensearch_popularity_by_region(seeker.region, limit=24)

def search_results(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    try:
        return opensearch_search(query, seeker)
    except OpenSearchUnavailable:
        logger.warning("opensearch_down, falling back to curated")
        return curated_top_by_region(query.region, limit=24)

Reference: Google SRE Book — Embracing Risk

Design for Cold Start from Day One

A marketplace accumulates new listings and new seekers continuously — cold start is not a bootstrap problem that goes away, it is a permanent operating condition. An architecture that relies entirely on historical interaction data produces a system that works well after twelve months and fails for every new provider and every new seeker forever. Wire cold-start strategies (metadata-based retrieval, popularity by segment, onboarding intent capture, exploration slots) into the architecture from the first design meeting, not as a retrofit when coverage collapses.

Incorrect (interaction-only retrieval, new listings invisible):

def homefeed(seeker: Seeker) -> list[Listing]:
    hits = opensearch.search(
        index="listings",
        body={
            "query": {
                "function_score": {
                    "query": {"term": {"region": seeker.region}},
                    "script_score": {"script": "doc['booking_count'].value"},
                },
            },
            "size": 24,
        },
    )["hits"]["hits"]
    return hydrate(hits)

Correct (cold-start reserved slots, segmented popularity for new seekers):

def homefeed(seeker: Seeker) -> list[Listing]:
    if seeker.lifetime_events < 5:
        return segment_popularity(seeker.region, seeker.declared_species, limit=24)

    warm = ranked_for_seeker(seeker, limit=20)
    fresh = newly_created_in_region(seeker.region, days=14, limit=4)
    return interleave(warm, fresh, fresh_ratio=0.2)

Reference: Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical Approach

Declare a Fallback Owner per Surface at Architecture Time

A zero-result response is a dead-end session, and the only thing worse than serving one is discovering six months later that a surface the team shipped has no fallback strategy at all because nobody was asked to own it. The architectural requirement — not the mechanics — is that every surface in the routing table declares who owns the fallback, what the fallback strategy is, and what the target non-empty-rate SLO is. The mechanics of the cascade belong to the blending rules (see `blend-never-return-zero-results`); this rule is about making the commitment visible in the routing table so it cannot be forgotten.

Incorrect (routing table has no fallback column, new surfaces ship without one):

SURFACE_ROUTES = {
    "homefeed": SurfaceRoute(primitive="recommender", owner="personalisation-team"),
    "search_results": SurfaceRoute(primitive="lexical_search", owner="search-team"),
    "item_page_related": SurfaceRoute(primitive="sims", owner="personalisation-team"),
}

Correct (routing table requires explicit fallback strategy, owner, SLO):

SURFACE_ROUTES = {
    "homefeed": SurfaceRoute(
        primitive="recommender",
        owner="personalisation-team",
        fallback_strategy="segment_popularity",
        fallback_owner="search-team",
        non_empty_slo=0.999,
    ),
    "search_results": SurfaceRoute(
        primitive="lexical_search",
        owner="search-team",
        fallback_strategy="relaxed_then_recommender_then_curated",
        fallback_owner="search-team",
        non_empty_slo=0.995,
    ),
    "item_page_related": SurfaceRoute(
        primitive="sims",
        owner="personalisation-team",
        fallback_strategy="same_category_popularity",
        fallback_owner="personalisation-team",
        non_empty_slo=0.99,
    ),
}

Reference: Google SRE Book — Embracing Risk

Map Each Surface to a Retrieval Primitive Deliberately

Every product surface that shows listings — homepage, search results, item-page related, category landing, saved-for-later, notification carousel — needs an explicit mapping to one retrieval primitive: lexical search, recommendations, curated/editorial, or hybrid blend. Without that mapping as a documented artefact, surfaces accrete retrieval code incrementally, each developer picks what they know, and the system grows inconsistent. A one-page surface → primitive table is the single most useful planning artefact in marketplace retrieval work.

Incorrect (each surface implements whatever the engineer picked that sprint):

def homefeed(seeker): return opensearch_random(seeker)
def search_results(query, seeker): return opensearch_match(query)
def item_page_related(listing): return cache_get(f"related:{listing.id}")
def category_landing(cat): return db.select_by_category(cat)

Correct (explicit per-surface mapping recorded as configuration):

SURFACE_PRIMITIVES = {
    "homefeed": Primitive.RECOMMENDER,
    "search_results": Primitive.LEXICAL_SEARCH,
    "item_page_related": Primitive.SIMS,
    "category_landing": Primitive.CURATED_PLUS_RANKING,
    "saved_for_later": Primitive.CURATED,
    "notification_carousel": Primitive.HYBRID,
}

def route(surface: str, seeker: Seeker, query: SearchQuery | None) -> list[Listing]:
    handler = PRIMITIVE_HANDLERS[SURFACE_PRIMITIVES[surface]]
    return handler(seeker=seeker, query=query)

Reference: Eugene Yan — Improving Recommendation Systems and Search

Route Surfaces to Search, Recs, or Hybrid Deliberately

A surface decision — "homefeed uses recommendations, search results use lexical, item-page related uses SIMS, category landing uses hybrid" — must be recorded in a single source of truth that the team reviews when adding new surfaces. Without that record, each new surface is routed based on whoever builds it, assumptions drift, and six months later nobody can explain why category pages and homepage use different retrieval primitives. The documented record becomes a decision log that carries context across team changes.

Incorrect (routing scattered across service files, no single source of truth):

def homefeed_service(seeker): return personalize.recommend(seeker)

def search_service(query, seeker): return opensearch.search(query)

def item_page_related(listing): return redis.get(f"sims:{listing.id}")

def category_page(cat): return db.select_by_cat(cat)

Correct (surface routing table in a single config with rationale and owner):

SURFACE_ROUTES: dict[str, SurfaceRoute] = {
    "homefeed": SurfaceRoute(
        primitive="recommender",
        owner="personalisation-team",
        reason="Warm seekers benefit from personalised ordering; cold seekers fall back to segmented popularity.",
    ),
    "search_results": SurfaceRoute(
        primitive="lexical_search",
        owner="search-team",
        reason="Transactional queries with hard filters; precision matters more than diversity.",
    ),
    "item_page_related": SurfaceRoute(
        primitive="sims",
        owner="personalisation-team",
        reason="Seed item is the signal; user history is not available on anonymous item pages.",
    ),
    "category_landing": SurfaceRoute(
        primitive="hybrid",
        owner="search-team",
        reason="Category is the retrieval filter; ranking uses popularity plus trust signals.",
    ),
}

Reference: Eugene Yan — Improving Recommendation Systems and Search

Split Candidate Generation from Ranking

A retrieval pipeline that combines candidate generation and ranking in a single OpenSearch query locks the two concerns together — changing the ranker forces re-tuning the candidate pool, and changing the feasible-set rules forces re-tuning the ranker. The industry standard is a two-stage pipeline: retrieval returns 100-500 feasible candidates, ranking re-orders the top-K. Each stage is then tunable, testable and replaceable without touching the other. Airbnb, Pinterest, DoorDash, Etsy — all converged on this structure independently.

Incorrect (single query mixes filter, candidate generation, and scoring):

def search(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    body = {
        "query": {
            "function_score": {
                "query": {
                    "bool": {
                        "must": [{"multi_match": {"query": query.text, "fields": ["title", "description"]}}],
                        "filter": [{"term": {"region": query.region}}],
                    },
                },
                "functions": [{"field_value_factor": {"field": "trust_score"}}],
            },
        },
        "size": 24,
    }
    return opensearch.search(index="listings", body=body)["hits"]["hits"]

Correct (stage 1 retrieves feasible candidates; stage 2 re-ranks top-K):

def search(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    candidates = retrieve_candidates(query, seeker, limit=300)
    if not candidates:
        return fallback_ranker(query, seeker)
    ranked = rerank(candidates, seeker, query)
    return ranked[:24]

def retrieve_candidates(query: SearchQuery, seeker: Seeker, limit: int) -> list[Listing]:
    body = {
        "query": {
            "bool": {
                "must": [{"multi_match": {"query": query.text, "fields": ["title", "description"]}}],
                "filter": [{"term": {"region": query.region}}],
            },
        },
        "size": limit,
    }
    return hydrate(opensearch.search(index="listings", body=body)["hits"]["hits"])

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking (KDD 2018)

Combine Search and Personalisation Scores with Normalised Weights

When a surface blends search and personalisation, the two must be combined as a single ordered list — but their raw scores are on incomparable scales. Normalise each side (min-max within the batch, or rank-based) to 0-1 before computing a weighted sum, and commit the weights to config where they can be tuned via A/B testing. The blending weights become an explicit hyperparameter, not a hardcoded constant, so they can evolve as the system learns.

Incorrect (raw scores summed directly, personalisation dominates):

def blend(search_hits: list, rec_hits: list) -> list:
    scored = {}
    for hit in search_hits:
        scored[hit.id] = hit.score
    for hit in rec_hits:
        scored[hit.id] = scored.get(hit.id, 0) + hit.score
    return sorted(scored.items(), key=lambda kv: -kv[1])

Correct (min-max normalise each side, blend with configurable weight):

def blend(search_hits: list, rec_hits: list, search_weight: float = 0.6) -> list:
    def normalise(hits):
        if not hits:
            return {}
        lo = min(h.score for h in hits)
        hi = max(h.score for h in hits)
        rng = hi - lo
        return {h.id: (h.score - lo) / rng if rng > 0 else 0.0 for h in hits}

    search_norm = normalise(search_hits)
    rec_norm = normalise(rec_hits)

    candidate_ids = set(search_norm) | set(rec_norm)
    scored = {
        listing_id: (
            search_weight * search_norm.get(listing_id, 0.0)
            + (1 - search_weight) * rec_norm.get(listing_id, 0.0)
        )
        for listing_id in candidate_ids
    }
    return sorted(scored.items(), key=lambda kv: -kv[1])

Reference: OpenSearch Blog — Building Effective Hybrid Search

Keep Hybrid Blending Explainable

A blended response where the top-3 listings are from different retrieval primitives needs to be debuggable: which primitive contributed each listing, what was its raw score, what was its normalised score, what was the final weighted score. Attaching a small trace object to each result that records the primitive source, the raw score, and the final weighted contribution is cheap (single-digit extra bytes per listing) and saves hours of blending debugging later. The trace is also invaluable for ranking tuners during relevance work.

Incorrect (blended response returns only the final ordered list):

def blend_response(search_hits, rec_hits) -> list[Listing]:
    return blended_sort(search_hits, rec_hits)[:24]

Correct (each result carries a blending trace for debugging and tuning):

def blend_response(search_hits, rec_hits) -> list[BlendedListing]:
    blended = blended_sort(search_hits, rec_hits)
    return [
        BlendedListing(
            listing=b.listing,
            final_score=b.final_score,
            trace=BlendTrace(
                search_raw=b.search_raw_score,
                search_norm=b.search_normalised,
                rec_raw=b.rec_raw_score,
                rec_norm=b.rec_normalised,
                search_weight=b.search_weight,
                primitive_source=b.primary_primitive,
            ),
        )
        for b in blended[:24]
    ]

Reference: Eugene Yan — Improving Recommendation Systems and Search

Never Return Zero Results

Zero results is never the right response for a discovery-style surface — it ends the session, wastes the acquisition cost that brought the seeker to the page, and produces no telemetry signal the team can act on. The blending layer's final responsibility is to guarantee a non-empty response: cascade through search → relaxed search → recommender → segment popularity → global popularity → curated fallback, and the only acceptable outcome is that something gets returned with a strategy label telling the UI what to show. Zero results should be a monitored incident, not a normal state.

Incorrect (empty list returned, session ends):

def final_response(query: ClassifiedQuery, seeker: Seeker) -> SearchResponse:
    hits = opensearch_search(query, seeker)
    return SearchResponse(listings=hits)

Correct (guaranteed non-empty cascade through five fallback strategies):

def final_response(query: ClassifiedQuery, seeker: Seeker) -> SearchResponse:
    strategies = [
        ("strict_search", lambda: opensearch_search(query, seeker)),
        ("relaxed_search", lambda: opensearch_search(relax(query), seeker)),
        ("recommender", lambda: personalize_recommender(seeker, limit=24)),
        ("segment_popularity", lambda: segment_popularity(seeker.region, seeker.declared_species)),
        ("curated_fallback", lambda: curated_top_by_region(seeker.region)),
    ]
    for strategy_name, strategy_fn in strategies:
        hits = strategy_fn()
        if hits:
            return SearchResponse(listings=hits, strategy=strategy_name)

    raise ShouldNeverHappen("All fallback strategies returned empty — investigate inventory")

Reference: Doug Turnbull & John Berryman — Relevant Search

Use Search Alone When Intent Is Specific

A seeker who types "sarah the dog sitter chiswick" has an exact target — injecting recommendations into that response shows unrelated listings ranked high by personalisation, which dilutes precision and frustrates the user. For queries with specific entities (named provider, exact location, exact species, specific date range), lexical search alone is the right answer, and the blending layer should recognise this and disable recommendation injection. Intent classification (from the query-classify-before-routing rule) provides the signal.

Incorrect (recommendations always blended into search results):

def search_response(query: ClassifiedQuery, seeker: Seeker) -> list[Listing]:
    search_hits = opensearch_search(query, seeker)
    rec_hits = personalize_recommender(seeker)
    return interleave(search_hits, rec_hits, rec_ratio=0.3)[:24]

Correct (specific intent bypasses the recommender blend):

def search_response(query: ClassifiedQuery, seeker: Seeker) -> list[Listing]:
    if query.intent == Intent.NAVIGATIONAL or query.has_named_entity:
        return opensearch_search(query, seeker)[:24]

    search_hits = opensearch_search(query, seeker)
    rec_hits = personalize_recommender(seeker)
    return interleave(search_hits, rec_hits, rec_ratio=0.3)[:24]

Reference: Eugene Yan — Improving Recommendation Systems and Search

Design Mappings Conservatively Because Reindex Is Expensive

OpenSearch mappings are additive — you can add new fields but cannot change the type, analyzer or multi-field layout of an existing field without reindexing the entire dataset. For a marketplace with millions of listings and an append-only interaction log, reindexing is a days-long operation with alias cut-over, dual-writes and rollback planning. The mapping decisions made in week 1 constrain every query shape for the life of the project, so treat each field as a lifetime commitment — add only what is stable, predictive and worth the migration cost to ever change.

Incorrect (speculative fields with ambiguous types that will churn in a month):

{
  "mappings": {
    "properties": {
      "listing_id": { "type": "keyword" },
      "title": { "type": "text" },
      "metadata": { "type": "object", "enabled": true },
      "tags": { "type": "text" },
      "extra": { "type": "object", "dynamic": true }
    }
  }
}

Correct (explicit, typed, stable fields with multi-field analyzer layout):

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "listing_id": { "type": "keyword" },
      "title": {
        "type": "text",
        "analyzer": "listing_text_en",
        "fields": { "raw": { "type": "keyword" } }
      },
      "region": { "type": "keyword" },
      "price_tier": { "type": "keyword" },
      "accepts_species": { "type": "keyword" },
      "trust_score": { "type": "float" },
      "created_at": { "type": "date" }
    }
  }
}

Reference: OpenSearch Documentation — Supported Field Types

Match Index-Time and Query-Time Analyzers

A field indexed with one analyzer and queried with another produces silent recall failures — tokens the indexer produced do not match tokens the query produced, and the listing is invisible to the seeker even though a human would obviously match them. The safe default is to use the same analyzer at both times by declaring it on the field; OpenSearch then uses it automatically for queries. Overriding the search-time analyzer is an expert move reserved for specific cases like autocomplete where index-time shingling and query-time prefix matching are intentional.

Incorrect (no analyzer declared, default standard analyzer used with no stemming):

{
  "mappings": {
    "properties": {
      "description": { "type": "text" }
    }
  }
}

Correct (analyzer declared on the field, applied at both index and query time):

{
  "mappings": {
    "properties": {
      "description": {
        "type": "text",
        "analyzer": "listing_text_en"
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "listing_text_en": {
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" },
        "english_stemmer": { "type": "stemmer", "language": "english" }
      }
    }
  }
}

Reference: OpenSearch Documentation — Text Analysis

Separate Searchable Fields from Display Fields

An OpenSearch index that stores every listing attribute as a searchable field inflates index size, slows refresh, and makes query analysis harder. Most listing attributes are display-only: description HTML, image URLs, provider bio, formatted price strings. These belong in _source or a separate document store, not in indexed fields. Index only what you search on; store the rest separately and hydrate at fetch time. The separation keeps the index small enough to fit comfortably in memory and query latency low.

Incorrect (every field indexed and searchable, including large display-only HTML):

{
  "mappings": {
    "properties": {
      "listing_id": { "type": "keyword" },
      "title": { "type": "text" },
      "description_html": { "type": "text" },
      "cover_image_url": { "type": "text" },
      "gallery_urls": { "type": "text" },
      "formatted_price_label": { "type": "text" },
      "provider_bio": { "type": "text" }
    }
  }
}

Correct (searchable fields only; display fields excluded from indexing):

{
  "mappings": {
    "properties": {
      "listing_id": { "type": "keyword" },
      "title": { "type": "text", "analyzer": "listing_text_en" },
      "description": { "type": "text", "analyzer": "listing_text_en" },
      "region": { "type": "keyword" },
      "price_tier": { "type": "keyword" },
      "trust_score": { "type": "float" },
      "display_blob_ref": { "type": "keyword", "index": false }
    }
  }
}

Reference: OpenSearch Documentation — Supported Field Types

Stream Listing Updates via CDC, Not Periodic Full Re-Import

A listing index that refreshes on a nightly or hourly batch pulls the entire listings table from the source-of-truth database, rebuilds a bulk-import job, and ships it to OpenSearch. The window between a provider updating their availability and that update reaching the index is hours, and the index is therefore always stale. Change Data Capture (CDC) — streaming row-level changes from the source database to OpenSearch via Debezium, native CDC connectors, or outbox polling — brings staleness down to seconds. Bulk re-import remains useful as a rebuild path for schema changes and disaster recovery, but it is not the refresh mechanism for production traffic.

Incorrect (nightly full re-import, hours of stale metadata):

def nightly_reindex() -> None:
    listings = source_db.execute("SELECT * FROM listings WHERE active = true")
    bulk_body = []
    for listing in listings:
        bulk_body.append({"index": {"_index": "listings", "_id": listing.id}})
        bulk_body.append(listing.to_search_document())
    opensearch.bulk(body=bulk_body, refresh=True)

Correct (CDC stream writes each row change as it happens):

async def consume_cdc_stream() -> None:
    async for change in cdc_consumer.consume(topic="listings.public.listings"):
        if change.op in ("c", "u"):
            opensearch.index(
                index="listings",
                id=change.after.id,
                body=listing_to_search_document(change.after),
                refresh=False,
            )
        elif change.op == "d":
            opensearch.delete(
                index="listings",
                id=change.before.id,
                refresh=False,
            )

Reference: OpenSearch Documentation — Bulk API and Ingestion

Use Index Templates to Enforce Consistency

Marketplace retrieval typically runs across multiple indices — one per listing type, one per region, or one per rollover window for interaction logs. Without an index template, each new index inherits whatever defaults the creator happened to set, and mapping drift accumulates silently until queries behave differently across indices. An index template defines the shared mapping, settings and analyzers once, and every matching new index uses it automatically, eliminating drift and making schema migrations trackable.

Incorrect (per-index mapping set at creation, drift guaranteed):

opensearch.indices.create(
    index="listings-uk",
    body={
        "mappings": {"properties": {"title": {"type": "text"}}},
        "settings": {"number_of_shards": 3},
    },
)
opensearch.indices.create(
    index="listings-fr",
    body={
        "mappings": {"properties": {"title": {"type": "text", "analyzer": "french"}}},
        "settings": {"number_of_shards": 1},
    },
)

Correct (index template applied automatically to matching index names):

opensearch.indices.put_index_template(
    name="listings_template",
    body={
        "index_patterns": ["listings-*"],
        "template": {
            "settings": {"number_of_shards": 3, "number_of_replicas": 1},
            "mappings": {
                "dynamic": "strict",
                "properties": {
                    "listing_id": {"type": "keyword"},
                    "title": {
                        "type": "text",
                        "analyzer": "listing_text_en",
                        "fields": {"raw": {"type": "keyword"}},
                    },
                    "region": {"type": "keyword"},
                    "trust_score": {"type": "float"},
                },
            },
        },
    },
)

Reference: OpenSearch Documentation — Creating a Custom Analyzer

Use keyword and text as Multi-Fields

A field is often needed for two different purposes: full-text search (analyzed, tokenized, stemmed) and exact match or sort (unanalyzed, case-sensitive). Declaring the field as text only means you cannot sort or filter on the exact value; declaring it as keyword only means you lose tokenization and stemming. The multi-field pattern — one text analyzed sub-field plus one keyword unanalyzed sub-field — gives both behaviours from a single source field and is the standard OpenSearch mapping pattern for any human-readable field.

Incorrect (text-only field — cannot sort, cannot term-filter):

{
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "english" }
    }
  }
}

Correct (multi-field: analyzed text plus unanalyzed keyword for sort and filter):

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "english",
        "fields": {
          "raw": { "type": "keyword" },
          "suggest": { "type": "completion" }
        }
      }
    }
  }
}

Reference: OpenSearch Documentation — Supported Field Types

Use Language Analyzers for Language-Sensitive Fields

Stemming "sitters" to "sit" only works with a language-aware analyzer — the standard analyzer tokenises but does not stem, so "sitter" and "sitters" are treated as different terms and the ranker loses recall. OpenSearch ships language analyzers for 30+ languages (english, french, spanish, portuguese, german, and more) that apply language-specific stemming and stopwords. For a multi-language marketplace, the standard pattern is a per-language sub-field plus a language-detection processor on ingest so each document is indexed with the correct analyzer per language.

Incorrect (standard analyzer on multi-language listing title — no stemming):

{
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "standard" }
    }
  }
}

Correct (per-language sub-fields with language-specific analyzers):

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "english",
        "fields": {
          "fr": { "type": "text", "analyzer": "french" },
          "es": { "type": "text", "analyzer": "spanish" },
          "pt": { "type": "text", "analyzer": "portuguese" },
          "de": { "type": "text", "analyzer": "german" }
        }
      }
    }
  }
}

Reference: OpenSearch Documentation — Language Analyzers

Audit Live Query Logs Before Designing

Design work based on what the team imagines users type is always wrong on the details. A one-hour audit of the actual query log surfaces the real distribution: how many queries contain a city, how many contain a date, how many are single-word, how many are mis-spelled, how many return zero results, and how many are reformulations of a previous query in the same session. That distribution should drive analyzer choice, synonym investment, autocomplete design and intent classifier boundaries — not whiteboard assumptions.

Incorrect (designing an analyzer based on imagined queries):

settings = {
    "analysis": {
        "analyzer": {
            "listing_text": {
                "type": "standard",
                "stopwords": "_english_",
            },
        },
    },
}
opensearch.indices.create(index="listings", body={"settings": settings})

Correct (audit the actual distribution, then design the analyzer):

def audit_query_log(days: int = 30) -> QueryLogReport:
    logs = query_log_store.fetch(window_days=days)
    return QueryLogReport(
        total=len(logs),
        has_city_token=sum(1 for q in logs if contains_city(q.raw)) / len(logs),
        has_date_token=sum(1 for q in logs if contains_date(q.raw)) / len(logs),
        avg_token_count=mean(len(q.raw.split()) for q in logs),
        zero_result_rate=sum(1 for q in logs if q.result_count == 0) / len(logs),
        reformulation_rate=reformulation_rate(logs),
        top_100_queries=top_n_by_count(logs, 100),
    )

Reference: Doug Turnbull & John Berryman — Relevant Search

Distinguish Transactional from Exploratory Intent

A transactional session has an unambiguous goal (seeker wants a sitter for specific dates in a specific city) and rewards hard filters, precision and exact availability. An exploratory session is open-ended (seeker is deciding whether to travel) and rewards diverse recall, personalisation and inspiration. Mixing them in one ranker makes transactional sessions drown in irrelevant inspiration and exploratory sessions starve of variety. Route by explicit signal (filter presence, date range specified, keyword specificity) and use different objectives per route.

Incorrect (one ranking objective for both transactional and exploratory):

def rank(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    candidates = retrieve(query, seeker)
    return sort_by_personalisation_score(candidates, seeker)[:24]

Correct (route by transactional signals; transactional gets precision, exploratory gets diversity):

def rank(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    candidates = retrieve(query, seeker)

    has_hard_filters = bool(query.date_range and query.region)
    if has_hard_filters:
        return sort_by_relevance_then_trust(candidates, query)[:24]

    return diversify_then_personalise(candidates, seeker)[:24]

Reference: Eugene Yan — Improving Recommendation Systems and Search

Map Queries to Intent Classes Before Touching Retrieval

User intent drives retrieval strategy: a navigational query ("sarah-the-sitter") wants an exact hit at rank 1, an informational query ("sitter with dog experience") wants a diverse ranked list, and an exploratory query ("london dog sitters for the holidays") wants personalisation and cold-start fallbacks. Applying one retrieval strategy to all three produces a system that is wrong for every query type in a different way. Classify queries first, then route to the right retrieval strategy.

Incorrect (one query handler for all intent classes):

def search(query: str, seeker: Seeker) -> list[Listing]:
    return opensearch.search(
        index="listings",
        body={
            "query": {"multi_match": {"query": query, "fields": ["title", "description"]}},
            "size": 24,
        },
    )["hits"]["hits"]

Correct (intent classifier routes to a specific strategy per class):

def search(query: str, seeker: Seeker) -> list[Listing]:
    intent = classify_intent(query, seeker)
    if intent == Intent.NAVIGATIONAL:
        return exact_match_top_1(query)
    if intent == Intent.TRANSACTIONAL:
        return filtered_ranked(query, seeker, filters=seeker.hard_filters)
    if intent == Intent.EXPLORATORY:
        return hybrid_bm25_knn_with_personalisation(query, seeker)
    return default_blended(query, seeker)

Reference: OpenSource Connections — What Is a Relevant Search Result?

Reject the One-Search-For-Everything Temptation

The most common architectural mistake in marketplace retrieval is trying to serve every surface from one query shape — one OpenSearch query, one ranker, one candidate set — because it is "simpler". The simplicity is a trap: you end up with a query that is sub-optimal for every surface, hard to tune, impossible to A/B test per surface, and brittle when one surface changes. Accept the cost of having multiple query shapes per surface (homefeed, search, item-page similar, category landing) so each can be tuned against its own metric.

Incorrect (one query template reused across homefeed, search and category):

def generic_listings(surface: str, seeker: Seeker, query: str | None) -> list[Listing]:
    body = {
        "query": {"multi_match": {"query": query or "*", "fields": ["title", "description"]}},
        "size": 24,
    }
    return opensearch.search(index="listings", body=body)["hits"]["hits"]

Correct (surface-specific query shapes each tuned for their own metric):

def homefeed(seeker: Seeker) -> list[Listing]:
    return opensearch.search(index="listings", body=homefeed_query(seeker))["hits"]["hits"]

def search(query: SearchQuery, seeker: Seeker) -> list[Listing]:
    return opensearch.search(index="listings", body=search_query(query, seeker))["hits"]["hits"]

def category_landing(category: str, seeker: Seeker) -> list[Listing]:
    return opensearch.search(index="listings", body=category_query(category, seeker))["hits"]["hits"]

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking

Separate Known-Item Search from Discovery

Known-item search ("the seeker already has a specific listing in mind and wants to find it") and discovery ("the seeker wants inspiration and would accept any of several good matches") have opposite failure modes: known-item fails on recall (the one right answer is not in the result set), discovery fails on diversity and personalisation. A single retrieval pipeline optimised for one collapses on the other. Identify the two at the routing layer and run separate retrieval strategies.

Incorrect (single ranked list for both known-item and discovery):

def search(query: str, seeker: Seeker) -> list[Listing]:
    return opensearch.search(
        index="listings",
        body={
            "query": {"match": {"name": query}},
            "size": 24,
        },
    )["hits"]["hits"]

Correct (known-item gets exact-match first, discovery gets ranked list):

def search(query: str, seeker: Seeker) -> list[Listing]:
    exact_candidates = opensearch.search(
        index="listings",
        body={
            "query": {"term": {"slug.keyword": normalise(query)}},
            "size": 1,
        },
    )["hits"]["hits"]
    if exact_candidates and looks_like_known_item(query):
        return exact_candidates

    return opensearch.search(
        index="listings",
        body={
            "query": {"multi_match": {"query": query, "fields": ["title^2", "description"]}},
            "size": 24,
        },
    )["hits"]["hits"]

Reference: Doug Turnbull & John Berryman — Relevant Search

Treat No-Search as a First-Class Choice

Some surfaces do not need a search box, a ranker or a recommender at all — a curated category tree, a region carousel, or a "featured this week" list is often the right primitive when the user's intent is browse-without-query. Forcing a search-style retrieval into a browse context produces a system that is technically smarter but operationally worse: the user clicks fewer things and reformulates more. Treat "no search, just a hand-curated or rule-based list" as a legitimate design choice for surfaces where intent is unstructured.

Incorrect (search-style retrieval powering a browse surface):

def browse_homepage(seeker: Seeker) -> list[Listing]:
    body = {
        "query": {
            "function_score": {
                "query": {"match_all": {}},
                "functions": [{"random_score": {}}],
            },
        },
        "size": 24,
    }
    return opensearch.search(index="listings", body=body)["hits"]["hits"]

Correct (curated, rule-based list served from a content store):

def browse_homepage(seeker: Seeker) -> list[BrowseModule]:
    return [
        BrowseModule.hero_banner(curated.current_hero()),
        BrowseModule.region_carousel(
            title="Popular this week",
            listings=catalog.top_bookings_last_7_days(seeker.region, limit=12),
        ),
        BrowseModule.category_tree(curated.category_tree()),
        BrowseModule.editorial_collection(curated.editorial_of_the_week()),
    ]

Reference: Eugene Yan — Improving Recommendation Systems and Search

Define Session Success per Surface

Each retrieval surface has a different definition of success. For a search surface, success might be "seeker clicked a result and did not reformulate within the session". For a homefeed, success might be "seeker spent more than 30 seconds on a listing or saved it". For item-page-related, success might be "seeker clicked a related listing and the related listing converted". A single success metric applied across surfaces gives the wrong answer for most of them. Define the success metric per surface as the first artefact of the project, alongside the intent mapping.

Incorrect (one success metric applied across all surfaces):

def measure_success(sessions: list[Session]) -> float:
    return sum(1 for s in sessions if s.clicked_any_listing) / len(sessions)

Correct (per-surface success definition):

SUCCESS_METRICS = {
    "homefeed": lambda s: s.spent_more_than(seconds=30) or s.saved_any_listing,
    "search_results": lambda s: s.clicked_any_listing and not s.reformulated_within(seconds=60),
    "item_page_related": lambda s: s.clicked_related_listing and s.related_click_converted,
    "category_landing": lambda s: s.scrolled_to_half_page and s.clicked_any_listing,
}

def measure_success(surface: str, sessions: list[Session]) -> float:
    success_fn = SUCCESS_METRICS[surface]
    return sum(1 for s in sessions if success_fn(s)) / len(sessions)

Reference: Google — Rules of Machine Learning, Rule 2: First, Design and Implement Metrics

Run Interleaving as a Cheap A/B Proxy

A traditional A/B test needs thousands of sessions to reach statistical significance on small ranking changes. Interleaving — showing a single list where alternating slots come from variant A and variant B — lets each seeker effectively vote on both variants simultaneously, reducing the sample size needed by 10-100x for the same statistical power. The trade-off is that interleaving answers "which variant is preferred within a single session" rather than "which variant produces better long-term outcomes", but for ranking-quality measurement that is usually enough.

Incorrect (full A/B split required for every ranking change, multi-week experiment runs):

def evaluate_ranking_change(variant_a: Ranker, variant_b: Ranker) -> Report:
    experiment = ab_test.create(
        name="ranker-v2-vs-v1",
        allocation={"a": 0.5, "b": 0.5},
        primary_metric="click_through_rate",
    )
    return experiment.wait_for_significance(min_sample_size=50_000)

Correct (team draft interleaving, 10x less sample needed):

def evaluate_ranking_change(variant_a: Ranker, variant_b: Ranker) -> Report:
    experiment = interleaving.create(
        name="ranker-v2-vs-v1",
        variants=[variant_a, variant_b],
        interleave_method="team_draft",
        primary_metric="clicks_by_variant",
    )
    return experiment.wait_for_significance(min_sample_size=5_000)

Reference: Pinecone — Evaluation Measures in Information Retrieval

Track NDCG, MRR and Zero-Result Rate

Three metrics together give a complete picture of search ranking quality: NDCG@10 measures how well the top-10 match graded relevance judgments, MRR measures how often the first relevant result is at rank 1, and zero-result rate measures how often the query finds nothing at all. Tracking any one in isolation hides failure modes — MRR can rise while NDCG drops if the first match improves at the expense of the rest, and a ranker can score perfectly on NDCG while the zero-result rate rises quietly. All three belong on the weekly dashboard.

Incorrect (only click-through rate tracked as ranking quality):

def weekly_search_quality() -> dict:
    return {"ctr": click_logs.ctr(window_days=7)}

Correct (NDCG, MRR, zero-result rate plus CTR as the full picture):

def weekly_search_quality() -> dict:
    return {
        "ndcg_at_10": offline_eval.ndcg_at_k(golden_set.current(), k=10),
        "mrr": offline_eval.mrr(golden_set.current()),
        "zero_result_rate": query_logs.zero_result_fraction(window_days=7),
        "ctr": click_logs.ctr(window_days=7),
        "reformulation_rate": query_logs.reformulation_fraction(window_days=7),
        "session_success_rate": session_success(window_days=7),
    }

Reference: Pinecone — Evaluation Measures in Information Retrieval

Track Reformulation Rate as a Failure Signal

When a seeker types a query, scrolls through results, and then types a different query within the same session, that is a strong implicit signal the first query failed — the seeker did not find what they wanted and had to try again. The reformulation rate (percentage of sessions with two or more queries within 60 seconds) is a near-universal proxy for search quality that requires no human judgments and no explicit feedback. It is the cheapest search-quality metric a team can add, and a rising reformulation rate is an early warning long before CTR or booking rate move.

Incorrect (reformulations not tracked; failed queries invisible):

def log_search_query(seeker_id: str, query: str) -> None:
    query_log.append({"seeker_id": seeker_id, "query": query, "ts": datetime.utcnow()})

Correct (reformulations detected via session-level query grouping):

def compute_reformulation_rate(window_days: int) -> float:
    sessions = session_store.fetch_with_queries(window_days=window_days)
    reformulated = 0
    for session in sessions:
        queries = sorted(session.queries, key=lambda q: q.timestamp)
        for earlier, later in zip(queries, queries[1:]):
            gap = (later.timestamp - earlier.timestamp).total_seconds()
            if gap < 60 and earlier.normalised != later.normalised:
                reformulated += 1
                break
    return reformulated / len(sessions) if sessions else 0.0

Reference: Doug Turnbull & John Berryman — Relevant Search

Use Click Models for Implicit Relevance Judgments

Human-graded relevance judgments are the gold standard, but they are slow, expensive and cover only a fraction of queries. Click models (Cascade, Dependent Click Model, Position-Based Model) infer implicit relevance judgments from click-through data by modelling the probability a seeker saw each result, clicked each result, and found it relevant — correcting for position bias in the process. A click-model-derived judgment set is not as clean as a human-judged one, but it scales to every query in the log and stays current automatically, which makes it the right primary source for ongoing offline evaluation.

Incorrect (raw CTR used as relevance proxy, position-biased):

def infer_relevance_from_clicks(clicks: list[ClickEvent]) -> dict[str, float]:
    grouped = defaultdict(lambda: {"impressions": 0, "clicks": 0})
    for click in clicks:
        grouped[click.listing_id]["impressions"] += 1
        if click.was_clicked:
            grouped[click.listing_id]["clicks"] += 1
    return {
        listing_id: stats["clicks"] / stats["impressions"]
        for listing_id, stats in grouped.items()
        if stats["impressions"] > 0
    }

Correct (position-based click model corrects for slot bias):

def infer_relevance_position_based(clicks: list[ClickEvent]) -> dict[str, float]:
    position_ctrs = learn_position_ctrs(clicks)
    relevance = {}
    for click in clicks:
        position_prior = position_ctrs[click.slot]
        examine_prob = position_prior
        relevance[click.listing_id] = relevance.get(click.listing_id, 0.0)
        if click.was_clicked:
            relevance[click.listing_id] += 1.0 / max(examine_prob, 0.01)
    return normalise_to_unit(relevance)

Reference: Pinecone — Evaluation Measures in Information Retrieval

Alert on Decision-Triggering Metrics, Not Just Error Rates

Traditional alerting fires on error rates, timeouts and service unavailability — the catastrophic failure modes. Search quality regressions almost never show up as catastrophes; they show up as slow drifts that are invisible until booking rate moves weeks later. Alerts on decision-triggering metrics — a 20% rise in zero-result rate, a 15% rise in reformulation rate, a 5% drop in NDCG against a frozen golden set, a 30% rise in exposure Gini — fire hours or days earlier and give the team time to diagnose before damage spreads. The alert payload should include the gotchas.md pointer for the likely diagnosis.

Incorrect (only infrastructure error rates alert):

alerts.create(name="search_error_rate", metric="search.5xx_rate", threshold=0.01)
alerts.create(name="search_latency_p99", metric="search.p99_ms", threshold=500)

Correct (decision-triggering quality alerts alongside infrastructure ones):

alerts.create(name="search_error_rate", metric="search.5xx_rate", threshold=0.01)
alerts.create(name="search_latency_p99", metric="search.p99_ms", threshold=500)

alerts.create(
    name="zero_result_spike",
    metric="search.zero_result_rate",
    threshold=Threshold(value=0.12, direction="above", window="15m"),
    runbook="references/playbooks/improving.md#zero-result-spike",
)
alerts.create(
    name="reformulation_spike",
    metric="search.reformulation_rate_60s",
    threshold=Threshold(value=0.25, direction="above", window="1h"),
    runbook="references/playbooks/improving.md#reformulation-spike",
)
alerts.create(
    name="ndcg_regression",
    metric="search.ndcg_at_10_vs_baseline",
    threshold=Threshold(value=-0.05, direction="below", window="24h"),
    runbook="references/playbooks/improving.md#ndcg-regression",
)

Reference: Google — Rules of Machine Learning, Rule 8: Know the Freshness Requirements

Build a Search Health Dashboard with Threshold Lines

A search system needs a single-pane-of-glass dashboard that shows the current state of every upstream and downstream metric with an explicit threshold line indicating the decision boundary — below the line means the team acts, above it means the system is healthy. Raw time series without threshold lines force the viewer to remember whether "12% zero results" is good or bad; a dashed horizontal line at 10% with colour banding makes the interpretation instant. The dashboard is not a metric showcase — it is a decision-making artefact.

Incorrect (raw metric dashboard with no thresholds, interpretation unclear):

dashboard.add_panel(title="Zero Result Rate", metric="search.zero_result_rate")
dashboard.add_panel(title="NDCG@10", metric="search.ndcg_at_10")
dashboard.add_panel(title="Reformulation Rate", metric="search.reformulation_rate")

Correct (every panel has a threshold line and colour-banded state):

dashboard.add_panel(
    title="Zero Result Rate",
    metric="search.zero_result_rate",
    threshold=Threshold(warning=0.08, critical=0.12, direction="below_is_better"),
    colour_bands=[(0, 0.08, "green"), (0.08, 0.12, "yellow"), (0.12, 1.0, "red")],
)
dashboard.add_panel(
    title="NDCG@10 (golden set v3.2)",
    metric="search.ndcg_at_10_golden_v3_2",
    threshold=Threshold(warning=0.68, critical=0.65, direction="above_is_better"),
    baseline_line=0.70,
)
dashboard.add_panel(
    title="Reformulation Rate (60s)",
    metric="search.reformulation_rate_60s",
    threshold=Threshold(warning=0.18, critical=0.25, direction="below_is_better"),
)

Reference: Google — Rules of Machine Learning, Rule 8: Know the Freshness Requirements of Your System

Log Every Query with Full Context for Counterfactual Replay

A query log that stores only the raw query string is insufficient for diagnosis — a week later when the team wants to understand why a specific query underperformed, they cannot reproduce the retrieval because they do not know what filters were applied, which ranker ran, what context was in play, or what the result set actually was. A structured log entry per query that captures query, normalised form, classifier output, filters, ranker version, top-K result IDs with scores, and the strategy that was chosen lets any engineer replay the query later and audit the decision path.

Incorrect (raw query logged as a line of text):

def search(raw_query: str, seeker: Seeker) -> list[Listing]:
    logger.info(f"search: seeker={seeker.id} query={raw_query}")
    return opensearch_search(normalise_query(raw_query), seeker)

Correct (structured event with every decision point and result captured):

def search(raw_query: str, seeker: Seeker) -> list[Listing]:
    request_id = str(uuid4())
    classified = classify(raw_query)
    hits = opensearch_search(classified, seeker)
    event = QueryEvent(
        request_id=request_id,
        seeker_id=seeker.id,
        timestamp=datetime.utcnow(),
        raw=raw_query,
        normalised=classified.normalised,
        intent=classified.intent,
        filters=classified.filters,
        ranker_version=current_ranker_version(),
        strategy=hits.strategy,
        top_k=[(h.listing_id, h.score, h.slot) for h in hits.listings[:24]],
    )
    query_log.put(event)
    return hits.listings

Reference: Eugene Yan — System Design for Discovery

Run a Weekly Search-Quality Review Ritual

Incident-driven quality reviews (the team meets when something breaks) are too late — by then the regression has been live for days. Calendar-driven weekly reviews are the structural fix: a 30-minute meeting every Monday where the team walks through the dashboard, the decisions log updated since last week, new gotchas captured from the last incident, and the top five queries with the largest zero-result, reformulation, or NDCG deltas. The review produces a set of tickets for the week's work, or an explicit "no action needed" decision that gets recorded.

Incorrect (ad-hoc review only when an alarm fires):

def on_alert_fired(alert: Alert) -> None:
    slack.post(f"Alert: {alert.name}")
    oncall_engineer.page()

Correct (weekly review scheduled, structured artefact produced):

def weekly_search_quality_review(week_start: date) -> ReviewArtefact:
    return ReviewArtefact(
        week_start=week_start,
        dashboard_snapshot=dashboard.snapshot(),
        new_decisions=decisions_log.entries_since(week_start - timedelta(days=7)),
        new_gotchas=gotchas_file.entries_since(week_start - timedelta(days=7)),
        top_5_degraded_queries=query_log.top_degraded(metric="zero_result", k=5),
        top_5_degraded_rankings=offline_eval.top_degraded_queries(golden_set.current(), k=5),
        actions=team_decides(),
    )

Reference: Google — Rules of Machine Learning, Rule 27: Try to Quantify Observed Undesirable Behavior

Scrub PII from Query Logs Before Warehouse Ingestion

Raw query logs contain personally identifiable information by construction — seeker identifiers, raw free-text queries that may contain names or addresses, IP-resolvable metadata, and sometimes contact details that users typed into the search box expecting them to be private. For a two-sided trust marketplace subject to GDPR and similar regimes, streaming those raw logs to the analytics warehouse without redaction creates a data-leak blast radius that grows every day. The fix is a redaction step in the ingestion pipeline: hash seeker identifiers, regex-strip common PII patterns from raw query text, and apply a separate retention policy to the redacted structured events versus the raw text.

Incorrect (raw query log streamed unredacted to the analytics warehouse):

def emit_query_event(event: QueryEvent) -> None:
    warehouse_stream.put({
        "request_id": event.request_id,
        "seeker_id": event.seeker_id,
        "raw_query": event.raw,
        "normalised_query": event.normalised,
        "top_k": event.top_k,
        "timestamp": event.timestamp.isoformat(),
    })

Correct (seeker ID hashed, raw text pattern-redacted, dual retention):

EMAIL_RE = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
PHONE_RE = re.compile(r"\b(\+?\d{1,3}[\s-]?)?\(?\d{2,4}\)?[\s-]?\d{3,4}[\s-]?\d{3,4}\b")

def scrub_query_text(raw: str) -> str:
    scrubbed = EMAIL_RE.sub("[email]", raw)
    scrubbed = PHONE_RE.sub("[phone]", scrubbed)
    return scrubbed

def emit_query_event(event: QueryEvent) -> None:
    warehouse_stream.put({
        "request_id": event.request_id,
        "seeker_hash": hashlib.blake2b(event.seeker_id.encode(), digest_size=16).hexdigest(),
        "scrubbed_query": scrub_query_text(event.raw),
        "normalised_query": event.normalised,
        "top_k": event.top_k,
        "timestamp": event.timestamp.isoformat(),
    })
    raw_store.put(event, retention_days=7)

Reference: Detecting PII in Log Data for GDPR Compliance_in_log_data_for_GDPR_compliance)

Track Ranking Stability as a Churn Metric

Ranking churn — how much the top-10 for a fixed query set changes from one release to the next — is a leading indicator of model instability. A small, deliberate change typically moves the top-10 by 5-15% of positions. A sudden 40% churn without a matching deliberate change suggests a data pipeline drift, an index refresh anomaly, or a silent feature regression, and it shows up in this metric days before the business metrics move. The churn metric is computed by running the golden query set before and after each release and comparing the result lists with a rank-correlation measure like Kendall's tau or rank-biased overlap (RBO).

Incorrect (ranking stability never measured, silent drift invisible):

def post_release_checks() -> None:
    run_smoke_tests()
    check_error_rates()

Correct (RBO churn measured on every release against the golden set):

def post_release_checks() -> None:
    run_smoke_tests()
    check_error_rates()

    golden = golden_set.load_current()
    pre_ranking = cache.get("pre_release_ranking")
    post_ranking = {q.text: current_ranker(q) for q in golden.queries}

    rbo_scores = [
        rank_biased_overlap(pre_ranking[q], post_ranking[q], p=0.9)
        for q in pre_ranking
    ]
    avg_rbo = mean(rbo_scores)
    dashboard.emit("search.release_churn_rbo", value=avg_rbo)
    if avg_rbo < 0.75:
        pager.alert(
            f"Ranking churn suspicious: avg RBO {avg_rbo:.3f}",
            runbook="references/playbooks/improving.md#ranking-churn",
        )

Reference: Pinecone — Evaluation Measures in Information Retrieval

Audit Before You Build — Gate Work on Instrumentation Readiness

Any ranking, retrieval, or model work that begins before the telemetry is known-good is guaranteed to produce surprising online metrics that nobody can explain. The cheap fix is a one-hour audit at the start of every project: verify impression coverage, outcome coverage, request-ID join rate, zero-result capture, and reformulation detection. If coverage is below threshold, fix the telemetry first — the model work waits. This rule is the hardest to follow because it feels like a delay, but it is always faster than debugging mysterious online metrics two months later.

Incorrect (project kick-off with no instrumentation audit):

def kickoff_project(project: str) -> None:
    create_jira_epic(project)
    schedule_design_review(project)
    allocate_engineers(project, headcount=3)

Correct (audit gate blocks kick-off on failing coverage):

def kickoff_project(project: str) -> None:
    audit = run_telemetry_audit(window_days=30)
    required = {
        "impression_coverage": 0.95,
        "outcome_coverage": 0.90,
        "request_id_join_rate": 0.98,
        "zero_result_capture": 1.00,
        "reformulation_detection": 0.90,
    }
    failing = {
        key: (value, audit.get(key))
        for key, value in required.items()
        if audit.get(key, 0.0) < value
    }
    if failing:
        create_jira_epic(f"{project}-telemetry-fix", blockers=failing)
        return
    create_jira_epic(project)
    schedule_design_review(project)

Reference: Google — Rules of Machine Learning, Rule 2: First, Design and Implement Metrics

Build a Golden Query Set as the First Artefact

A golden query set is a curated collection of representative queries with expected top results, graded by human judges — it is the reference point against which every ranking change is offline-evaluated before any online A/B test. Without a golden set, regressions are invisible until they show up in production metrics, and every ranking experiment is a high-variance bet. The golden set is built once at the start of a retrieval project, frozen per evaluation cycle, and versioned as a living artefact that grows with the domain. Building it is a one-week exercise; not building it adds months of debugging time.

Incorrect (no golden set — ranking changes merged based on intuition):

def deploy_ranking_change(change: RankingChange) -> None:
    run_unit_tests()
    if all_tests_pass():
        deploy_to_production()

Correct (golden set regression test as a mandatory gate):

def deploy_ranking_change(change: RankingChange) -> None:
    run_unit_tests()
    offline_metrics = evaluate_against_golden_set(
        change=change,
        golden_set=golden_set.load_version("v3.2-frozen-2026-03"),
        metrics=["ndcg@10", "mrr", "zero_result_rate"],
    )
    if offline_metrics.ndcg_at_10 < 0.98 * production.ndcg_at_10:
        raise RegressionError(
            f"NDCG@10 dropped from {production.ndcg_at_10} to {offline_metrics.ndcg_at_10}"
        )
    deploy_to_shadow_traffic(change)

Reference: Pinecone — Evaluation Measures in Information Retrieval

Find the Bottleneck Before Optimising

Goldratt's Theory of Constraints observes that the throughput of a system is governed by a single bottleneck — work on anything else produces no end-to-end improvement. In retrieval systems, the bottleneck is rarely ranking sophistication; it is usually zero-result rate, stale index, missing intent classes, or broken telemetry. A one-day diagnostic that measures each layer (audit coverage, index freshness, query-log zero-result rate, retrieval recall on golden set, ranking NDCG gap to baseline) identifies the bottleneck and directs work to the layer that will actually move the needle.

Incorrect (jumping straight to ranking tuning without diagnosis):

def next_sprint() -> list[Task]:
    return [
        Task("Tune BM25 k1 and b parameters"),
        Task("Deploy Learning to Rank model v2"),
        Task("Add cross-encoder re-ranker"),
    ]

Correct (diagnostic-driven sprint planning):

def next_sprint() -> list[Task]:
    diagnosis = run_bottleneck_diagnostic()

    if diagnosis.zero_result_rate > 0.12:
        return [Task("Add relaxed-query fallback"), Task("Expand synonym dictionary")]
    if diagnosis.index_freshness_p99 > timedelta(hours=2):
        return [Task("Switch to PutItems stream for metadata")]
    if diagnosis.recall_at_100_on_golden_set < 0.85:
        return [Task("Retrieval recall improvement"), Task("Hybrid BM25 plus KNN")]
    if diagnosis.ndcg_gap_to_baseline < 0.03:
        return [Task("Ranking improvements worth justifying")]
    return [Task("Exploration: collect more labelled data")]

Reference: Google — Rules of Machine Learning, Rule 16: Plan to Launch and Iterate

Hand Off to the Personalisation Skill When the Bottleneck Is Personalisation

This skill covers the retrieval planning layer — intent framing, architecture, OpenSearch index and query design, search relevance, and the meta-methodology for planning and diagnosing. When the diagnostic identifies that the bottleneck is personalisation-specific — impression tracking for a recommender, AWS Personalize schema design, feedback-loop bias, mutual-fit ranking in a marketplace — the next step is the companion skill marketplace-personalisation, which has 49 rules specifically about that layer. Recognising the hand-off point is part of planning: do not re-derive rules that already exist in the companion skill.

Incorrect (trying to solve a personalisation problem inside the search planning skill):

def improve_homefeed_ranking() -> None:
    audit = run_bottleneck_diagnostic()
    if audit.bottleneck == "impression_tracking":
        write_rule_in_this_skill_about_impressions()
        write_rule_in_this_skill_about_negative_signals()

Correct (bottleneck identified, hand off to the personalisation skill):

def improve_homefeed_ranking() -> None:
    audit = run_bottleneck_diagnostic()
    if audit.bottleneck in {"impression_tracking", "feedback_loop", "cold_start"}:
        refer_to_skill(
            "marketplace-personalisation",
            playbook="references/playbooks/improving.md",
            reason=f"Retrieval layer is healthy; bottleneck is {audit.bottleneck}",
        )
        return
    run_search_planning_workflow(audit)

Reference: Google — Rules of Machine Learning, Rule 38: Do Not Waste Time on New Features if Unaligned Objectives Become the Issue

Maintain a Decisions Log as Living Context

Every ranking tweak, schema change, synonym addition and filter edit is a decision made with context — a hypothesis, an offline metric, an A/B result, a reason. Without a decisions log, that context evaporates within a quarter: the team that inherits the system cannot explain why the trust_score boost is 1.8 instead of 2.0, why a particular synonym exists, or why a feature was removed. A plain-text decisions log, committed alongside the code, captures the why of every change so future engineers inherit context, not just artefacts. Treat the decisions log as a living document that the agent reads before suggesting changes and updates after every shipped experiment.

Incorrect (change committed with no recorded rationale):

git commit -m "Update trust score boost"

Correct (structured decisions-log entry committed alongside the change):

# decisions/2026-04-11-trust-score-boost.md

## Decision
Raise `trust_score` function_score weight from 1.5 to 1.8 in homefeed ranker.

## Why
Q1 audit showed top-10 results were dominated by high-CTR but low-trust listings
(provider decline rate 18% on top-1). Raising the weight shifts rank toward
higher-trust providers even when CTR is slightly lower.

## Evidence
- Offline: NDCG@10 +0.008 on golden set v3.2
- A/B: +2.1% booking_completed, -0.5% CTR, -6pp provider decline rate
- Ship criterion: booking_completed ≥ +1.5% (met)

## Rollback
Revert `weight` to 1.5 in `configs/homefeed_ranker.json`.

## Related
- Replaces earlier experiment 2026-02-08-trust-score-experiment
- References: rules/match-rank-mutual-fit.md

Reference: Google — Rules of Machine Learning, Rule 27: Try to Quantify Observed Undesirable Behavior

Freeze and Version the Golden Set per Evaluation Cycle

A golden query set that changes mid-evaluation cycle produces uncomparable numbers: version A was evaluated on 500 queries, version B on 540, and the NDCG delta is partly an artefact of the new queries. Freezing the golden set per cycle — committing it to the repo with a version tag — makes every version in the cycle comparable. The set still grows across cycles as the domain expands, but within a cycle it is immutable. This is the same principle that applies to benchmark sets in academic ML: fixed inputs, varying models.

Incorrect (queries added to the golden set ad-hoc during an eval):

def add_golden_query(query: str, expected_listings: list[str]) -> None:
    golden_set.append(query, expected_listings)
    save_golden_set()

Correct (golden set versioned and frozen per evaluation cycle):

def add_golden_query(query: str, expected_listings: list[str]) -> None:
    if current_cycle.is_frozen():
        raise CycleFrozenError(
            f"Cycle {current_cycle.version} is frozen. "
            f"Add to next cycle via golden_set.open_next_cycle()"
        )
    current_cycle.append(query, expected_listings)

def freeze_current_cycle() -> str:
    version = current_cycle.freeze_and_tag()
    git.commit(f"golden_set/{version}.jsonl", message=f"Freeze golden set {version}")
    return version

Reference: Pinecone — Evaluation Measures in Information Retrieval

Build Autocomplete on a Separate Index

Autocomplete has different latency requirements and different retrieval semantics from main search — every keystroke fires a query, results must return in <50ms, and the ranking is driven by prefix match and popularity, not full relevance scoring. Running autocomplete from the same OpenSearch index as main search couples the two, so autocomplete traffic contends for the same query slots and one slow autocomplete query slows everything. A dedicated completion suggester index with a smaller denormalised document gives you single-digit millisecond latency and isolates the traffic.

Incorrect (autocomplete served from the main listings index with match_phrase_prefix):

{
  "query": {
    "match_phrase_prefix": {
      "title": { "query": "dog sit" }
    }
  },
  "size": 10
}

Correct (dedicated completion suggester on a small-footprint index):

{
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "analyzer": "simple",
        "preserve_separators": true,
        "preserve_position_increments": true,
        "max_input_length": 50
      }
    }
  }
}

Reference: OpenSearch Documentation — Text Analyzers

Classify Queries Before Routing

Intent classification turns a raw query string into a structured record — {type: transactional, region: "london", date_range: next_week, species: dog} — that downstream retrieval and ranking can use deliberately. Without classification, the system treats every query the same and downstream code tries to reverse-engineer intent from the string repeatedly. Build the classifier as a single-pass component at the top of the request pipeline; it can be rule-based at the start and replaced with a model later without changing the downstream code.

Incorrect (no classification, downstream code re-parses the query repeatedly):

def search(query: str, seeker: Seeker) -> list[Listing]:
    has_date = re.search(r"\b(next|this|tomorrow|\d{1,2}[/-]\d{1,2})\b", query)
    has_city = any(city in query.lower() for city in KNOWN_CITIES)
    if has_date and has_city:
        return transactional_search(query, seeker)
    if has_city:
        return exploratory_search(query, seeker)
    return default_search(query, seeker)

Correct (single-pass classifier returns a structured query record):

def classify(raw: str) -> ClassifiedQuery:
    normalised = normalise_query(raw)
    return ClassifiedQuery(
        raw=raw,
        normalised=normalised,
        intent=rule_based_intent(normalised),
        entities=extract_entities(normalised),
        date_range=extract_date_range(normalised),
        species=extract_species(normalised),
    )

def search(raw: str, seeker: Seeker) -> list[Listing]:
    classified = classify(raw)
    return router.route(classified, seeker)

Reference: Eugene Yan — Improving Recommendation Systems and Search

Curate Synonyms by Domain Intent

Generic synonym lists (WordNet, thesauri) match vocabulary but not domain intent — they expand "dog" to "puppy, pooch, canine" which is fine, but miss "sitter ↔ carer ↔ walker ↔ minder" which is exactly what the marketplace needs. Domain synonyms must be curated by someone who understands the product vocabulary, versioned alongside the code, and grown from real zero-result query logs. They belong in an OpenSearch synonym file or synonym graph filter, loaded at index time for bidirectional expansion.

Incorrect (no synonyms — query "dog carer" misses "dog sitter" listings):

{
  "settings": {
    "analysis": {
      "analyzer": {
        "listing_text": {
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      }
    }
  }
}

Correct (curated domain synonyms from zero-result query analysis):

{
  "settings": {
    "analysis": {
      "filter": {
        "marketplace_synonyms": {
          "type": "synonym_graph",
          "synonyms": [
            "sitter, carer, walker, minder, host",
            "stay, visit, booking, trip",
            "dog, puppy, pooch, canine",
            "cat, kitten, feline"
          ]
        }
      },
      "analyzer": {
        "listing_text": {
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "marketplace_synonyms", "english_stemmer"]
        }
      }
    }
  }
}

Reference: OpenSearch Documentation — Creating a Custom Analyzer