Marketplace Personalisation

Marketplace Engineering Two-Sided Personalisation Best Practices

Comprehensive guide for designing, building and improving personalisation and recommendation systems in two-sided trust marketplaces on AWS Personalize. Contains 49 rules across 9 categories, ordered by cascade impact on the personalisation lifecycle, plus two playbooks for planning a new system from scratch and diagnosing an existing one.

When to Apply

Reference this skill when:

• Designing the event schema and tracking for a new recommender system
• Choosing an AWS Personalize recipe (USER_PERSONALIZATION_v2, SIMS, PERSONALIZED_RANKING_v2)
• Writing or reviewing candidate-generation and re-ranking code for marketplace search or homefeed
• Handling cold start for new providers, new seekers, or new catalog regions
• Diagnosing a live system that "mostly works but feels stale, unfair, or unpersonalised"
• Planning the next experiment, baseline comparison, or A/B test for the recommender
• Investigating concentration, coverage collapse, death spirals, or training-serving skew
• Adding observability dashboards, drift detection, or online metric slicing

Setup

This skill has no user-specific configuration — it is self-contained. References are live URLs to official AWS Personalize documentation, academic papers on bias and exposure, and engineering blogs from Airbnb and DoorDash.

Rule Categories

Categories are ordered by cascade impact: earlier stages poison everything downstream.

Quick Reference

1. Event Tracking and Capture (CRITICAL)

• `track-log-impressions-alongside-clicks` — the denominator that turns clicks into a rate and unlocks unbiased training
• `track-use-stable-opaque-item-ids` — prevents history loss when listings rename or move
• `track-stamp-events-with-request-id` — the join key that enables impression-to-outcome attribution
• `track-stream-events-via-putevents` — real-time adaptation versus end-of-day bulk import
• `track-capture-negative-signals` — dismissal is information, silence is not
• `track-measure-outcomes-not-clicks` — reward the completed booking, not the clickbait

2. Dataset and Schema Design (CRITICAL)

• `schema-design-conservatively` — Interactions schemas are immutable, Users/Items are painful to change
• `schema-keep-user-item-thin` — volatile fields belong in events
• `schema-enforce-metadata-freshness` — PutItems on every metadata change
• `schema-prefer-categorical-fields` — unlock per-value features
• `schema-weight-event-value` — align the model with the business outcome
• `schema-include-context-everywhere` — train-serve feature parity
• `schema-meet-minimum-dataset-sizes` — 50 users / 50 items / 1000 interactions before training

3. Two-Sided Matching Patterns (CRITICAL)

• `match-rank-mutual-fit` — rank by mutual accept probability
• `match-hard-filter-before-ranking` — retrieval enforces feasibility
• `match-cap-provider-exposure` — diversity as a fairness constraint
• `match-model-capacity-constraints` — capacity-discounted scoring
• `match-balance-supply-demand` — per-segment strategy routing

4. Simple Baselines and Theory of Constraints (HIGH)

• `simple-ship-popularity-baseline` — a reference point that every ML model must beat
• `simple-find-bottleneck-first` — diagnostic before optimisation
• `simple-heuristic-rerank-cold-cohorts` — trust × recency × proximity
• `simple-budget-complexity` — ship or kill criterion before running
• `simple-audit-before-build` — telemetry audit gates model work
• `simple-measure-gap-to-baseline` — baseline retained as permanent minority bucket

5. Feedback Loops and Bias Control (HIGH)

• `loop-log-ranking-slot` — slot data for position-bias correction
• `loop-reserve-random-exploration` — unbiased training data
• `loop-optimize-completed-outcome` — reward the goal, not the proxy
• `loop-decay-event-weights` — old preferences fade
• `loop-detect-death-spirals` — exposure Gini as a leading indicator

6. Cold Start and Coverage (HIGH)

• `cold-use-v2-recipe-with-metadata` — metadata extrapolates to new listings
• `cold-best-of-segment-popularity` — segmentation beats global top-N
• `cold-capture-onboarding-intent` — ask instead of guessing
• `cold-reserve-exploration-slots` — promotions filter for fresh inventory
• `cold-tag-cold-start-recs` — warm-versus-cold metric slicing

7. Recipe and Pipeline Selection (MEDIUM-HIGH)

• `recipe-default-to-user-personalization-v2` — discovery default
• `recipe-sims-for-item-page-only` — similar-items is not a homepage recipe
• `recipe-personalized-ranking-as-reranker` — not a candidate generator
• `recipe-build-candidate-rerank-pipeline` — two layers, two concerns
• `recipe-defer-hpo-until-baseline-measured` — prove the model before tuning

8. Inference, Filters and Re-ranking (MEDIUM-HIGH)

• `infer-use-filters-api` — Personalize backfills to numResults
• `infer-rerank-rules-after-model` — preserve the model distribution
• `infer-deduplicate-canonical-entity` — provider-level dedup, not listing-level
• `infer-enforce-exposure-caps` — rolling fairness constraints
• `infer-cache-responses-short-ttl` — session continuity and cost control

9. Observability and Online Metrics (MEDIUM-HIGH)

• `obs-always-ab-test` — before-and-after is never enough
• `obs-track-coverage-and-gini` — exposure-health signals
• `obs-slice-metrics-by-segment` — aggregate metrics hide segment regressions
• `obs-watch-online-offline-divergence` — proxy overfitting detector
• `obs-alarm-on-prediction-drift` — distribution KL-divergence as early warning

Planning and Improving Recommendations

Two playbooks drive end-to-end workflows that compose the rules above:

• `references/playbooks/planning.md` — Plan a new recommender system from scratch: a nine-step workflow that starts with instrumentation and ends with the first A/B-tested ML lift over a popularity baseline.
• `references/playbooks/improving.md` — Diagnose and improve an existing recommender: a decision tree that identifies the current bottleneck (telemetry, freshness, coverage, feedback loop, algorithm) and routes to the specific rules that fix it.

Read the playbooks first when the task is "design a recommender" or "this recommender is underperforming". Read the individual rules when a specific question arises during implementation or review.

How to Use

• Read `references/_sections.md` for category structure and impact ordering.
• Read individual rule files under references/ when a specific rule matches the task at hand.
• Read `references/playbooks/planning.md` to design a new system.
• Read `references/playbooks/improving.md` to diagnose an existing system.
• Use `assets/templates/_template.md` to author new rules as the skill grows.

Reference Files

Two-Sided Personalisation

Version 0.1.0 Marketplace Engineering April 2026

Note:

This document is mainly for agents and LLMs to follow when maintaining,

generating, or refactoring codebases. Humans may also find it useful,

but guidance here is optimized for automation and consistency by AI-assisted workflows.

---

Abstract

Comprehensive guide for designing, building and improving personalisation and recommendation systems in two-sided trust marketplaces on AWS Personalize. Contains 49 rules across 9 categories, ordered by cascade impact on the personalisation lifecycle — from event tracking and schema design through two-sided matching, cold start, feedback loops, bias control, recipe selection, serving-time re-ranking, and observability. Includes two playbooks that walk through planning a new recommender from scratch and diagnosing an existing one against common failure modes (instrumentation gaps, stale metadata, winner-take-all, death spirals, training-serving skew).

---

Table of Contents

1. Event Tracking and Capture — CRITICAL

• 1.1 Capture Negative Signals Explicitly — CRITICAL (prevents silence-as-acceptance bias)
• 1.2 Log Impressions Alongside Clicks — CRITICAL (enables unbiased CTR training)
• 1.3 Stamp Events with a Request-ID Join Key — CRITICAL (enables impression-to-outcome attribution)
• 1.4 Stream Events via PutEvents in Real Time — CRITICAL (1-2 second recommendation adaptation)
• 1.5 Track Outcomes to Completion, Not Clicks — CRITICAL (prevents clickbait reward shaping)
• 1.6 Use Stable Opaque Item IDs — CRITICAL (prevents history loss on listing rename)

2. Dataset and Schema Design — CRITICAL

• 2.1 Design Schemas Conservatively Because They Are Immutable — CRITICAL (avoids full dataset rebuild)
• 2.2 Enforce Metadata Freshness as a First-Class Signal — CRITICAL (prevents stale price and availability recommendations)
• 2.3 Include Context Fields in Training and Inference — HIGH (prevents training-serving feature divergence)
• 2.4 Keep User and Item Metadata Thin and Stable — CRITICAL (prevents training-serving skew)
• 2.5 Meet the AWS Personalize Minimum Dataset Sizes Before Training — HIGH (prevents training on below-threshold data)
• 2.6 Prefer Categorical Fields over Free Text — HIGH (enables per-value learned features)
• 2.7 Use EVENT_VALUE to Weight Outcomes over Clicks — HIGH (enables outcome-weighted training)

3. Two-Sided Matching Patterns — CRITICAL

• 3.1 Balance Supply and Demand per Segment — HIGH (prevents segment liquidity collapse)
• 3.2 Cap Provider Exposure to Prevent Winner-Take-All — CRITICAL (prevents supply monopolisation)
• 3.3 Filter Infeasible Candidates Before Ranking — CRITICAL (reduces wasted model capacity on impossible candidates)
• 3.4 Model Capacity Constraints at Rank Time — HIGH (prevents over-saturation of popular providers)
• 3.5 Rank by Mutual Fit, Not One Side — CRITICAL (3-5% booking lift per Airbnb study)

4. Simple Baselines and Theory of Constraints — HIGH

• 4.1 Audit Instrumentation Before Any Model Work — HIGH (prevents building on broken telemetry)
• 4.2 Budget Each Model with a Ship or Kill Criterion — HIGH (prevents indefinite incubation of dead experiments)
• 4.3 Find the Bottleneck Before Optimizing — HIGH (prevents work on non-bottleneck stages)
• 4.4 Measure the Gap to Baseline on Every Change — HIGH (prevents accidental regression against popularity)
• 4.5 Ship a Popularity Baseline Before ML — HIGH (reduces premature ML spend by 100%)
• 4.6 Use Heuristic Re-ranking for Cold Cohorts — HIGH (enables useful ranking with zero interactions)

5. Feedback Loops and Bias Control — HIGH

• 5.1 Decay Event Weights over Time — HIGH (prevents stale preferences dominating)
• 5.2 Detect Popularity Death Spirals via Top-N Gini — HIGH (prevents silent concentration collapse)
• 5.3 Log the Ranking Slot with Every Impression — HIGH (enables position-bias correction)
• 5.4 Optimize for Completed Outcome, Not Click — HIGH (prevents clickbait reward in feedback loop)
• 5.5 Reserve a Random Exploration Slice for Unbiased Training — HIGH (enables counterfactual evaluation)

6. Cold Start and Coverage — HIGH

• 6.1 Capture Explicit Intent at Onboarding — HIGH (saves days of interaction accumulation)
• 6.2 Reserve Exploration Slots for New Inventory — HIGH (enables new-listing discovery)
• 6.3 Tag Cold-Start Recommendations for Separate Measurement — HIGH (enables warm-vs-cold cohort comparison)
• 6.4 Use Best-of-Segment Popularity for New Users — HIGH (prevents global-popularity blandness)
• 6.5 Use USER_PERSONALIZATION_v2 with Rich Item Metadata — HIGH (enables same-day relevance for new listings)

7. Recipe and Pipeline Selection — MEDIUM-HIGH

• 7.1 Build a Candidate-Generation and Re-rank Pipeline — MEDIUM-HIGH (enables business rules and personalization to coexist)
• 7.2 Default to USER_PERSONALIZATION_v2 for Discovery — MEDIUM-HIGH (enables 5 million item catalog with lower latency)
• 7.3 Defer HPO Until the Baseline Is Measured — MEDIUM (prevents wasted training spend)
• 7.4 Use PERSONALIZED_RANKING_v2 as a Re-ranker, Not a Generator — MEDIUM-HIGH (enables business-rule compatible ranking)
• 7.5 Use SIMS Only for Item-Page Similar Recommendations — MEDIUM-HIGH (prevents user-history waste on item-page surfaces)

8. Inference, Filters and Re-ranking — MEDIUM-HIGH

• 8.1 Apply Business Rules After Model Scoring, Not Before — MEDIUM-HIGH (preserves model distribution information)
• 8.2 Cache Responses by User Context with a Short TTL — MEDIUM (reduces duplicate inference calls)
• 8.3 Deduplicate by Canonical Entity Before Returning — MEDIUM-HIGH (prevents duplicate-entity erosion of trust)
• 8.4 Enforce Provider Exposure Caps at Inference — MEDIUM-HIGH (prevents supply-side concentration at inference)
• 8.5 Use the Filters API for Hard Exclusions, Not Client Code — MEDIUM-HIGH (prevents numResults shortfall on exclusion)

9. Observability and Online Metrics — MEDIUM-HIGH

• 9.1 Alarm on Prediction Distribution Drift — MEDIUM (prevents silent model staleness)
• 9.2 Always A/B Test, Never Before-and-After — MEDIUM-HIGH (prevents confounding with seasonality)
• 9.3 Slice Metrics by User Segment — MEDIUM-HIGH (prevents aggregate-hides-regression failures)
• 9.4 Track Coverage and Exposure Gini — MEDIUM-HIGH (enables death-spiral detection)
• 9.5 Watch for Online and Offline Metric Divergence — MEDIUM-HIGH (prevents proxy-metric overfitting)

---

References

1. https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-user-personalization-v2.html 2. https://docs.aws.amazon.com/personalize/latest/dg/working-with-predefined-recipes.html 3. https://docs.aws.amazon.com/personalize/latest/dg/recording-events.html 4. https://docs.aws.amazon.com/personalize/latest/dg/custom-datasets-and-schemas.html 5. https://docs.aws.amazon.com/personalize/latest/dg/event-values-types.html 6. https://docs.aws.amazon.com/personalize/latest/dg/optimizing-solution-events-config.html 7. https://docs.aws.amazon.com/personalize/latest/dg/interactions-dataset-requirements.html 8. https://docs.aws.amazon.com/personalize/latest/dg/item-dataset-requirements.html 9. https://docs.aws.amazon.com/personalize/latest/dg/updating-dataset-schema.html 10. https://docs.aws.amazon.com/personalize/latest/dg/frequently-asked-questions.html 11. https://arxiv.org/abs/1205.2618 12. https://docs.aws.amazon.com/personalize/latest/dg/recommendations.html 13. https://aws.amazon.com/blogs/machine-learning/recommend-and-dynamically-filter-items-based-on-user-context-in-amazon-personalize/ 14. https://github.com/aws-samples/amazon-personalize-samples/blob/master/PersonalizeCheatSheet2.0.md 15. https://www.kdd.org/kdd2018/accepted-papers/view/real-time-personalization-using-embeddings-for-search-ranking-at-airbnb 16. https://medium.com/airbnb-engineering/how-airbnb-uses-machine-learning-to-detect-host-preferences-18ce07150fa3 17. https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789 18. https://medium.com/airbnb-engineering/learning-market-dynamics-for-optimal-pricing-97cffbcc53e3 19. https://careersatdoordash.com/blog/homepage-recommendation-with-exploitation-and-exploration/ 20. https://careersatdoordash.com/blog/doordash-kdd-llm-assisted-personalization-framework/ 21. https://pubsonline.informs.org/doi/10.1287/mksc.2022.0238 22. https://dl.acm.org/doi/10.1145/3712292 23. https://arxiv.org/pdf/2010.03240 24. https://developers.google.com/machine-learning/guides/rules-of-ml 25. https://developers.google.com/machine-learning/recommendation/overview 26. https://experimentguide.com/

---

Source Files

This document was compiled from individual reference files. For detailed editing or extension:

{Action-Oriented Rule Title in Title Case}

{One-to-three sentences explaining WHY this rule matters. Focus on the cascade effect — what goes wrong when the rule is not followed, and how the problem propagates to every downstream stage. This is the most important part of the rule; LLMs generalise better from understood reasoning than from dictation. Do not say "always do X" — explain what happens when you do not, in concrete terms the model can internalise.}

Incorrect ({specific failure mode in 3-6 words}):

```{language}

Production-realistic bad code — not a strawman.

Keep it under 20 lines. Use realistic names like seeker, provider,

listing, booking, requestId — never foo, bar, MyComponent, doSomething.

**Correct ({specific benefit in 3-6 words}):**

Production-realistic good code that differs minimally from the incorrect version —

only the key insight changes. Same variable names, same structure, different

behaviour where it matters. Under 20 lines.

Reference: [{Authoritative Source Title}]({https://url-to-primary-source})

## Authoring Checklist

Before saving, verify:

- [ ] Frontmatter has `title`, `impact`, `impactDescription`, `tags`
- [ ] First tag matches the category prefix from `_sections.md`
- [ ] Impact level is consistent with sibling rules in the same category
- [ ] Impact description is quantified (contains a number, verb like `prevents` / `reduces` / `enables`, or `O(n)` notation)
- [ ] The H2 title matches the frontmatter title exactly
- [ ] Both code blocks have a language specifier (```typescript, ```python, ```json)
- [ ] Both annotations `(failure mode)` and `(benefit)` are specific, not `(bad)` or `(good)`
- [ ] No vague language: `might want`, `perhaps`, `it is recommended`, `you may want`
- [ ] No marketing language: `powerful`, `seamless`, `amazing`, `blazing fast`
- [ ] No generic names: `foo`, `bar`, `MyComponent`, `doSomething`, `processData`
- [ ] Reference is from primary maintainer, peer-reviewed research, or production engineering blog
- [ ] Rule validates with `node scripts/validate-skill.js {skill-dir}` with no new errors

Marketplace Personalisation Skill

Best-practices skill for designing, building and improving personalisation and recommendation systems in two-sided trust marketplaces on AWS Personalize.

Overview

This skill is a distillation of authoritative guidance from AWS Personalize documentation, academic research on recommender bias, and production engineering blogs from two-sided marketplaces. It contains 49 rules across 9 categories, ordered by cascade impact on the personalisation lifecycle, and two playbooks for end-to-end planning and diagnostic workflows.

Structure

marketplace-personalisation/
├── SKILL.md                    # Entry point with category index and quick reference
├── AGENTS.md                   # Compiled navigation document (built by script)
├── metadata.json               # Version, discipline, authoritative references
├── README.md                   # This file
├── references/
│   ├── _sections.md            # Category definitions and impact ordering
│   ├── track-*.md              # Event Tracking and Capture (6 rules)
│   ├── schema-*.md             # Dataset and Schema Design (7 rules)
│   ├── match-*.md              # Two-Sided Matching Patterns (5 rules)
│   ├── simple-*.md             # Simple Baselines and Theory of Constraints (6 rules)
│   ├── loop-*.md               # Feedback Loops and Bias Control (5 rules)
│   ├── cold-*.md               # Cold Start and Coverage (5 rules)
│   ├── recipe-*.md             # Recipe and Pipeline Selection (5 rules)
│   ├── infer-*.md              # Inference, Filters and Re-ranking (5 rules)
│   ├── obs-*.md                # Observability and Online Metrics (5 rules)
│   └── playbooks/
│       ├── planning.md         # Plan a new recommender from scratch
│       └── improving.md        # Diagnose and improve an existing one
└── assets/
    └── templates/
        └── _template.md        # Template for authoring new rules

Getting Started

From the repo root, install plugin dependencies and run the skill validator:

pnpm install
pnpm build
pnpm validate

The validator runs structural and substance checks against the skill:

node scripts/validate-skill.js skills/.experimental/marketplace-personalisation

Build the compiled navigation document:

node scripts/build-agents-md.js skills/.experimental/marketplace-personalisation

Creating a New Rule

Rules go in references/ with a filename of the form {prefix}-{slug}.md, where {prefix} matches an existing category in references/_sections.md. Copy assets/templates/_template.md as a starting point and fill in the frontmatter and body.

A rule must include:

• YAML frontmatter: title, impact, impactDescription, tags (first tag is the prefix)
• One-to-three sentence explanation of why the rule matters and its cascade effect
• An **Incorrect (specific description):** code block with production-realistic code
• A **Correct (specific description):** code block that differs minimally from the incorrect
• A Reference: line linking to an authoritative source

Run pnpm validate after adding or editing rules.

Rule File Structure

Each rule has a strict structure enforced by the validator:

---
title: Use Exposure Caps Across Providers
impact: HIGH
impactDescription: prevents supply monopolisation
tags: match, fairness, exposure-cap
---

## Use Exposure Caps Across Providers

Explanation paragraph — why the rule matters, what goes wrong without it,
and how the cascade effect plays out downstream.

**Incorrect (concrete failure mode):**

Production-realistic bad example

**Correct (concrete solution):**

Production-realistic good example

Reference: [Source Title](https://example.com/source)

File Naming Convention

• Skill directory: kebab-case matching the skill name (marketplace-personalisation)
• Rule files: {category-prefix}-{slug}.md with kebab-case slugs (match-rank-mutual-fit.md)
• Playbook files: references/playbooks/{name}.md
• Templates: assets/templates/_template.md (underscore prefix to exclude from rule listings)
• Category prefixes are 3-8 lowercase letters and defined once in _sections.md

Impact Levels

Categories and rules use six impact levels ordered from highest to lowest cascade impact:

Target distribution for a 40-60 rule distillation: 2-3 CRITICAL categories, 2-4 HIGH, the rest MEDIUM or lower. This skill has 3 CRITICAL, 3 HIGH and 3 MEDIUM-HIGH categories.

Scripts

The dev-skill plugin provides two scripts used by this skill:

• scripts/validate-skill.js — runs structural and substance validation (required before shipping)
• scripts/build-agents-md.js — compiles the navigation document (never write AGENTS.md manually)

Example invocations:

node scripts/validate-skill.js skills/.experimental/marketplace-personalisation
node scripts/validate-skill.js skills/.experimental/marketplace-personalisation --sections-only
node scripts/build-agents-md.js skills/.experimental/marketplace-personalisation

Contributing

• New rules must follow the structure above and pass pnpm validate with zero errors
• Every rule must have incorrect and correct examples with specific annotations
• References must be from primary maintainers, peer-reviewed research, or engineering blogs with data
• Avoid hedging language (might, perhaps, it is recommended) — use imperative form
• Quantify impact where possible (2-10×, 200ms, prevents stale closures, O(n) to O(1))
• Playbooks in references/playbooks/ compose rules into end-to-end workflows
• Never edit AGENTS.md manually — it is regenerated by build-agents-md.js

Sections

This file defines all sections, their ordering, impact levels, and descriptions. The section ID (in parentheses) is the filename prefix used to group rules.

Categories are ordered by cascade impact on the personalisation lifecycle: problems at earlier stages poison every downstream stage. Tracking mistakes cannot be recovered from; schema mistakes force a full dataset rebuild; matching-design mistakes produce a system that is technically working but operationally wrong.

---

1. Event Tracking and Capture (track)

Impact: CRITICAL Description: Instrumentation is the foundation of every downstream stage — without logged impressions, stable IDs, outcome events, and join keys, no model, filter, or metric is recoverable.

2. Dataset and Schema Design (schema)

Impact: CRITICAL Description: Personalize schemas are effectively immutable, so early shape decisions constrain every future solution — bad schemas force a full dataset rebuild rather than an incremental fix.

3. Two-Sided Matching Patterns (match)

Impact: CRITICAL Description: Marketplace matching must respect both sides' preferences, feasibility constraints, and provider capacity — monolithic one-sided ranking produces winner-take-all dynamics that erode supply quality and long-term liquidity.

4. Simple Baselines and Theory of Constraints (simple)

Impact: HIGH Description: The bottleneck in a recommender is rarely the algorithm — it is usually instrumentation, coverage, or freshness, so simple baselines measure the gap before complexity is added and protect against over-engineering.

5. Feedback Loops and Bias Control (loop)

Impact: HIGH Description: Selection bias, positional bias, and popularity death spirals compound silently if feedback signals are not instrumented, exploration is not reserved, and the system is not optimising for the real downstream outcome.

6. Cold Start and Coverage (cold)

Impact: HIGH Description: New providers and new seekers arrive continuously in a marketplace, so cold-start handling via metadata-based recipes, explicit onboarding intent, and exploration slots determines whether inventory gets discovered at all.

7. Recipe and Pipeline Selection (recipe)

Impact: MEDIUM-HIGH Description: Choosing the correct recipe and pipeline shape — USER_PERSONALIZATION_v2 for discovery, SIMS for item-page similarity, PERSONALIZED_RANKING_v2 as a re-ranker — matches the algorithm to the problem and avoids training cost without justified lift.

8. Inference, Filters and Re-ranking (infer)

Impact: MEDIUM-HIGH Description: Serving-time correctness depends on applying hard exclusions via the Filters API, deduplicating by canonical entity, enforcing fairness caps, and caching responses — business rules must run after model scoring, not before.

9. Observability and Online Metrics (obs)

Impact: MEDIUM-HIGH Description: Online and offline metrics diverge silently, coverage collapses invisibly, and distribution drift goes unnoticed unless A/B tests, segment-sliced metrics, and exposure-health signals are first-class infrastructure.

Use Best-of-Segment Popularity for New Users

A global popularity fallback for a new seeker shows the same top listings to everyone — blandness that erases any chance of finding their niche. Best-of-segment popularity partitions the catalogue by a cheap signal (device locale, referral source, declared region, pet species) and shows the popular items within that segment. It is still heuristic, still cheap, but already personalised at the segment level and closes most of the gap to a fully-trained model for the first few sessions.

Incorrect (global top-24 — every new seeker sees identical content):

def new_user_homefeed(seeker: Seeker) -> list[Listing]:
    return catalog.top_by_completed_bookings(window_days=30, limit=24)

Correct (segmentation by declared intent and referral locale):

def new_user_homefeed(seeker: Seeker) -> list[Listing]:
    segment = (
        seeker.declared_region or seeker.geoip_region,
        seeker.declared_pet_species,
        seeker.referral_source,
    )
    cohort_top = catalog.top_by_completed_bookings_in_segment(
        region=segment[0],
        species=segment[1],
        referral=segment[2],
        window_days=30,
        limit=24,
    )
    if len(cohort_top) < 12:
        cohort_top.extend(catalog.top_by_completed_bookings(window_days=30, limit=24))
    return cohort_top[:24]

Reference: Google — Recommendations: What and Why?

Capture Explicit Intent at Onboarding

A seeker who just registered has zero interactions, but they can tell the system what they want — in one or two well-designed onboarding screens — if you ask. Explicit intent capture (region, date range, species, budget, trip type) seeds the user profile with stronger signals than clicks would provide in the first ten sessions. These declared preferences go straight into the Users dataset and into every GetRecommendations call's context block, so the first homepage is already differentiated rather than generic.

Incorrect (new seeker registered with only auth fields, no intent captured):

def on_signup(email: str, locale: str) -> Seeker:
    seeker = Seeker(id=str(uuid4()), email=email, locale=locale)
    seekers.save(seeker)
    return seeker

Correct (two-question onboarding feeds Users dataset and inference context):

def on_signup(email: str, locale: str, onboarding: OnboardingAnswers) -> Seeker:
    seeker = Seeker(
        id=str(uuid4()),
        email=email,
        locale=locale,
        declared_region=onboarding.region,
        declared_species=onboarding.species,
        declared_trip_type=onboarding.trip_type,
    )
    seekers.save(seeker)
    personalize_events.put_users(
        datasetArn=USERS_DATASET_ARN,
        users=[{
            "userId": seeker.id,
            "properties": json.dumps({
                "REGION": seeker.declared_region,
                "SPECIES": seeker.declared_species,
                "TRIP_TYPE": seeker.declared_trip_type,
            }),
        }],
    )
    return seeker

Reference: AWS Personalize — Handling New Users with PutUsers

Reserve Exploration Slots for New Inventory

A newly created listing has no interaction history, so a pure-relevance ranker never surfaces it, so it never accumulates interactions, so the ranker keeps ignoring it. Reserving a fixed fraction of slots (or injecting a new-listing promotion filter) gives new inventory a guaranteed chance to be discovered. Personalize supports this directly via promotion filters on recommendation requests — the promotion filter selects items by CREATION_TIMESTAMP and reserves a percentage of the output for matches.

Incorrect (straight ranker output, new inventory never surfaces):

response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker.id,
    numResults=24,
)

Correct (20% of slots reserved for listings created in the last 14 days):

response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker.id,
    numResults=24,
    promotions=[{
        "name": "fresh-listings",
        "percentPromotedItems": 20,
        "filterArn": FILTER_ARN_NEW_LISTINGS,
        "filterValues": {
            "MIN_CREATION_TIMESTAMP": str(int((datetime.utcnow() - timedelta(days=14)).timestamp())),
        },
    }],
)

Reference: AWS Personalize — Recommend and Dynamically Filter Based on User Context

Tag Cold-Start Recommendations for Separate Measurement

Aggregate CTR and booking-rate metrics hide a catastrophic truth: warm cohorts can be lifting while cold cohorts collapse, and the blended number looks flat. Tagging every cold-start response with an explicit cold_start=true property — and emitting the tag into every impression event — lets you slice every online metric by warmth and catch a cold-cohort regression before it drags down the aggregate. The tag is also what lets you A/B different cold-start strategies against each other.

Incorrect (no warmth tag — cold and warm cohorts are indistinguishable in metrics):

def homefeed(seeker: Seeker) -> list[Listing]:
    if seeker.lifetime_events < 5:
        return best_of_segment_popularity(seeker)
    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
    )
    return hydrate_listings(response["itemList"])

Correct (warmth tag propagated into every downstream event):

def homefeed(seeker: Seeker, request_id: str) -> list[Listing]:
    if seeker.lifetime_events < 5:
        listings = best_of_segment_popularity(seeker)
        log_exposure(request_id, listings, cold_start=True, policy="segment_popularity")
        return listings

    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
    )
    listings = hydrate_listings(response["itemList"])
    log_exposure(request_id, listings, cold_start=False, policy="personalize_v2")
    return listings

Reference: Airbnb — Machine Learning-Powered Search Ranking of Airbnb Experiences

Use USER_PERSONALIZATION_v2 with Rich Item Metadata

New listings enter the catalog constantly — they have no interaction history, so a collaborative-filtering model has no basis for ranking them beyond global popularity. USER_PERSONALIZATION_v2 combines interaction signals with rich item metadata, which lets the model extrapolate from attributes (region, category, price tier, verification level) before interactions accumulate. The weaker the item metadata, the longer the cold-start penalty; the richer the metadata, the faster a new listing earns its first relevance signal.

Incorrect (Items dataset has only ID and category — cold items invisible):

{
  "type": "record",
  "name": "Items",
  "fields": [
    { "name": "ITEM_ID", "type": "string" },
    { "name": "CATEGORY", "type": "string", "categorical": true }
  ]
}

Correct (rich categorical metadata lets v2 extrapolate to new listings):

{
  "type": "record",
  "name": "Items",
  "fields": [
    { "name": "ITEM_ID", "type": "string" },
    { "name": "CATEGORY", "type": "string", "categorical": true },
    { "name": "REGION", "type": "string", "categorical": true },
    { "name": "PRICE_TIER", "type": "string", "categorical": true },
    { "name": "VERIFICATION_LEVEL", "type": "string", "categorical": true },
    { "name": "ACCEPTS_SPECIES", "type": "string", "categorical": true },
    { "name": "CREATION_TIMESTAMP", "type": "long" }
  ]
}

Reference: AWS Personalize — User-Personalization-v2 Recipe

Cache Responses by User Context with a Short TTL

A seeker who reloads the homepage three times in ten seconds does not need three fresh inference calls — the recommendations should stay stable within a session. Caching GetRecommendations responses by a composite key of (userId, surface, context) with a short TTL (30-120 seconds) reduces Personalize cost, preserves session continuity (the same listings appear in the same order) and cuts latency. The TTL must be short enough that a real preference change (booking, dismissal) invalidates the cache within a reasonable window.

Incorrect (every request fires a fresh GetRecommendations call):

def homefeed(seeker: Seeker, surface: str) -> list[Listing]:
    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
        context={"SURFACE": surface},
    )
    return hydrate_listings(response["itemList"])

Correct (short-TTL cache keyed on user + surface + context hash):

def homefeed(seeker: Seeker, surface: str) -> list[Listing]:
    cache_key = f"rec:{seeker.id}:{surface}:{hash_context(seeker)}"
    cached = redis.get(cache_key)
    if cached:
        return deserialize_listings(cached)

    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
        context={"SURFACE": surface},
    )
    listings = hydrate_listings(response["itemList"])
    redis.setex(cache_key, 60, serialize_listings(listings))
    return listings

Reference: AWS Personalize — Real-time Item Recommendations

Deduplicate by Canonical Entity Before Returning

A seeker who sees the same provider appear in slots 1, 3 and 7 under three different listing variants will notice — and the system will look broken. Deduplication must happen on the canonical entity (provider, household, legal entity) not just the listing ID, because a provider often has multiple sub-listings that map to the same real-world resource. Deduplication runs at the tail of the inference pipeline, after model scoring and before the response is returned.

Incorrect (no deduplication, same provider dominates top-24):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    ranked = rank_with_personalize(seeker, feasible)
    return ranked[:24]

Correct (deduplicate by provider_id while preserving order):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    ranked = rank_with_personalize(seeker, feasible)

    seen_providers: set[str] = set()
    deduped: list[Listing] = []
    for listing in ranked:
        if listing.provider_id in seen_providers:
            continue
        deduped.append(listing)
        seen_providers.add(listing.provider_id)
        if len(deduped) == 24:
            break
    return deduped

Reference: Airbnb — Machine Learning-Powered Search Ranking of Airbnb Experiences

Enforce Provider Exposure Caps at Inference

Even with deduplication by canonical entity, a small set of providers can capture a disproportionate share of impressions across sessions — not in any single response, but in aggregate over a day or a region. A rolling-window exposure cap at the inference layer (e.g., "no provider appears in more than 10% of responses from a region in the last hour") is a global fairness constraint that protects long-tail supply from being starved by the most popular providers. This cap is enforced by the inference layer, not by the model.

Incorrect (no rolling exposure tracking, short-term monopolisation possible):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    ranked = rank_with_personalize(seeker, feasible)
    return dedupe_by_provider(ranked)[:24]

Correct (rolling exposure map filters saturated providers at inference):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    ranked = rank_with_personalize(seeker, feasible)

    window_exposure = exposure_tracker.recent_share(
        region=seeker.current_region,
        window_minutes=60,
    )
    allowed = [
        listing for listing in ranked
        if window_exposure.get(listing.provider_id, 0.0) < 0.10
    ]
    return dedupe_by_provider(allowed)[:24]

Reference: Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical Approach

Apply Business Rules After Model Scoring, Not Before

Soft business rules (a provider's promotion boost, a fresh-listing bonus, a strategic category uplift) must run after the model has scored the candidates — not before. Applying them before model scoring biases the candidate set and the model loses the ability to learn what users actually prefer. Applying them after preserves the model's relevance signal and lets the business rule act as a transparent, tunable re-ordering layer that product teams can adjust without retraining.

Incorrect (boosting a category by injecting it into the candidate set):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    promoted = catalog.filter_by_category("verified_premium")
    candidates = list(set(feasible + promoted))
    ranked = rank_with_personalize(seeker, candidates)
    return ranked[:24]

Correct (soft business rules applied as a post-hoc score adjustment):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = retrieve_feasible(seeker)
    scored = rank_with_personalize(seeker, feasible)

    for listing in scored:
        if listing.category == "verified_premium":
            listing.score *= 1.15
        if listing.created_at > datetime.utcnow() - timedelta(days=7):
            listing.score *= 1.10

    return sorted(scored, key=lambda listing: -listing.score)[:24]

Reference: DoorDash — Homepage Recommendation with Exploitation and Exploration

Use the Filters API for Hard Exclusions, Not Client Code

Filtering out already-booked or blocked listings in the client means Personalize returns 24 items and the client discards half, leaving gaps in the response. The Filters API applies the exclusion during retrieval — Personalize knows what was filtered, backfills with additional candidates, and returns a full numResults list. Filter DSL supports dynamic parameters so the same filter works for multiple users, and filter expressions can reference both Items and Interactions data.

Incorrect (client-side filtering, visible gaps in response):

response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker.id,
    numResults=24,
)
listings = hydrate_listings(response["itemList"])
already_booked_ids = bookings.completed_listing_ids(seeker.id)
listings = [l for l in listings if l.id not in already_booked_ids]
# Now returning ~13 items instead of 24 — UI gap

Correct (server-side Filters API backfills to numResults):

response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker.id,
    numResults=24,
    filterArn=EXCLUDE_ALREADY_BOOKED_FILTER_ARN,
    filterValues={
        "SEEKER_ID": seeker.id,
    },
)
listings = hydrate_listings(response["itemList"])

Reference: AWS Personalize — Recommend and Dynamically Filter Based on User Context

Decay Event Weights over Time

A booking from eighteen months ago is weak evidence that a seeker still wants the same thing today — life circumstances change, preferences shift, the market moves. Without time decay, old events pile up and dominate the training signal, so the model anchors on long-gone preferences. Decaying weight with event age (exponential or bucketed) focuses the model on what the seeker currently wants while still using old events for structural patterns.

Incorrect (full training window with uniform event weight):

def build_training_interactions(events: Iterable[Event]) -> list[dict]:
    return [
        {
            "USER_ID": e.user_id,
            "ITEM_ID": e.item_id,
            "TIMESTAMP": int(e.timestamp.timestamp()),
            "EVENT_TYPE": e.event_type,
            "EVENT_VALUE": 1.0,
        }
        for e in events
    ]

Correct (exponential time decay applied via EVENT_VALUE):

def build_training_interactions(events: Iterable[Event], now: datetime) -> list[dict]:
    half_life_days = 90
    return [
        {
            "USER_ID": e.user_id,
            "ITEM_ID": e.item_id,
            "TIMESTAMP": int(e.timestamp.timestamp()),
            "EVENT_TYPE": e.event_type,
            "EVENT_VALUE": float(0.5 ** ((now - e.timestamp).days / half_life_days)),
        }
        for e in events
    ]

Reference: Google — Rules of Machine Learning, Rule 32: Re-use Code Between Training and Serving Pipelines

Detect Popularity Death Spirals via Top-N Gini

A popularity death spiral is silent: the model ranks popular items higher, they get more impressions, more clicks, rise in the training data, and the next generation ranks them even higher — until the long tail is invisible and coverage collapses. Tracking the Gini coefficient of top-N exposure as a health signal catches this before it shows up in booking rates. A monotonically rising Gini over weeks is the death-spiral fingerprint — alert on it and inject exploration immediately.

Incorrect (no concentration metric, collapse goes unnoticed):

def weekly_recommender_health_check() -> None:
    metrics = dashboard.fetch(["ctr", "booking_rate", "session_length"])
    alert_on_regression(metrics)

Correct (exposure Gini tracked weekly, alerts on monotonic rise):

def weekly_recommender_health_check() -> None:
    metrics = dashboard.fetch(["ctr", "booking_rate", "session_length"])
    alert_on_regression(metrics)

    gini_series = dashboard.fetch_series("exposure_gini_top_24", weeks=6)
    if is_monotonically_increasing(gini_series) and gini_series[-1] > 0.65:
        pager.alert(
            "Exposure Gini rising for 6 consecutive weeks — likely death spiral",
            runbook="playbooks/improving.md#death-spiral",
        )

Reference: Bias and Debias in Recommender System: A Survey (arXiv 2010.03240)

Log the Ranking Slot with Every Impression

A click on slot 1 is not worth the same as a click on slot 24 — slot 1 is looked at 10× more often, so the same click rate in slot 24 represents stronger preference. Without logging the slot, the training data mixes strong and weak signals as if they were equal, and the model learns that whatever currently sits in slot 1 is the correct answer. Slot logging makes position-bias correction possible (via IPS weighting or similar) and is the cheapest single improvement you can make after impression logging itself.

Incorrect (impressions logged without slot — positional bias uncorrectable):

await personalize.putEvents({
  trackingId: env.PERSONALIZE_TRACKING_ID,
  userId: seeker.id,
  sessionId: seeker.sessionId,
  eventList: listings.map((listing) => ({
    eventType: 'impression',
    itemId: listing.id,
    sentAt: new Date(),
  })),
})

Correct (slot and surface travel with every impression):

await personalize.putEvents({
  trackingId: env.PERSONALIZE_TRACKING_ID,
  userId: seeker.id,
  sessionId: seeker.sessionId,
  eventList: listings.map((listing, slot) => ({
    eventType: 'impression',
    itemId: listing.id,
    sentAt: new Date(),
    properties: JSON.stringify({
      requestId,
      slot,
      surface: 'homefeed',
      page: Math.floor(slot / 24),
    }),
  })),
})

Reference: Bias and Debias in Recommender System: A Survey (arXiv 2010.03240)

Optimize for Completed Outcome, Not Click

What you reward is what you will get. If the training signal is dominated by clicks, the next model generation will rank for clickbait — eye-catching photos, aggressive pricing, sensational titles — even when those listings underperform on bookings. The feedback loop reinforces the proxy, not the goal. Setting event weights so that booking_completed dominates click at training time realigns the loop with the business outcome and compounds every retraining cycle.

Incorrect (implicit equal weight — the loop reinforces the click proxy):

solution_config = {
    "name": "homefeed-v2",
    "datasetGroupArn": DATASET_GROUP_ARN,
    "recipeArn": "arn:aws:personalize:::recipe/aws-user-personalization-v2",
}
personalize.create_solution(**solution_config)

Correct (explicit outcome-weighted training reinforces completion):

solution_config = {
    "name": "homefeed-v2",
    "datasetGroupArn": DATASET_GROUP_ARN,
    "recipeArn": "arn:aws:personalize:::recipe/aws-user-personalization-v2",
    "eventsConfig": {
        "eventParametersList": [
            {"eventType": "booking_completed", "weight": 12.0},
            {"eventType": "booking_request", "weight": 5.0},
            {"eventType": "click", "weight": 1.0},
            {"eventType": "dismiss", "weight": 0.1},
        ],
    },
}
personalize.create_solution(**solution_config)

Reference: AWS Personalize — Optimizing a Solution with Events Configuration

Reserve a Random Exploration Slice for Unbiased Training

Every deployed ranker biases its own training data — it only shows users what the current model thinks they want, so the model never learns what it would have missed. Reserving a small random-exploration slice (2-5% of requests) shows an unbiased slate with a known sampling probability, which is the only way to train subsequent models without inheriting the previous model's blind spots. The exploration slice is also the gold-standard slice for counterfactual policy evaluation.

Incorrect (all traffic routed through the model, feedback loop closed on itself):

def homefeed(seeker: Seeker) -> list[Listing]:
    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
    )
    return hydrate_listings(response["itemList"])

Correct (3% random exploration slice, logged with propensity):

def homefeed(seeker: Seeker, request_id: str) -> list[Listing]:
    if random.random() < 0.03:
        feasible = retrieve_feasible(seeker)
        shown = random.sample(feasible, k=min(24, len(feasible)))
        log_exposure(request_id, shown, policy="exploration", propensity=1 / len(feasible))
        return shown

    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
    )
    shown = hydrate_listings(response["itemList"])
    log_exposure(request_id, shown, policy="personalize", propensity=None)
    return shown

Reference: BPR: Bayesian Personalized Ranking from Implicit Feedback (Rendle et al., UAI 2009)

Balance Supply and Demand per Segment

Marketplace liquidity is not uniform — some city-date segments are demand-heavy (dozens of seekers chasing three providers) while others are supply-heavy (providers with empty calendars). A global ranking strategy treats both identically and makes the wrong trade-off: in demand-heavy segments it over-diversifies and in supply-heavy segments it under-explores. Measuring per-segment supply/demand ratio and routing to different ranking strategies restores segment-level liquidity.

Incorrect (one global ranking strategy for all segments):

def homefeed(seeker: Seeker, request: Request) -> list[Listing]:
    feasible = retrieve_feasible(seeker, request)
    return rank_default(seeker, feasible)[:24]

Correct (segment-aware routing to different ranking strategies):

def homefeed(seeker: Seeker, request: Request) -> list[Listing]:
    feasible = retrieve_feasible(seeker, request)
    segment = (request.region, request.date_range.iso_week)
    ratio = liquidity.supply_demand_ratio(segment)

    if ratio < 0.5:  # demand-heavy — aggressive diversification
        return rank_with_exposure_caps(seeker, feasible, cap_per_provider=1)[:24]
    if ratio > 3.0:  # supply-heavy — explore cold providers
        return rank_with_exploration(seeker, feasible, explore_rate=0.2)[:24]
    return rank_default(seeker, feasible)[:24]

Reference: Airbnb — Learning Market Dynamics for Optimal Pricing

Cap Provider Exposure to Prevent Winner-Take-All

Without exposure caps, a handful of highly-rated providers dominate every top-24 slot, their calendars saturate, remaining demand bounces to lower-ranked providers and the middle of the supply distribution starves of signal. Capping how often any single provider appears in a result set — or in a rolling window of requests — forces diversity at the top of the funnel and keeps secondary supply alive. This is a fairness constraint, not a penalty: it simply acknowledges that a provider who is already booked cannot absorb more demand.

Incorrect (no exposure cap, top provider monopolises results):

def ranked_homefeed(seeker: Seeker) -> list[Listing]:
    candidates = retrieve_feasible_listings(seeker)
    scored = score_with_personalize(seeker, candidates)
    return scored[:24]

Correct (round-robin cap of 2 listings per provider in the top 24):

def ranked_homefeed(seeker: Seeker) -> list[Listing]:
    candidates = retrieve_feasible_listings(seeker)
    scored = score_with_personalize(seeker, candidates)

    result: list[Listing] = []
    per_provider: dict[str, int] = {}
    for listing in scored:
        if per_provider.get(listing.provider_id, 0) >= 2:
            continue
        result.append(listing)
        per_provider[listing.provider_id] = per_provider.get(listing.provider_id, 0) + 1
        if len(result) == 24:
            break
    return result

Reference: DoorDash — Homepage Recommendation with Exploitation and Exploration

Filter Infeasible Candidates Before Ranking

A listing outside the seeker's travel radius, unavailable on their requested dates, or excluding their pet species cannot be booked — ranking such candidates wastes model capacity and clutters the output with false promises. Hard feasibility constraints (geography, availability, species, hard preferences) belong in a retrieval/candidate-generation step that runs before the ranker ever sees the listing. The ranker then optimises over a feasible set rather than learning "which infeasible listings do seekers ignore least often".

Incorrect (ranker sees all inventory, filters applied post-rank):

def homefeed(seeker: Seeker, request: Request) -> list[Listing]:
    all_listings = catalog.list_all_active()
    ranked = rank_listings(seeker, all_listings)
    return [
        listing for listing in ranked
        if listing.region == request.region
        and listing.is_available_on(request.date_range)
    ][:24]

Correct (retrieval applies hard constraints first, ranker re-scores the feasible set):

def homefeed(seeker: Seeker, request: Request) -> list[Listing]:
    feasible = catalog.search(
        region=request.region,
        available_on=request.date_range,
        accepts_species=seeker.pet_species,
    )
    ranked = rank_listings(seeker, feasible)
    return ranked[:24]

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking

Model Capacity Constraints at Rank Time

In a marketplace with finite provider capacity, a top-ranked provider becomes less valuable as their remaining slots fill — their marginal value decays with every booking. Ranking that ignores this treats a provider with one remaining slot identically to one with ten, oversells the popular supply and leaves the long tail invisible. The fix is capacity-aware scoring: discount each provider's score by a function of remaining capacity so rank naturally rotates toward open inventory.

Incorrect (static score, no awareness of remaining capacity):

def rank(seeker: Seeker, candidates: list[Listing]) -> list[Listing]:
    scores = {
        listing.id: predict_mutual_fit(seeker, listing)
        for listing in candidates
    }
    return sorted(candidates, key=lambda c: -scores[c.id])

Correct (capacity-discounted score, rotates toward open inventory):

def rank(seeker: Seeker, candidates: list[Listing]) -> list[Listing]:
    scores = {}
    for listing in candidates:
        base_score = predict_mutual_fit(seeker, listing)
        capacity_ratio = listing.remaining_slots / max(listing.total_slots, 1)
        scores[listing.id] = base_score * (0.3 + 0.7 * capacity_ratio)
    return sorted(candidates, key=lambda c: -scores[c.id])

Reference: Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical Approach

Rank by Mutual Fit, Not One Side

A marketplace where only the seeker's preferences drive ranking produces matches the seeker wants but the provider would reject — leading to declined requests, withdrawn listings and seeker frustration. The ranking objective must combine P(seeker likes provider) with P(provider accepts seeker); downweighting candidates with a high predicted decline rate removes dead-end matches from the top of the results. Airbnb reported a 3.75% booking conversion lift from incorporating host preferences into search ranking.

Incorrect (one-sided scoring, ignores provider decline probability):

def rank_listings(seeker: Seeker, candidates: list[Listing]) -> list[Listing]:
    scores = {
        listing.id: predict_seeker_affinity(seeker, listing)
        for listing in candidates
    }
    return sorted(candidates, key=lambda c: -scores[c.id])

Correct (mutual-fit objective combines both sides):

def rank_listings(seeker: Seeker, candidates: list[Listing]) -> list[Listing]:
    scores = {}
    for listing in candidates:
        seeker_affinity = predict_seeker_affinity(seeker, listing)
        accept_prob = predict_provider_accept(listing.provider, seeker)
        scores[listing.id] = seeker_affinity * accept_prob
    return sorted(candidates, key=lambda c: -scores[c.id])

Reference: Airbnb — Machine Learning to Detect Host Preferences

Alarm on Prediction Distribution Drift

A deployed model's prediction distribution (score histogram, category mix of top-24, average position of cold items) is stable in normal operation. When it shifts — a seasonal trend, a schema import glitch, a silent dataset corruption, a deployment regression — the business metrics take days to reflect the damage but the prediction distribution shifts within hours. Monitoring KL-divergence between today's distribution and a rolling reference distribution catches these failures early and is often the first indicator that something deeper is wrong.

Incorrect (only business metrics are monitored — prediction drift invisible):

def daily_health_check() -> None:
    metrics.check("booking_rate", threshold=-0.05)
    metrics.check("ctr", threshold=-0.05)

Correct (prediction distribution KL-divergence alarm alongside business metrics):

def daily_health_check() -> None:
    metrics.check("booking_rate", threshold=-0.05)
    metrics.check("ctr", threshold=-0.05)

    today_dist = predictions.distribution_histogram(day=date.today())
    baseline_dist = predictions.rolling_reference_distribution(window_days=14)
    divergence = kl_divergence(today_dist, baseline_dist)

    if divergence > 0.15:
        pager.alert(
            f"Prediction distribution drift: KL={divergence:.3f}",
            runbook="playbooks/improving.md#prediction-drift",
        )

Reference: Google — Rules of Machine Learning, Rule 38: Do Not Waste Time on New Features if Unaligned Objectives Become the Issue

Always A/B Test, Never Before-and-After

A before-and-after comparison conflates the change under test with every other thing that happened in the same window — seasonality, marketing campaigns, population shifts, supply changes, bug fixes. Only a randomised A/B split gives a causal estimate of the change's effect. Before-and-after is the most common source of "we shipped a model and booking rate went up 4%" claims that turn out to be coincidence when an A/B test is eventually run.

Incorrect (before-and-after comparison against last week):

def evaluate_new_model(model: str) -> Report:
    this_week = metrics.fetch(
        model=model,
        start=date.today() - timedelta(days=7),
        end=date.today(),
    )
    last_week = metrics.fetch(
        model="previous_production",
        start=date.today() - timedelta(days=14),
        end=date.today() - timedelta(days=7),
    )
    return Report(
        lift=(this_week.booking_rate - last_week.booking_rate) / last_week.booking_rate,
    )

Correct (randomised A/B with control and treatment in the same window):

def evaluate_new_model(treatment_model: str, control_model: str) -> Report:
    experiment = experiments.create(
        name=f"compare-{treatment_model}-vs-{control_model}",
        variants={"control": control_model, "treatment": treatment_model},
        allocation={"control": 0.5, "treatment": 0.5},
        primary_metric="booking_completed_per_session",
    )
    return experiment.wait_for_significance(
        min_sample_size=40_000,
        max_p_value=0.05,
    )

Reference: Trustworthy Online Controlled Experiments (Kohavi, Tang, Xu)

Slice Metrics by User Segment

Aggregate metrics hide segment-level regressions with alarming regularity: a new model can lift booking rate 3% overall while collapsing it 15% for first-time users whose volume happens to be small. Slicing every primary metric by user segment (new vs repeat, cold vs warm, by region, by device, by referral source) surfaces those regressions early. Simpson's paradox — where an aggregate lift hides a segment loss — is routine in recommender A/B tests and only slicing catches it.

Incorrect (only aggregate booking rate inspected — Simpson's paradox invisible):

def evaluate(experiment: Experiment) -> Decision:
    metrics = experiment.primary_metric()
    if metrics.treatment > metrics.control and metrics.p_value < 0.05:
        return Decision.SHIP
    return Decision.KILL

Correct (segment breakdown is a ship-blocker if any segment regresses):

def evaluate(experiment: Experiment) -> Decision:
    overall = experiment.primary_metric()
    if overall.treatment <= overall.control or overall.p_value >= 0.05:
        return Decision.KILL

    segments = ["new_users", "repeat_users", "cold_cohort", "warm_cohort"]
    for segment in segments:
        sliced = experiment.primary_metric(segment=segment)
        if sliced.treatment < sliced.control * 0.98:
            return Decision.INVESTIGATE
    return Decision.SHIP

Reference: Google — Rules of Machine Learning, Rule 28: Beware of Feedback Loops at Serving Time

Track Coverage and Exposure Gini

Coverage — the percentage of active inventory recommended at least once in a rolling window — and the Gini coefficient of exposure — how unequally impressions are distributed across items — are the two health signals that catch winner-take-all pathologies before users notice. A falling coverage with a rising Gini is the fingerprint of a death spiral. Dashboard these two metrics alongside CTR and booking rate; alerts on them fire earlier than business metrics and give the team time to inject exploration before damage spreads.

Incorrect (only CTR and booking rate tracked — death spiral invisible):

def publish_weekly_health() -> None:
    metrics.publish({
        "ctr": dashboard.fetch("ctr"),
        "booking_rate": dashboard.fetch("booking_rate"),
    })

Correct (coverage and Gini published alongside business metrics):

def publish_weekly_health() -> None:
    metrics.publish({
        "ctr": dashboard.fetch("ctr"),
        "booking_rate": dashboard.fetch("booking_rate"),
        "catalog_coverage_7d": dashboard.fetch("percent_items_recommended", window_days=7),
        "catalog_coverage_30d": dashboard.fetch("percent_items_recommended", window_days=30),
        "exposure_gini_top_24": dashboard.fetch("gini_coefficient_top_n", n=24),
        "provider_gini_7d": dashboard.fetch("provider_exposure_gini", window_days=7),
    })

Reference: Bias and Debias in Recommender System: A Survey (arXiv 2010.03240)

Watch for Online and Offline Metric Divergence

Offline metrics (precision@k, recall@k, NDCG on held-out history) are a proxy, not the truth — the truth is the online A/B test. A new solution version with higher offline AUC but flat online CTR is overfitting the proxy, not learning real preference. Tracking the divergence over successive solution versions (Δ offline-metric vs Δ online-metric) catches when the team is optimising the wrong thing. A persistent divergence is the signal to rebuild the offline evaluation against a held-out time window that reflects real use.

Incorrect (offline metric is the only gate for a model promotion):

def promote_if_better(candidate: SolutionVersion, current: SolutionVersion) -> None:
    if candidate.offline_metrics.auc > current.offline_metrics.auc:
        deploy_campaign(candidate)

Correct (online A/B is the final gate, divergence dashboarded):

def promote_if_better(candidate: SolutionVersion, current: SolutionVersion) -> None:
    if candidate.offline_metrics.auc <= current.offline_metrics.auc:
        return

    experiment = ab_test(
        control=current,
        treatment=candidate,
        primary_metric="booking_completed_per_session",
    )
    result = experiment.wait_for_significance()

    dashboard.record_divergence(
        offline_delta=candidate.offline_metrics.auc - current.offline_metrics.auc,
        online_delta=result.relative_lift,
        version=candidate.arn,
    )
    if result.relative_lift > 0.0 and result.p_value < 0.05:
        deploy_campaign(candidate)

Reference: Google — Rules of Machine Learning, Rule 36: Avoid Feedback Loops with Positional Features

Improvement Playbook: Diagnosing an Existing Recommender

This playbook is a decision tree that walks through diagnosing an existing two-sided marketplace recommender and choosing the next intervention. It applies the theory-of-constraints principle from `simple-find-bottleneck-first`: there is always exactly one bottleneck, work on any other layer is wasted, and finding the bottleneck takes hours while fixing the wrong thing takes weeks.

Use this playbook when:

• The recommender "mostly works" but online metrics are flat or declining
• A new experiment fails to beat the current production model
• A seeker complaint, a provider complaint, or a product manager intuition suggests something is wrong
• Online metrics have drifted without any deploy — silent regression
• The team is debating whether to tune the current model, retrain, or rewrite
• Planning the next quarter of recsys work and asking "what is the highest-leverage change"

The Diagnostic Sequence

Run the diagnostic in order. Each step is a cheap check (hours, not weeks) that either clears the layer or points to a specific rule. Do not skip steps — the bottleneck is almost always earlier than the team thinks it is.

   Start
     │
     ▼
  ┌─────────────────────────────────────────────────────┐
  │ 1. Telemetry audit                                   │
  │    → impression coverage, outcome coverage,          │
  │      requestId join rate, dismissal capture          │
  │    ← fix: track-* rules                              │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 2. Metadata freshness audit                          │
  │    → Items dataset p99 staleness                     │
  │    ← fix: schema-enforce-metadata-freshness          │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 3. Coverage and Gini audit                           │
  │    → catalog coverage, exposure Gini, top-N Gini     │
  │    ← fix: loop-detect-death-spirals, loop-reserve-*  │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 4. Baseline gap audit                                │
  │    → compare current model vs popularity baseline    │
  │    ← fix: simple-ship-popularity-baseline            │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 5. Two-sided fairness audit                          │
  │    → provider exposure distribution, decline rates   │
  │    ← fix: match-rank-mutual-fit, match-cap-*         │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 6. Segment regression audit                          │
  │    → cold/warm, new/repeat, per-region breakdown     │
  │    ← fix: cold-*, obs-slice-metrics-by-segment       │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 7. Online/offline divergence audit                   │
  │    → Δ offline AUC vs Δ online booking rate          │
  │    ← fix: obs-watch-online-offline-divergence        │
  └─────────────────────────────────────────────────────┘
     │
     ▼ pass
  ┌─────────────────────────────────────────────────────┐
  │ 8. Algorithm iteration                               │
  │    → recipe choice, HPO, pipeline changes            │
  │    ← fix: recipe-*, careful A/B                      │
  └─────────────────────────────────────────────────────┘

Every exit routes to specific rules. Steps 1-7 are cheap to run and typically surface the real bottleneck. Step 8 — algorithm work — is the last resort, not the first.

Step 1 — Telemetry Audit

Question: is the instrumentation actually recording what we think it is?

Run the audit from `simple-audit-before-build`:

If any check fails: stop here. Fix instrumentation before touching the model. A recommender trained on broken telemetry is confidently wrong, and every subsequent layer inherits the damage. Expect this step to be the bottleneck in 40-60% of diagnoses.

Step 2 — Metadata Freshness Audit

Question: is the Items dataset accurate, or does Personalize think listings are available when they are actually booked out?

Measure the p99 staleness of the Items dataset versus the source-of-truth catalog:

def items_freshness_p99() -> timedelta:
    listings = source_catalog.recent_active(limit=10_000)
    personalize_items = personalize_items_dataset.fetch_all()

    deltas = []
    for listing in listings:
        personalize_item = personalize_items.get(listing.id)
        if personalize_item is None:
            deltas.append(timedelta(days=365))
            continue
        deltas.append(listing.updated_at - personalize_item.updated_at)
    return percentile(deltas, 99)

Threshold: p99 staleness < 2 hours. Anything higher means Personalize is ranking ghosts — listings that look attractive in the dataset but are not really available.

If this check fails: fix per `schema-enforce-metadata-freshness` — wire PutItems to every metadata change event, not the weekly batch import.

Step 3 — Coverage and Gini Audit

Question: is the recommender in a death spiral?

Fetch the weekly exposure metrics:

If any check fails: inject exploration immediately — both a random exploration slice (see `loop-reserve-random-exploration`) and promotional slots for fresh inventory (see `cold-reserve-exploration-slots`). Death spirals compound — every week of delay makes recovery harder.

Step 4 — Baseline Gap Audit

Question: does the current production model still beat a popularity baseline?

Run a small A/B test against a retained popularity baseline per `simple-measure-gap-to-baseline`. If the baseline was retired months ago, rebuild it in a day and run the test.

Threshold: production model should lift booking-completed-per-session by ≥5% over popularity baseline, and the delta should not be declining over time.

If the check fails: the model has drifted below the baseline — likely due to accumulated changes in data shape, schema, or training configuration. Rollback to an earlier known-good solution version, or rebuild from a clean baseline.

Step 5 — Two-Sided Fairness Audit

Question: is the ranker producing one-sided results that providers will reject?

Compute two metrics:

If a check fails: wire provider-accept prediction into the ranking objective, enforce exposure caps, and apply capacity-discounted scoring. This is typically a 1-2 week change and produces measurable conversion lift at the provider-side level.

Step 6 — Segment Regression Audit

Question: is the aggregate metric hiding a segment regression?

Slice every primary metric (booking rate, CTR, session length, mutual-rating average) by:

• Cold cohort vs warm cohort (lifetime_events < 5 vs ≥ 5)
• New users vs repeat users (first_session vs ≥ 2 sessions)
• Demand-heavy segments vs supply-heavy segments
• By region, device, referral source

If any segment regresses ≥5% below aggregate: that segment is the bottleneck. For cold cohorts, route to cold-* rules. For specific regions, check supply/demand ratio and consider segment-aware ranking per `match-balance-supply-demand`.

Step 7 — Online/Offline Divergence Audit

Question: is the offline evaluation metric misleading the team?

Compare the delta in offline AUC vs the delta in online booking rate across the last five solution versions. If offline keeps rising while online is flat, the team is overfitting the offline proxy — see `obs-watch-online-offline-divergence`.

If divergence is persistent: rebuild the offline evaluation. Use a held-out time window (not a random split) so the evaluation mirrors real use, and weight the evaluation by outcome not click.

Step 8 — Algorithm Iteration

Only run this step if Steps 1-7 pass. Algorithm changes are the most expensive and lowest-leverage intervention in most recommender systems, and the team arrives here rarely.

Options, in order of cost:

1. Tune event weights (cheap, no retraining) — see `loop-optimize-completed-outcome`. 2. Rich item metadata (moderate, requires schema discussion) — see `cold-use-v2-recipe-with-metadata`. 3. Context fields at inference (cheap if schema allows) — see `schema-include-context-everywhere`. 4. Candidate-generation → re-rank pipeline (expensive, 2-4 weeks) — see `recipe-build-candidate-rerank-pipeline`. 5. HPO (only with a ship/kill criterion set upfront) — see `recipe-defer-hpo-until-baseline-measured` and `simple-budget-complexity`.

Every option requires an A/B test and a written ship/kill criterion. No exceptions.

When the Playbook Says to Rollback

Some bottlenecks take days or weeks to fix. While the fix is in progress, revert to a known-good state rather than letting the broken system serve real users. Rollback options, in order of reach:

• Change traffic allocation to route the offending variant to 0% immediately
• Promote the previous production solution version
• Fall back to the popularity baseline (the permanent 3% bucket scales up)
• For a telemetry failure, stop training on the corrupted window until the pipeline is fixed

The goal of the rollback is to stop the bleeding — full diagnosis and fix come after, on a timeline that does not include the word "hotfix".

Using the Playbook in a Design Review

When the team debates "what should we work on next quarter", run through Steps 1-7 with real numbers on a spreadsheet before the meeting. The step that fails — or the step with the weakest margin over threshold — is the next quarter's work. Steps that pass comfortably can be ignored. This converts recsys planning from an opinion exercise into a deterministic one and usually collapses a two-week debate into a one-hour decision.

Planning Playbook: Building a Two-Sided Recommender from Scratch

This playbook walks through designing a new recommendation system for a two-sided marketplace end-to-end. It composes the rules from every category into a nine-step workflow that starts with instrumentation and ends with the first A/B-tested ML lift over a popularity baseline.

Use this playbook when:

• Launching a new surface (homefeed, search, category page, related-items shelf)
• Rebuilding a recommender that has accumulated too much technical debt to incrementally fix
• Planning the first personalisation work in a product that currently has none
• Preparing a design document or RFC for a new recsys initiative

Skip to the Improvement Playbook if the system already exists and the question is "how do we make it better" rather than "how do we design it".

Summary

Total: ~8-12 weeks from zero to a shipped A/B-tested ML model with online lift over a popularity baseline. Every step is a gate: do not proceed to the next step until the current step passes its exit criterion.

Step 1 — Define the Mutual-Fit Outcome

Goal: one sentence that states the primary metric the recommender will optimise.

A two-sided marketplace has more than one plausible objective: click rate, booking rate, booking rate weighted by provider acceptance, repeat booking rate, mutual-rating average, long-term marketplace liquidity. Pick one before writing any code — and write down why the other objectives were rejected. See `simple-audit-before-build` for the rationale.

Exit criterion: a written decision naming the primary metric, its formula, and the definition of a "successful match" (typically booking_completed AND mutual_rating >= 4).

Step 2 — Instrument the Lifecycle

Goal: every stage of the seeker-to-completion journey emits an event to Personalize.

Build the event schema and telemetry before the model. Every rule in the track-* category applies here. Minimum viable instrumentation:

Every event carries a requestId that joins it back to the ranker response, per `track-stamp-events-with-request-id`. Stream via PutEvents per `track-stream-events-via-putevents`; bulk import is only for historical backfill.

Exit criterion: instrumentation audit shows impression coverage ≥95%, booking_completed coverage ≥90%, and requestId join rate ≥98%. See `simple-audit-before-build` for the audit format.

Step 3 — Ship a Popularity Baseline

Goal: serve a non-ML top-N through the real inference path.

Before any Personalize work, ship a popularity baseline end-to-end: retrieve top-N completed-booking-count listings from the feasible set, render them through the real homefeed component, log impressions and outcomes. The baseline becomes the permanent control against which every future model is measured — see `simple-ship-popularity-baseline` and `simple-measure-gap-to-baseline`.

Exit criterion: popularity baseline serves production traffic behind a feature flag, with online metrics dashboarded per `obs-always-ab-test`.

Step 4 — Design the Dataset Schemas

Goal: three immutable schemas (Interactions, Items, Users) that will last years.

Schemas are effectively immutable in Personalize — adding a field means creating a new dataset group and re-importing all history. Follow every rule in the schema-* category:

• Keep Items and Users thin per `schema-keep-user-item-thin`
• Use categorical fields for bounded vocabularies per `schema-prefer-categorical-fields`
• Include context fields that will be populated at inference per `schema-include-context-everywhere`
• Weight event types with a plan for EVENT_VALUE per `schema-weight-event-value`

Exit criterion: schemas written as Avro JSON, reviewed by the team, and committed to the repository. Every field has a one-line rationale.

Step 5 — Import Historical Data

Goal: dataset group populated with historical events, items, and users.

Bulk import via S3 for the historical window (typically 90-365 days of interactions). Run PutItems / PutUsers in parallel for the current state of the catalog and user base. Verify dataset sizes meet AWS Personalize minimums: 50 users, 50 items, 1000 active interactions — below this, results degrade per the AWS Personalize cheat sheet.

Exit criterion: a dataset group with all three datasets imported, schemas matching Step 4, and row counts logged as a sanity check.

Step 6 — Build Candidate Generation and Re-rank

Goal: a two-layer pipeline where retrieval enforces hard rules and a re-ranker applies personalisation to the feasible set.

Follow `recipe-build-candidate-rerank-pipeline`. The candidate generator uses existing catalog search (region, date, species, legal compliance) and returns 100-500 items. The re-ranker is a PERSONALIZED_RANKING_v2 campaign that takes the candidate list and returns it sorted by relevance — see `recipe-personalized-ranking-as-reranker`.

Alternatively, for surfaces where the feasible set is small enough to pre-filter via Personalize filters, USER_PERSONALIZATION_v2 with a filter can serve as both generator and ranker in one call — see `recipe-default-to-user-personalization-v2`.

Exit criterion: the pipeline returns a non-empty, feasible, personalised list for a smoke-test set of test seekers. Cached responses expire within 60-120 seconds per `infer-cache-responses-short-ttl`.

Step 7 — Apply Two-Sided Matching

Goal: the ranker accounts for both sides' preferences and enforces fairness.

Every rule in the match-* category applies here. Wire the ranker to:

• Score by mutual fit per `match-rank-mutual-fit`
• Enforce feasibility in retrieval per `match-hard-filter-before-ranking`
• Cap provider exposure per `match-cap-provider-exposure`
• Discount by remaining capacity per `match-model-capacity-constraints`
• Route per segment-level liquidity per `match-balance-supply-demand`

Apply business rules after model scoring per `infer-rerank-rules-after-model`, deduplicate by provider per `infer-deduplicate-canonical-entity`, and enforce rolling exposure caps per `infer-enforce-exposure-caps`.

Exit criterion: top-24 responses on a diverse seeker sample show no provider appearing more than twice, feasibility 100%, and exposure distribution reasonably balanced across the feasible set.

Step 8 — Close the Feedback Loop

Goal: the system learns from every session without reinforcing bias.

Every rule in the loop-* category applies here. Wire:

• Slot logging per `loop-log-ranking-slot`
• Random exploration slice (3-5%) per `loop-reserve-random-exploration`
• Outcome-weighted training per `loop-optimize-completed-outcome`
• Event-weight decay per `loop-decay-event-weights`
• Death-spiral detection per `loop-detect-death-spirals`

Exit criterion: exploration slice is logged with propensity, slot is recorded on every impression, and the weekly exposure-Gini metric is dashboarded.

Step 9 — Launch A/B and Measure

Goal: an online A/B test shows statistically significant lift over the popularity baseline.

Run the full A/B test described in `obs-always-ab-test` with the popularity baseline as control and the Personalize pipeline as treatment. Slice metrics by segment per `obs-slice-metrics-by-segment`; watch the online-versus-offline divergence per `obs-watch-online-offline-divergence`.

Define the ship/kill criterion upfront per `simple-budget-complexity`. A typical criterion: ship if booking-completed-per-session lifts ≥2% with p<0.05, no segment regresses by >1%, and exposure-Gini does not rise; otherwise kill and diagnose via the improvement playbook.

Exit criterion: a shipped decision (ship or kill), documented in the experiments log, with the reason and the next step.

After Launch

The `improvement playbook` takes over. Every new experiment is framed as a delta against the current production model AND the retained popularity baseline, so drift against the baseline is detectable per `simple-measure-gap-to-baseline`.

Build a Candidate-Generation and Re-rank Pipeline

Marketplace ranking has two distinct concerns that fight each other inside a monolithic model: hard business rules (geography, availability, compliance) and soft preference learning. A candidate-generation → re-rank pipeline separates them — the candidate generator enforces hard rules and returns a feasible set of 100-500 items, then the re-ranker applies personalisation to that set. This is how Airbnb, DoorDash and Uber structure their marketplace ranking and the structure that lets you change one layer without touching the other.

Incorrect (monolithic call — model sees the full catalog, business rules tried post-hoc):

def homefeed(seeker: Seeker) -> list[Listing]:
    response = personalize_runtime.get_recommendations(
        campaignArn=CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=500,
    )
    all_ranked = hydrate_listings(response["itemList"])
    filtered = [
        listing for listing in all_ranked
        if listing.region == seeker.current_region
        and listing.is_available_today()
    ]
    return filtered[:24]

Correct (retrieval → re-rank — hard rules and personalisation are separate layers):

def homefeed(seeker: Seeker) -> list[Listing]:
    feasible = catalog.retrieve_feasible(
        region=seeker.current_region,
        available_on=seeker.current_date_range,
        accepts_species=seeker.pet_species,
        limit=300,
    )
    if not feasible:
        return []

    response = personalize_runtime.get_personalized_ranking(
        campaignArn=RERANK_CAMPAIGN_ARN,
        userId=seeker.id,
        inputList=[listing.id for listing in feasible],
    )
    ranked_ids = [item["itemId"] for item in response["personalizedRanking"][:24]]
    return [catalog.get(item_id) for item_id in ranked_ids]

Reference: Airbnb — Real-time Personalization using Embeddings for Search Ranking

Default to USER_PERSONALIZATION_v2 for Discovery

For discovery surfaces (homefeed, category landing, personalised shelves), USER_PERSONALIZATION_v2 is the default choice: it supports up to five million items, trains faster than v1, produces lower-latency recommendations, uses both item metadata and interactions for cold-start, and supports contextual features at inference. Pick a different recipe only when there is a specific reason — similar-item recommendations on a detail page (SIMS), re-ranking a user-provided list (PERSONALIZED_RANKING), or a baseline fallback.

Incorrect (using SIMS for homepage, ignores user history):

personalize.create_solution(
    name="homefeed-sims",
    datasetGroupArn=DATASET_GROUP_ARN,
    recipeArn="arn:aws:personalize:::recipe/aws-sims",
)

Correct (USER_PERSONALIZATION_v2 for discovery surfaces):

personalize.create_solution(
    name="homefeed-user-personalization-v2",
    datasetGroupArn=DATASET_GROUP_ARN,
    recipeArn="arn:aws:personalize:::recipe/aws-user-personalization-v2",
    eventsConfig={
        "eventParametersList": [
            {"eventType": "booking_completed", "weight": 10.0},
            {"eventType": "click", "weight": 1.0},
        ],
    },
)

Reference: AWS Personalize — User-Personalization-v2 Recipe

Defer HPO Until the Baseline Is Measured

Hyperparameter optimisation multiplies training cost and duration, so it is only justifiable after a default-hyperparameter baseline has demonstrated lift over the popularity baseline on online metrics. Running HPO before the baseline exists optimises a model you have not yet proven is worth deploying, and spends budget that would be better used on instrumentation or richer metadata. The correct order is: popularity baseline → default-hyperparameter ML → HPO only if the ML baseline is beaten and further tuning is warranted.

Incorrect (HPO enabled on the first solution version):

personalize.create_solution(
    name="homefeed-v1-hpo",
    datasetGroupArn=DATASET_GROUP_ARN,
    recipeArn="arn:aws:personalize:::recipe/aws-user-personalization-v2",
    performHPO=True,
    solutionConfig={
        "hpoConfig": {
            "algorithmHyperParameterRanges": {
                "integerHyperParameterRanges": [
                    {"name": "bptt", "minValue": 20, "maxValue": 40},
                ],
            },
        },
    },
)

Correct (default hyperparameters first, HPO deferred to a later iteration):

personalize.create_solution(
    name="homefeed-v1",
    datasetGroupArn=DATASET_GROUP_ARN,
    recipeArn="arn:aws:personalize:::recipe/aws-user-personalization-v2",
    performHPO=False,
)

Reference: Amazon Personalize Cheat Sheet — Premature HPO Pitfall

Use PERSONALIZED_RANKING_v2 as a Re-ranker, Not a Generator

PERSONALIZED_RANKING takes a caller-supplied list of items and returns it sorted by relevance to the user — it is a re-ranker, not a candidate generator. That matters because marketplace ranking must start from a set that already respects business rules (geography, availability, legal compliance, provider preferences), so the candidate-generation step belongs to the application and the re-ranking step belongs to Personalize. Trying to use it as a candidate generator produces empty responses — it has no way to retrieve items that were never supplied.

Incorrect (no input list — recipe treated as a candidate generator):

response = personalize_runtime.get_personalized_ranking(
    campaignArn=PERSONALIZED_RANKING_CAMPAIGN_ARN,
    userId=seeker.id,
    inputList=[],
)

Correct (application retrieves the feasible set, recipe re-ranks it):

def search(seeker: Seeker, query: SearchQuery) -> list[Listing]:
    feasible = catalog.search(
        region=query.region,
        date_range=query.date_range,
        accepts_species=seeker.pet_species,
    )
    if not feasible:
        return []

    response = personalize_runtime.get_personalized_ranking(
        campaignArn=PERSONALIZED_RANKING_CAMPAIGN_ARN,
        userId=seeker.id,
        inputList=[listing.id for listing in feasible],
    )
    ranked_ids = [item["itemId"] for item in response["personalizedRanking"]]
    return [catalog.get(item_id) for item_id in ranked_ids]

Reference: AWS Personalize — Choosing a Recipe

Use SIMS Only for Item-Page Similar Recommendations

SIMS (Similar-Items) is a collaborative-filtering recipe that finds items frequently co-interacted with a given seed item — it ignores user history entirely. That makes it exactly right for item-page surfaces ("other listings you might consider") where the seed item is the signal, and exactly wrong for homefeeds or personalised shelves where user history is the signal. Deploying SIMS on a homepage produces recommendations that are independent of who is logged in, which is the opposite of personalisation.

Incorrect (SIMS on a homefeed — user identity ignored):

response = personalize_runtime.get_recommendations(
    campaignArn=SIMS_CAMPAIGN_ARN,
    numResults=24,
)

Correct (SIMS on an item-page surface, user-personalization elsewhere):

def item_page_similar(current_listing: Listing) -> list[Listing]:
    response = personalize_runtime.get_recommendations(
        campaignArn=SIMS_CAMPAIGN_ARN,
        itemId=current_listing.id,
        numResults=12,
    )
    return hydrate_listings(response["itemList"])

def homefeed(seeker: Seeker) -> list[Listing]:
    response = personalize_runtime.get_recommendations(
        campaignArn=USER_PERSONALIZATION_CAMPAIGN_ARN,
        userId=seeker.id,
        numResults=24,
    )
    return hydrate_listings(response["itemList"])

Reference: AWS Personalize — Choosing a Recipe

Design Schemas Conservatively Because They Are Immutable

The Interactions dataset schema cannot be altered after creation — adding a field forces you to create a new dataset group and re-import every historical interaction. Users and Items datasets do support schema replacement to add nullable fields, but every added field still costs a full re-import of that dataset. This makes schema design a lifetime commitment for the Interactions table and a painful migration for Users/Items: add only fields that are stable, predictive and worth months of production history. Volatile or speculative fields belong in event properties, not the schema.

Incorrect (speculative fields that will be churned within a month):

{
  "type": "record",
  "name": "Interactions",
  "fields": [
    { "name": "USER_ID", "type": "string" },
    { "name": "ITEM_ID", "type": "string" },
    { "name": "TIMESTAMP", "type": "long" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "EXPERIMENT_BUCKET", "type": "string" },
    { "name": "PROMO_CAMPAIGN_ID", "type": ["null", "string"] },
    { "name": "UI_VARIANT", "type": "string" }
  ]
}

Correct (only stable, predictive fields in the schema):

{
  "type": "record",
  "name": "Interactions",
  "fields": [
    { "name": "USER_ID", "type": "string" },
    { "name": "ITEM_ID", "type": "string" },
    { "name": "TIMESTAMP", "type": "long" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "EVENT_VALUE", "type": ["null", "float"] },
    { "name": "SURFACE", "type": "string", "categorical": true },
    { "name": "DEVICE", "type": "string", "categorical": true }
  ]
}

Reference: AWS Personalize — Custom Datasets and Schemas · AWS Personalize — Replacing a Dataset's Schema to Add New Columns

Enforce Metadata Freshness as a First-Class Signal

A recommender that returns a listing marked "available" when it is actually booked out erodes trust faster than any ranking quality problem. Metadata freshness is a product-level contract: the Items dataset must be updated when availability windows change, when a provider deactivates, when a price tier shifts. Treating re-import latency as an operational SLO prevents the "why is the recommender showing me ghosts" failure mode.

Incorrect (weekly bulk re-import, days of stale metadata):

# Cron: every Sunday at 02:00 UTC
def weekly_items_refresh() -> None:
    export_items_to_s3(ITEMS_S3_URI)
    personalize.create_dataset_import_job(
        jobName=f"items-weekly-{date.today()}",
        datasetArn=ITEMS_DATASET_ARN,
        dataSource={"dataLocation": ITEMS_S3_URI},
        roleArn=PERSONALIZE_ROLE_ARN,
    )

Correct (incremental PutItems stream on every metadata change):

def on_listing_metadata_changed(listing: Listing) -> None:
    personalize_events.put_items(
        datasetArn=ITEMS_DATASET_ARN,
        items=[{
            "itemId": listing.id,
            "properties": json.dumps({
                "CATEGORY": listing.category,
                "REGION": listing.region,
                "PRICE_TIER": listing.price_tier,
                "ACTIVE": listing.is_active,
            }),
        }],
    )

Reference: AWS Personalize — PutItems API for Incremental Metadata Updates

Include Context Fields in Training and Inference

Context fields (surface, device, hour-of-day, weather) are only useful if they exist at both train time and serve time. A field that was present during training but omitted from the GetRecommendations call is silently defaulted by Personalize, so the model applies its learned weights to a missing feature and recommendations drift. The rule is brutal but simple: every context field in the schema must be populated in every inference call.

Incorrect (context fields in schema, missing from inference):

# Interactions schema declares SURFACE and DEVICE as categorical fields...
response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker_id,
    numResults=24,
)

Correct (every declared context field is passed at inference):

response = personalize_runtime.get_recommendations(
    campaignArn=CAMPAIGN_ARN,
    userId=seeker_id,
    numResults=24,
    context={
        "SURFACE": request.surface,
        "DEVICE": request.device,
        "HOUR_OF_DAY": str(datetime.utcnow().hour),
    },
)

Reference: AWS Personalize — Recommend and Dynamically Filter Based on User Context

Keep User and Item Metadata Thin and Stable

The Users and Items datasets describe attributes that "rarely or never change". If you put price, availability, last-login or session state there, the model trains on a stale snapshot — by the time it serves, the feature means something different, and you have training-serving skew that manifests as silent ranking quality drift. Volatile attributes belong in event context, not metadata.

Incorrect (volatile fields polluting the Items dataset):

{
  "type": "record",
  "name": "Items",
  "fields": [
    { "name": "ITEM_ID", "type": "string" },
    { "name": "CATEGORY", "type": "string", "categorical": true },
    { "name": "REGION", "type": "string", "categorical": true },
    { "name": "CURRENT_PRICE", "type": "float" },
    { "name": "AVAILABLE_THIS_WEEK", "type": "boolean" },
    { "name": "LAST_BOOKED_AT", "type": "long" }
  ]
}

Correct (stable attributes in metadata, volatile signals in events):

{
  "type": "record",
  "name": "Items",
  "fields": [
    { "name": "ITEM_ID", "type": "string" },
    { "name": "CATEGORY", "type": "string", "categorical": true },
    { "name": "REGION", "type": "string", "categorical": true },
    { "name": "PRICE_TIER", "type": "string", "categorical": true },
    { "name": "CREATION_TIMESTAMP", "type": "long" }
  ]
}

Reference: Google — Rules of Machine Learning (Rules 29, 32: training-serving skew)

Meet the AWS Personalize Minimum Dataset Sizes Before Training

AWS Personalize has hard minimums for solution training: 50 users, 50 items, and 1000 active interactions at the time the solution version is created. Below this, training may succeed but produces essentially random recommendations that obscure whether the architecture is correct. For a new surface, the first milestone is collecting enough data to clear the minimums — not tuning the first model. Gating training on a dataset-size check turns a silent "why is the model terrible" debugging session into an explicit "not enough data yet" message.

Incorrect (training kicked off regardless of dataset size):

def train_homefeed_solution() -> None:
    personalize.create_solution_version(
        solutionArn=SOLUTION_ARN,
        trainingMode="FULL",
    )

Correct (gate training on dataset minimums with explicit reporting):

def train_homefeed_solution() -> None:
    stats = dataset_stats.fetch(DATASET_GROUP_ARN)
    required = {"users": 50, "items": 50, "active_interactions": 1_000}
    missing = {
        key: required[key] - getattr(stats, key)
        for key in required
        if getattr(stats, key) < required[key]
    }
    if missing:
        logger.info(f"Dataset below training minimums; waiting on: {missing}")
        return
    personalize.create_solution_version(
        solutionArn=SOLUTION_ARN,
        trainingMode="FULL",
    )

Reference: Amazon Personalize Cheat Sheet — Insufficient Data Pitfall

Measure the Gap to Baseline on Every Change

When teams retire the baseline after the first ML win, they lose the reference point that would have caught a silent regression months later — a model that beat popularity at launch may drift below it after six recipe upgrades, training-data rewrites and schema changes. Keep the popularity baseline alive in a permanent minority bucket (1-5% of traffic) so every subsequent experiment can compare against it, not just against the current production model.

Incorrect (baseline turned off after first ML launch, no regression guard):

experiments.set_traffic_allocation({
    "popularity_baseline": 0,
    "user_personalization_v2": 80,
    "new_rerank_model": 20,
})

Correct (baseline retained as permanent minority bucket):

experiments.set_traffic_allocation({
    "popularity_baseline": 3,
    "user_personalization_v2": 77,
    "new_rerank_model": 20,
})

Reference: Google — Rules of Machine Learning, Rule 27: Try to Quantify Observed Undesirable Behaviour