Recsys Pipeline Architect

# Recsys Pipeline Architect

A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. Encodes the six-stage pattern popularized by xAI's open-sourced [For You algorithm](https://github.com/xai-org/x-algorithm) (Apache 2.0) and applies it to any "top K for (user, context)" problem.

## Overview

Most "recommendation systems" in production aren't exotic ML — they're *pipelines*: fetch candidates from one or more sources, enrich them with metadata, drop the ineligible, score the rest, sort and pick the top K, then fire async side effects. The pattern is universal. The scoring function and the items change; the pipeline shape doesn't.

This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.

## When to Use This Skill

- Building any system that returns "the top K items for a user/context"
- Designing or refactoring a personalized feed (content, search results, notifications)
- Wrapping an LLM/ML scorer in proper pipeline plumbing (sources, hydration, filters, side effects)
- Adding multi-action prediction with tunable weights (instead of a single relevance score)
- Building a RAG retrieval reranker (cheap retrieval → expensive rerank)
- Designing a task prioritizer or alert triage system

## The Six-Stage Framework

| # | Stage | Job | Parallel? |
|---|---|---|---|
| 1 | **Source** | Fetch candidates from one or more origins | Yes — multiple sources run in parallel |
| 2 | **Hydrator** | Enrich candidates with metadata needed for filtering and scoring | Yes — independent hydrators run in parallel |
| 3 | **Filter** | Drop ineligible candidates (blocked, expired, duplicate, ineligible) | Sequential — each filter sees fewer items |
| 4 | **Scorer** | Assign each surviving candidate one or more scores | Sequential — later scorers see earlier scores |
| 5 | **Selector** | Sort by final score, return top K | Single op |
| 6 | **SideEffect** | Cache, log, emit events, update served-history | Async — must never block the response |

### Why this exact order

- Sources before hydration: know what candidates exist before paying to enrich
- Hydration before filtering: many filters need metadata the source didn't provide
- Filtering before scoring: scoring is the expensive stage — drop the ineligible first
- Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
- Selector after scoring: keeps scoring deterministic and cacheable
- SideEffects last and async: side effects must never block the user response

## Workflow When Invoked

Walk the user through eight steps:

1. **Clarify the use case** (one round, three questions only if missing): items being ranked, input context, language/runtime
2. **Identify the candidate sources** (usually in-network + out-of-network, but single-source also valid)
3. **List required hydrations** — for each filter and scorer, what data does it need that the source didn't provide?
4. **List the filters** — cheap before expensive, universal before user-specific (duplicate, self, age, block/mute, previously-served, eligibility)
5. **Design the scorer chain** — primary ML/heuristic → combiner (multi-action with weights) → diversity → business rules
6. **Selector** — sort descending by final score, take top K (or stratified mix)
7. **SideEffects** — cache served IDs, emit impression events, update counters, log analytics; all fire-and-forget
8. **Generate the scaffold** in the user's stack

## Key Trade-offs to Surface

Never default silently on these — they a