Pinecone:Full Text Search

Pinecone Full-Text Search

Requires `pinecone` Python SDK ≥ 9.0 (pip install pinecone>=9.0). The FTS document-schema API lives under pinecone.preview and is incomplete or absent in earlier SDK builds. The packaged helper scripts pin pinecone==9.0.0 via PEP 723 inline metadata; if you're writing your own code against this skill, pin v9 explicitly. The wire API version is 2026-01.alpha.

Authoritative reference (last resort). If you hit a question this skill and its references/*.md files don't answer, the official Pinecone FTS docs are at <https://docs.pinecone.io/guides/search/full-text-search>. Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API) that don't apply to the document-schema FTS path. Consult the link only when you're genuinely stuck.

Tell the user up front: "This skill ships a helper at scripts/ingest.py that handles bulk ingestion safely (batched upsert, error inspection, readiness polling). When we get to the ingest step, I'll use it." Surface this at the start of the conversation so the user knows the helper exists. Query construction is hand-written documents.search(...) per the Querying section below — there is no query helper.

A workflow skill for building a Pinecone full-text-search index with the preview API (pinecone.preview, API version 2026-01.alpha, public preview as of April 2026). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (text / query_string / dense_vector / sparse_vector scoring; $match_phrase / $match_all / $match_any text-match filters; $eq / $in / $gte / $exists / $and / $or / $not metadata filters).

Scope — this skill is for the document-schema FTS API only

This skill covers pc.preview.indexes.create(..., schema=...), pc.preview.index(name), idx.documents.upsert(...) / idx.documents.batch_upsert(...) / idx.documents.search(...). If you find yourself reaching for any of the following, stop — those are different Pinecone APIs and this skill's guidance and helpers won't apply:

• Classic vector / records API: pc.Index(name), index.upsert(vectors=[...]) / index.upsert_records(...), index.query(vector=..., sparse_vector=...), index.search_records(...), pc.create_index(...) with ServerlessSpec, the legacy pinecone_text.sparse.BM25Encoder for sparse-dense hybrid. For indexes WITHOUT a schema (raw vectors).
• Integrated-embedding indexes: pc.create_index_for_model(...) with embed={...}. Pinecone vectorizes text server-side. Different upsert/search shapes. Cannot be combined with full_text_search fields in the same index.

If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic index after the fact.

Querying — construct documents.search(...) calls

For any task that asks you to query an FTS index, you write a documents.search(...) call directly. The schema is authoritative — describe the index live before constructing the call so you know which fields are FTS-enabled, which are filterable, and which are vectors.

Workflow:

1. Discover the schema. Call pc.preview.indexes.describe(<index>) and read the schema.fields dict. Each field's class indicates its type (PreviewStringField, PreviewIntegerField, PreviewDenseVectorField, etc.); attributes tell you whether it's FTS-enabled (full_text_search), filterable, or carries a dimension. Skip this step only if you've already seen the schema in this conversation. 2. Construct the call matching the rules below — one scoring type per request, hard requirements in filter, ranking signals in score_by, include_fields explicit on every call. 3. Execute with idx = pc.preview.index(name=<index>); resp = idx.documents.search(...) and read resp.matches.

Canonical shapes:

# Pure BM25 keyword search
resp = idx.documents.search(
    namespace="__default__",
    top_k=10,
    score_by=[{"type": "text", "field": "body", "query": "machine learning"}],
    filter={"year": {"$gt": 2024}, "category": {"$eq": "ai"}},  # optional
    include_fields=["*"],   # always pass explicitly
)

# Hybrid: dense ranking with a lexical filter (one type in score_by + filter narrows)
resp = idx.documents.search(
    namespace="__default__",
    top_k=10,
    score_by=[{"type": "dense_vector", "field": "embedding", "values": query_embedding}],
    filter={"body": {"$match_all": "TensorFlow"}, "year": {"$gt": 2024}},
    include_fields=["*"],
)

Key rules (the server enforces these; following them locally keeps the agent loop tight):

• score_by is a list of clauses, but exactly one scoring type per request (server rejects mixed types). Multi-field BM25 is the one exception: multiple text clauses, or one query_string with fields: [...]. To combine BM25 + dense signals, restrict the dense search with a text-match filter ($match_all / $match_phrase / $match_any); do NOT mix scoring types in score_by.
• filter keys are field names (must exist in schema and be filterable) OR logical operators ($and, $or, $not). Field values are operator dicts ({"$gt": 5}, NOT bare values).
• include_fields is required on every call. Pass ["*"] for all stored fields, [] for ids+score only, or a list of names. Some SDK builds 400/422 if it's omitted.

Clause shapes (for score_by):

text / dense_vector / sparse_vector use singular field. Only query_string accepts a fields array (and also accepts singular field as an alias). sparse_vector uses sparse_values (NOT values) — distinct from dense.

Filter operators by field type:

Match shape on response:

for m in resp.matches:
    m._id        # document id
    m._score     # match score (NOT `score`); some older SDK builds may also surface `score`
    m.to_dict()  # full doc payload (when include_fields includes the field)

For deeper coverage — multi-field BM25, Lucene patterns, hybrid composition, RRF merges, common error symptoms — see references/querying.md. For schema field types and what they enable on the query side, see references/schema-design.md.

Ingesting — use the packaged helper

For any task that asks you to bulk-ingest a JSONL file into an existing FTS index, the canonical path is to invoke the bundled helper, NOT to hand-write a Python script. Do not read the script's source — everything you need is in this section.

The script does three things bare-LLM ingest code reliably skips, each of which corresponds to a silent production failure:

1. Bulk-upserts in batches. No per-doc upsert loops. 2. Inspects every batch result. batch_upsert returns 202 even when individual documents fail; the failures live in result.errors / result.has_errors. Without inspection, "100 docs ingested" silently becomes "73 docs ingested + 27 lost." 3. Polls until searchable. After upsert, Pinecone is still building the inverted index. A documents.search call during that window returns empty. Without the poll, the user debugs their query code for an hour without finding the indexing race.

You provide a prepared, schema-conformant JSONL file and the index name; the script does the rest. Schema validation is upstream concerns (your prep pipeline, or prepare_documents.py when it lands) — ingest.py trusts what you hand it.

Invocation:

uv run --script scripts/ingest.py \
  --data processed.jsonl \
  --index <index_name> \
  --sentinel-field <fts_field>

Flags:

What the script prints:

Loading processed.jsonl ...
Loaded 5000 document(s).
Sentinel: body='The'

Upserting in batches of 100 ...
  batch @     0:  100 docs in  0.42s  (total: 100/5000)
  batch @   100:  100 docs in  0.39s  (total: 200/5000)
  ...

Upsert complete: 5000 doc(s) in 21.4s.

Polling for searchability (deadline 300s) ...
Searchable after 12.3s (3 probe(s)).

Done — total 33.7s.

If a batch fails, the script prints every error message and exits non-zero. If the poll deadline expires, the script prints a hint about why (sentinel field isn't FTS-enabled, deadline too tight, docs structurally upserted but rejected by the inverted-index builder) and exits non-zero. Don't suppress these errors — they're surfacing real problems with the data or the index.

When you should NOT use the script:

• The user is doing per-doc patch updates (single-doc documents.upsert calls with selective fields). The script is for bulk loads, not per-record operations.
• The user is ingesting from a non-JSONL source (CSV, Parquet, Postgres dump). Convert to JSONL first; the script doesn't parse other formats.
• The user explicitly asks you to write the ingestion code from scratch (teaching context). Honor the request and follow the canonical pattern: documents.batch_upsert + result.has_errors inspection + documents.search polling with sentinel and deadline.

The script lives at scripts/ingest.py relative to this skill directory. PEP 723 inline-metadata script — uv run --script installs typer and pinecone automatically on first invocation. No setup needed.

Use cases

Three concrete shapes to model your task on. Match the user's request to the closest one and follow its steps; improvise if the task is genuinely a hybrid.

UC-1: Index a new corpus end-to-end

Trigger. "Index this CSV / JSONL / folder for search," "build a search backend over [my articles / products / tickets / transcripts]," "make my [dataset] searchable."

For unprocessed / messy data, load the onboarding walkthrough first. If the user is showing up with raw data (unclear field types, possibly long text fields exceeding FTS limits, comma-separated tag strings, dates as strings, possibly duplicate IDs, etc.) and they haven't given you an explicit schema, read `references/onboarding-walkthrough.md` and follow it stage-by-stage. It's a conversational guide — meet the data, surface the processing decisions to the user, propose a schema, confirm before creating, then process+ingest+verify together. The walkthrough exists because schemas are immutable and "onboarding a new corpus" is a high-stakes flow that benefits from explicit user buy-in at each decision point.

If the user already gave you a clean JSONL + a schema spec, follow the abbreviated steps below.

Steps (when data is already prepared and the schema is decided): 1. Inspect the corpus shape — text fields, structured metadata, do you also need a vector? Match it to one of the canonical shapes in references/schema-design.md (articles, products, tickets, image library, code). 2. Pick analyzer settings on each text field — language, stemming, stop_words. Stemming on for long prose, off for proper nouns / identifiers. 3. Assemble the schema with SchemaBuilder and confirm it with the user before calling `indexes.create` — schemas are immutable in 2026-01.alpha, so a wrong call costs a re-ingest. 4. Create the index, poll describe() until status.ready: true. 5. Run `scripts/ingest.py --data <jsonl> --index <name> --sentinel-field <fts_field>` — see the Ingesting — use the packaged helper section above. The script handles batch_upsert + per-batch error inspection + post-upsert readiness polling in one invocation. Don't hand-write the loop unless the user explicitly asks you to. 6. (The script polls automatically — by the time it exits cleanly, the index is searchable. If you skip the script and roll your own, you must poll documents.search with a sentinel query and a deadline; batch_upsert returning ≠ searchable.) 7. Validate with one or two probe queries against fields you know contain the sentinel content.

Result. A working documents.search call against the user's data, returning ranked matches.

UC-2: Add a dense (or sparse) signal to a text-only corpus

Trigger. "Add semantic search," "add embeddings," "make this hybrid," or any prompt that describes a query pattern text alone can't serve (visual similarity, mood, cross-modal "looks like").

Steps. 1. Confirm the new signal represents a modality or signal text can't express — image / audio / external score, or a different corpus than the existing FTS field. Re-encoding the same text into a dense field is an anti-pattern (references/schema-design.md → "When to add a dense field at all"). 2. Because schemas are immutable, plan a new index, not a migration. Get user confirmation before recreating. 3. Pick an embedding provider and pin its output dimension at schema time. Beware payload-size pitfalls at native dimensions — Gemini-3072 etc. need truncation (references/ingestion.md → "Dense-vector payload size"). 4. Schema → create → wait Ready → ingest with embeddings inline or pre-cached. 5. Validate with a hybrid query: dense_vector score_by + text-match filter ($match_phrase / $match_all). That's the supported single-call cross-modal shape.

Result. One index, two retrieval shapes — pure text and dense+filter hybrid — both runnable without further setup.

UC-3: Build a documents.search call from a natural-language user prompt (agent mode)

Trigger. Agent receives a user prompt like "find articles about machine learning that mention TensorFlow and were published after 2024" or "documents about climate policy ranked by similarity to this paragraph." The index already exists.

Steps. 1. (Optional) Discover the schema by calling pc.preview.indexes.describe(<NAME>) and reading schema.fields. Skip if you already know the field types from earlier in the conversation. 2. Decompose the user's prompt into score_by / filter shapes using the agent-mode decomposition table below. (Hard requirements → filter. Ranking signals → score_by. Always include include_fields explicitly.) 3. Construct the `documents.search(...)` call following the rules in the Querying section above — one scoring type per request, operator/field-type matching, include_fields always set. 4. Execute the call. The response carries resp.matches; iterate to get m._id, m._score, and field values via m.to_dict(). Use the matches in whatever shape the user asked for. 5. If results come back empty or wrong, walk the failure tree in Common gotchas.

Result. Live search results matching the user's intent.

The four common UC-3 mistakes to actively avoid:

• Mixing scoring types in score_by (server rejects). Put hard requirements in filter; rank by one signal in score_by.
• Putting hard requirements in score_by as BM25 terms instead of in filter as $match_all / $match_phrase (returns ranked results that don't guarantee the term is present).
• Operator/field-type mismatches (e.g. $match_all on a float field, $gt on a string field). Consult the operator table in the Querying section.
• Omitting include_fields (some SDK builds 400/422). Always pass it explicitly.

Agent-mode query decomposition

Map user prompt cues to API shapes. Read top-down — identify the cue, copy the corresponding shape.

Two structural rules the agent must enforce, no exceptions:

• One scoring type per request. score_by accepts text / query_string / dense_vector / sparse_vector, but a request ranks by one. Don't mix dense + text in score_by — the server rejects it. Multi-field BM25 is the only "list" pattern that's allowed (multiple text clauses, or one cross-field query_string).
• Hybrid = filter + score_by, not two `score_by` clauses. When a prompt has both a lexical requirement and a semantic ranking signal, lexical goes in filter (via $match_* operators) and semantic goes in score_by. If both signals genuinely need to drive ranking, run two searches and merge IDs client-side.

Workflow at a glance

Three phases. Each has its own reference file — consult it before writing code for that phase.

1. Design the schema. Decide which string fields are full-text-searchable, which are filterable metadata, whether you need a dense_vector field (and whether it earns its place), whether you also need a sparse_vector field, and which numeric / boolean / array filters to declare. Schemas are fixed at index creation in 2026-01.alpha — plan carefully. → references/schema-design.md 2. Ingest documents. For bulk loads from a prepared JSONL, run the bundled scripts/ingest.py helper (it does batch_upsert + error inspection + readiness polling correctly by construction — see the Ingesting — use the packaged helper section above). For per-doc patch updates, hand-call documents.upsert. Either way, documents are indexed asynchronously after the HTTP call returns; batch_upsert returning 202 ≠ searchable. → references/ingestion.md for the canonical pattern in detail. 3. Query the index. A single search request ranks by one scoring type — pass exactly one of text, query_string, dense_vector, or sparse_vector in score_by (multi-field BM25 is supported via multiple text clauses or a cross-field query_string). Layer filter={...} for text-match ($match_phrase / $match_all / $match_any) and metadata filters ($eq / $in / $gte / $exists / $and / $or / $not). Control the response payload with include_fields. → references/querying.md

Quick template

End-to-end skeleton for a minimal text + filterable-metadata index. Copy it and edit every spot marked # TODO:. The template deliberately omits external embedding calls so it stays generic; see references/ingestion.md for dense / sparse field patterns and embedding-provider integration, and references/querying.md for the four scoring shapes plus text-match and metadata filters.

import time
from pinecone import Pinecone
from pinecone.preview import SchemaBuilder

INDEX_NAME = "my-fts-index"        # TODO: name your index (lowercase alphanumeric + hyphens, ≤45 chars)
NAMESPACE = "__default__"          # TODO: pick a namespace; auto-created on first upsert

pc = Pinecone()                    # reads PINECONE_API_KEY
# TODO: preprod backends require an x-environment header on the client:
#   pc = Pinecone(additional_headers={"x-environment": "preprod-aws-0"})

# 1. Schema — one FTS string field, one filterable string, one filterable float.
#    Field names must NOT start with `_` (reserved for `_id` / `_score`) or `$`
#    (reserved for filter operators), and are limited to 64 bytes.
schema = (
    SchemaBuilder()
    .add_string_field("body", full_text_search={"language": "en"})  # TODO: rename for your content
    .add_string_field("category", filterable=True)                   # TODO: any exact-match metadata
    .add_integer_field("year", filterable=True)                      # TODO: any numeric filter — emits `"type": "float"` on the wire
    .build()
)

# 2. Create the index. read_capacity defaults to {"mode": "OnDemand"}; pass
#    {"mode": "Dedicated", ...} only if you specifically want provisioned reads.
if not pc.preview.indexes.exists(INDEX_NAME):
    pc.preview.indexes.create(name=INDEX_NAME, schema=schema)

# 3. Wait for the index itself to become Ready.
while not pc.preview.indexes.describe(INDEX_NAME).status.ready:
    time.sleep(5)

idx = pc.preview.index(name=INDEX_NAME)

# 4. Upsert a single document. `_id` is required, every other field is optional.
#    upsert REPLACES the document on conflict — there is no per-field merge in 2026-01.alpha.
idx.documents.upsert(
    namespace=NAMESPACE,
    documents=[{
        "_id": "doc-1",
        "body": "Full-text search is great for keyword queries.",
        "category": "intro",
        "year": 2025.0,
    }],
)

# 5. Poll until the FTS side is searchable (upsert returns BEFORE docs are indexed).
deadline = time.time() + 300
while time.time() < deadline:
    resp = idx.documents.search(
        namespace=NAMESPACE, top_k=1,
        score_by=[{"type": "text", "field": "body", "query": "search"}],  # TODO: sentinel query likely to hit
        include_fields=[],          # required on every search; [] = lightest payload (ids + _score only)
    )
    if resp.matches:
        break
    time.sleep(5)

# 6. Search — text scoring composed with metadata filter.
resp = idx.documents.search(
    namespace=NAMESPACE,
    top_k=5,
    score_by=[{"type": "text", "field": "body", "query": "keyword queries"}],
    filter={"year": {"$gte": 2024}},        # TODO: adjust filter or drop it
    include_fields=["*"],                    # "*" = all stored fields; [] = `_id` + `_score` only
)
for m in resp.matches:
    print(m._id, getattr(m, "_score", getattr(m, "score", None)), m.to_dict())

Common gotchas

• One scoring type per search request. score_by accepts text, query_string, dense_vector, or sparse_vector — but a request ranks by one type. Multi-field BM25 is fine (pass several text clauses, or a single cross-field query_string). To combine BM25 ranking with a dense_vector (or sparse_vector) signal, restrict the dense search with a text-match filter operator ($match_phrase / $match_all / $match_any) on the lexical field, not by mixing types in score_by. The "blend a dense vector and a text clause in score_by" pattern is rejected by the server.
• Text-match filter operators are the cross-modal hinge. $match_phrase (exact phrase), $match_all (every token, any order), $match_any (at least one token) are filter-side operators on full_text_search fields. Each takes a single string (max 128 tokens). They reuse the field's tokenizer / stemmer, compose under $and / $or / $not, and are the supported way to compose lexical pre-filtering with dense or sparse ranking. *Phrase slop (`"…"~N`), term boost (`^N`), and phrase prefix (`"… word") are scoring-only — they live in query_string, not in filter`.**
• Preprod backends need `additional_headers={"x-environment": "..."}` on the `Pinecone()` client. Missing the header lands you on prod and you'll see "index not found" / empty-result symptoms that look like code bugs but aren't.
• `include_fields` is required on every `documents.search(...)` call. When omitted, defaults to [] (_id + _score only). Pass ["*"] for all stored fields or a list of names to project. Omitting it on some SDK builds yields 400 / 422 instead of the documented default; always pass it explicitly to avoid surprises.
• Match score is `_score`; doc id is `_id`. Public-preview docs return the system match score on the _score field so a user metadata field literally named score can coexist. Always prefer _score on read; some older SDK builds may still surface plain score, so for defensive code use getattr(m, "_score", getattr(m, "score", None)).
• Reserved field names: leading `_` and `$`, max 64 bytes. _ is for system fields (_id, _score); $ is for filter operators. Schema validation rejects names that violate either rule. Length cap is bytes, not characters — be careful with non-ASCII names.
• Vector-field cardinality: at most one `dense_vector` and at most one `sparse_vector` per index in 2026-01.alpha. Multiple text fields are fine.
• `batch_upsert` failures are silent by default. The return value carries has_errors, failed_batch_count, and a list of BatchError objects with error_message. If you don't inspect them, you'll see "Uploaded 0 / N" and an indefinite "not yet indexed" poll — with the real cause (payload-too-large, schema mismatch, reserved field name) hidden. Always print result.errors[*].error_message before downstream steps.
• Dense-vector payload size matters at batch time. A 50-doc batch with 3072-dim float vectors lands around 5–10 MB and can be rejected by the preview backend. If every batch fails, try reducing the embedding dimension via your provider's truncation knob (e.g. Gemini's output_dimensionality=768) before debugging schema.
• Async indexing: `batch_upsert` returning ≠ searchable. The server builds inverted indexes in the background after the HTTP call returns. If you query immediately you'll see empty result sets. Always poll documents.search with a sentinel query and a deadline (pattern in references/ingestion.md).
• String FTS field shape is `full_text_search={...}` (dict). Pass {} to enable with all server defaults. User-settable sub-fields: language, stemming, stop_words. Server-applied (visible in describe() responses but NOT settable at index creation): lowercase (default true) and max_token_length (default 40). Stemming is opt-in (default false); stop_words is opt-in (default false, opposite of pre-public-preview docs). The earlier SDK shape full_text_searchable=True, language="en" is legacy and should be avoided.
• Schemas are fixed at index creation in `2026-01.alpha`. Adding, removing, or retyping fields after creation is not supported. Changing dimension or metric on an existing vector field requires a new index. Plan the schema once.
• No partial / per-field updates. documents.upsert always replaces the entire document for a given _id. To update one field, fetch the doc, modify in client code, and upsert the full doc back under the same _id.
• Document operations: search supports `filter`, fetch and delete do not. Fetch is ID-only (POST /documents/fetch with ids: [...]); delete accepts only ids or delete_all: true. To act on a metadata expression, search first to collect IDs, then fetch or delete those IDs.
• Namespaces auto-create on first upsert. Pass any namespace string to documents.upsert / batch_upsert and the namespace is created on the fly; documents from different namespaces are fully isolated. Use "__default__" if you don't need partitioning. Caveat: the namespace management endpoints (POST /namespaces, GET /namespaces, DELETE /namespaces/{namespace}) and describe_index_stats are NOT yet supported on indexes with document schemas — you can write to a namespace, you just can't list / delete them via the API yet.
• Document and request size limits (preview): per-document max 2 MB; per-request max 2 MB and 1000 documents; per FTS-enabled string field max 100 KB and 10,000 tokens (tokens > 256 bytes are truncated by the analyzer); per-document filterable metadata (everything not in an FTS field) max 40 KB. A schema can declare up to 100 FTS string fields. For long-prose corpora, chunk before ingest — see references/ingestion.md.
• `score_by` clause shape — singular `field` is canonical for `text`/`dense_vector`/`sparse_vector`; only `query_string` takes a `fields` array.
• text: {"type":"text", "field":"<fts_field>", "query":"<terms>"}.
• query_string: {"type":"query_string", "query":"<lucene>", "fields":["<a>","<b>"]} (the optional fields array; query_string also accepts a bare "fields":"body" string and the legacy "field":"body" as an alias).
• dense_vector: {"type":"dense_vector", "field":"<dense_field>", "values":[/*floats*/]}.
• sparse_vector: {"type":"sparse_vector", "field":"<sparse_field>", "sparse_values":{"indices":[...],"values":[...]}} — note sparse_values (NOT values) for sparse clauses.
• Single-term prefix wildcards aren't supported. auto* doesn't work in query_string; use phrase prefix ("machine lea"* — phrase must contain at least two terms, last term is matched as prefix).
• Indexes can't be created in CMEK-enabled projects, no backup/restore, no fuzzy or regex search, no S3 bulk import for document-shaped indexes in 2026-01.alpha. If any of these are hard requirements, the public-preview FTS surface isn't yet ready.

Extension points

Currently shipped under scripts/:

• scripts/ingest.py — bulk-ingest a prepared JSONL into an existing FTS index. Handles batch_upsert in safe-sized chunks, inspects every batch's result.errors and aborts loudly on failure, then polls documents.search with a sentinel + deadline until docs are searchable. Schema-agnostic: takes only --data, --index, --sentinel-field. Usage in Ingesting — use the packaged helper section above.

Query construction does NOT have a packaged helper — write documents.search(...) calls directly per the Querying section above.

Ingestion

Writing documents into a Pinecone preview document index uses two methods. Pick based on volume, then handle the async indexing gotcha on the other side.

documents.upsert — small writes / patches

idx = pc.preview.index(name=INDEX_NAME)

upsert_resp = idx.documents.upsert(
    namespace=NAMESPACE,
    documents=[
        {
            "_id": "doc-1",
            "title": "A landmark work that every reader should experience.",
            "body": "Lorem ipsum...",
            "category": "fiction",
            "year": 2024.0,
        },
        # ... up to ~1000 documents per call (per public-preview docs)
    ],
)
print(upsert_resp.upserted_count)

Use upsert when:

• You're writing a single document (e.g. a sentinel doc to verify end-to-end before a bulk load).
• You're "patching" a doc after a correction. Note: 2026-01.alpha has no per-field merge — every upsert replaces the entire document on conflicting _id. To update a single field, fetch the doc, modify in client code, and upsert the full doc back under the same _id.
• You're streaming writes from user actions and each request fits in a single batch.

Each document is a dict keyed by field name. _id is required and must be a non-empty unique string within the namespace. Values must match the declared schema types (FTS strings → str, filterable float → int|float, dense vectors → list[float], sparse → {"indices": [...], "values": [...]}). Field names that start with _ or $ are rejected; field names are limited to 64 bytes.

The endpoint returns 202 Accepted (async) and the body's upserted_count is the number of items accepted, not the number that have finished indexing.

documents.batch_upsert — bulk loads

result = idx.documents.batch_upsert(
    namespace=NAMESPACE,
    documents=documents,        # list of dicts, any length
    batch_size=50,
    max_workers=2,
    show_progress=True,
)
print(f"{result.successful_item_count:,} / {result.total_item_count:,} succeeded")
if result.has_errors:
    print(f"Failed batches: {result.failed_batch_count}")
    # Always surface the actual reason — silent failures mask payload-size
    # caps, schema mismatches, and reserved-field-name violations.
    for err in result.errors[:3]:
        sample = err.items[0].get("_id") if err.items else "?"
        print(f"  batch #{err.batch_index} ({len(err.items)} items, "
              f"first _id={sample!r}): {err.error_message}")

The SDK splits documents into batch_size-sized chunks and uploads them over max_workers parallel HTTP connections. show_progress=True prints a tqdm-style bar.

Tuning batch_size and max_workers

• `batch_size=50` is the sweet spot — comfortably below the per-request cap and small enough that transient failures cost less to redo.
• `max_workers=2` is a safe default. Bump to 4 for large (thousands-of-docs) loads where you're not simultaneously embedding. Ramp cautiously above 4 — you'll hit Pinecone or upstream embedding-provider rate limits first.
• If you're embedding on the fly (computing vectors inside the upsert loop), keep max_workers low so embedding latency dominates rather than index write latency.

Document and request size caps

Hard limits in `2026-01.alpha`:

• Per document: max 2 MB (serialized JSON, all stored fields combined).
• Per `full_text_search` string field: max 100 KB AND max 10,000 tokens. Tokens longer than 256 bytes are silently truncated by the analyzer.
• Per upsert request: max 2 MB total AND max 1,000 documents.
• Per document filterable metadata (everything not in an FTS field): max 40 KB combined.
• Schema-level: up to 100 FTS string fields per index.

If any one of these is exceeded, the batch fails as a whole. The most common limit to hit on long-prose corpora is the per-FTS-field 100 KB / 10,000-token cap on a single body field — chunking is the standard fix (see below).

Dense-vector payload size

A high-dimensional dense field can silently turn a 50-doc batch into a 5–10 MB request, which the preview backend will reject wholesale. If every batch fails and the error message is opaque, the first thing to try is dropping the embedding dimension before debugging schema:

• Gemini: pass config=types.EmbedContentConfig(output_dimensionality=768). The model uses Matryoshka representations, so smaller dimensions are valid truncations of the native output. 768 is usually a 4× payload reduction vs. the native 3072 and costs very little quality.
• *OpenAI `text-embedding-3-**: pass dimensions=768 (or similar) to embeddings.create`.
• Pinecone hosted / fixed-dim models: dimension is fixed; the only levers are batch_size (halve it to 25) and per-document body size.

The async-indexing footgun

After batch_upsert returns, your documents are written but not yet searchable. The server builds inverted indexes for FTS fields and ANN graphs for vector fields in the background. A search query issued immediately will return empty matches. Schemas with multiple indexed fields (e.g. text + dense + sparse) may take slightly longer.

Always poll with a deadline before trusting the index:

import time

deadline = time.time() + 300  # up to 5 minutes
while time.time() < deadline:
    resp = idx.documents.search(
        namespace=NAMESPACE, top_k=1,
        score_by=[{"type": "text", "field": "<any_fts_field>", "query": "<sentinel>"}],
        include_fields=[],   # required on every search; [] = ids + _score only
    )
    if resp.matches:
        print("Data is searchable.")
        break
    time.sleep(5)
    print("Not yet indexed, retrying...")
else:
    print("WARNING: Documents may not be fully indexed after 5 minutes.")

Pick a sentinel query likely to hit at least one document. For a typical corpus, a single common token works (e.g. "book" for a book-reviews corpus). For a small corpus, use a term you know appears in at least one document.

Chunking oversized text

Per the public-preview docs (above), the per-FTS-field hard limits are 100 KB and 10,000 tokens. In practice, plan for the token limit kicking in first on natural prose (~5,000 English words at ~2 tokens each is the rough ceiling). Probe before ingesting at scale — chunk anything that approaches either bound, with safety margin.

Strategy: probe first, then chunk if needed.

1. Find the longest document in your corpus: max(len(doc["body"]) for doc in docs). 2. Try upserting it as-is. If the upsert errors, chunk.

Chunking pattern:

def chunk_text(text, max_chars=32_000):
    # Simple paragraph-aware chunking. Adjust the boundary for your corpus.
    paras = text.split("\n\n")
    chunks, cur = [], []
    cur_len = 0
    for p in paras:
        if cur_len + len(p) > max_chars and cur:
            chunks.append("\n\n".join(cur))
            cur, cur_len = [p], len(p)
        else:
            cur.append(p)
            cur_len += len(p)
    if cur:
        chunks.append("\n\n".join(cur))
    return chunks

docs = []
for doc_id, text, title in source:
    chunks = chunk_text(text)
    for i, chunk in enumerate(chunks):
        chunk_id = doc_id if i == 0 else f"{doc_id}#p{i + 1}"
        docs.append({
            "_id": chunk_id,
            "parent_doc_id": doc_id,    # duplicate identifying metadata across chunks
            "title": title,             # so title matches hit every chunk
            "body": chunk,
        })

Conventions:

• Shared key prefix. First chunk keeps the original _id; subsequent chunks append #p2, #p3. Easy to parse client-side.
• Duplicate identifying metadata. Fields like title, parent_doc_id, url, or whatever identifies the logical document should be present on every chunk so queries that filter or score against those fields work uniformly.
• Deduplicate at query time. After documents.search, group matches by parent_doc_id (or strip the #p* suffix from _id) and keep the highest-scoring chunk per parent. This preserves relevance ranking while collapsing duplicates in the UI.

Updating documents

There is no per-field update or merge in 2026-01.alpha. documents.upsert always replaces the entire document for a given _id. To update one field:

fetched = idx.documents.fetch(
    namespace=NAMESPACE,
    ids=["doc-42"],
    include_fields=["*"],          # need the full doc to round-trip it
)
doc = fetched.documents["doc-42"].to_dict()
doc["category"] = "biography"      # patch in client code

idx.documents.upsert(namespace=NAMESPACE, documents=[doc])

If the document includes a dense vector, you re-upsert that vector verbatim. If it changes, embed the new content first.

Deletes

documents.delete accepts either ids: [...] (1–1000 items) or delete_all: true. There is no delete-by-filter — to delete documents matching a metadata expression, search first to collect IDs, then pass them in:

ids_to_kill = [
    m._id for m in idx.documents.search(
        namespace=NAMESPACE, top_k=1000,
        score_by=[{"type": "text", "field": "body", "query": "deprecated"}],
        filter={"category": {"$eq": "archive"}},
        include_fields=[],
    ).matches
]
idx.documents.delete(namespace=NAMESPACE, ids=ids_to_kill)

delete_all=True wipes the entire namespace. Use carefully.

Integrating embedding providers

If your index has a dense or sparse vector field, you need embeddings. Three common paths:

Pinecone hosted inference

Cleanest integration — no extra API keys, same client as the index.

# Indexing side: use input_type="passage" for stored content
resp = pc.inference.embed(
    model="multilingual-e5-large",
    inputs=[doc["body"] for doc in batch],
    parameters={"input_type": "passage", "truncate": "END"},
)
embeddings = [e.values for e in resp.data]

# Query side: use input_type="query" for query strings
q_resp = pc.inference.embed(
    model="multilingual-e5-large",
    inputs=[user_query],
    parameters={"input_type": "query"},
)
q_emb = q_resp.data[0].values

The distinction between input_type="passage" (stored content) and input_type="query" (runtime queries) matters for models that encode them asymmetrically (multilingual-e5-large is one). For sparse learned embeddings like pinecone-sparse-english-v0, the same convention applies, and each embedding has .sparse_indices / .sparse_values rather than .values.

Batch size: ~96 inputs per embed call is the typical server limit. Loop in chunks:

EMBED_BATCH = 96
embeddings = []
for i in range(0, len(docs), EMBED_BATCH):
    chunk = docs[i : i + EMBED_BATCH]
    resp = pc.inference.embed(
        model="multilingual-e5-large",
        inputs=[d["body"] for d in chunk],
        parameters={"input_type": "passage", "truncate": "END"},
    )
    embeddings.extend(e.values for e in resp.data)

Generic pattern — any third-party provider

Wrap the provider-specific call in a thin adapter so ingestion logic doesn't know which provider is in use:

def embed(content) -> list[float]:
    """Return a single dense embedding for a piece of content.

    `content` may be a string or a PIL.Image, depending on the provider.
    Swap the implementation to change providers without touching callers.
    """
    resp = provider.embed(content)
    return resp.values  # or resp.data[0].embedding, etc.

docs = [{"_id": d["id"], "body": d["text"], "embedding": embed(d["text"])} for d in source]

This adapter also gives you a single chokepoint for retries, rate-limit backoff, and caching — add them once in embed() rather than at every call site.

Limits to be aware of

• No bulk import (S3 import job) for document-shaped indexes in 2026-01.alpha. Load through documents.upsert / documents.batch_upsert.
• No backup/restore. If you need recoverability, snapshot your source data, not the index.
• No CMEK projects — indexes can't be created in CMEK-enabled projects.
• Indexing latency: documents become searchable in ≲1 minute typically; multi-field schemas can take slightly longer.

Onboarding walkthrough — taking unprocessed data to a working FTS index

Load this file when the user shows up with unprocessed data and asks "make this searchable in Pinecone." The walkthrough is conversational on purpose — the goal is to help the user understand what decisions are being made, not to surprise them with a finished schema. At each `ASK` beat, stop and wait for the user's response before proceeding. Schemas are immutable; ingest is async; getting any of this wrong costs a re-ingest, so it's worth the chat round-trips.

When this walkthrough applies

• The user has data sitting in a file (CSV, JSONL, JSON, Parquet, Postgres dump, etc.)
• The data hasn't been pre-cleaned for Pinecone — types may be off, fields may be messy, long bodies may exceed FTS limits, duplicates may exist
• The user said something like "make this searchable" / "build me a search index" / "put this in Pinecone"
• The user did NOT provide an explicit schema or describe one in detail

If the user already gave you a schema spec or a JSONL of clean records, skip to UC-1's standard steps in SKILL.md.

If the user has an integrated-embedding records index already and wants to add FTS to it, see the Scope section in SKILL.md — that's a different surface and you can't add FTS fields after the fact.

Stage 1 — Meet the data

Goal: Both you and the user need to see the actual shape of the data before you can decide what to do with it.

Action: 1. Read the first 3-5 records of the file. Don't skim — look at field names, types, lengths, anything that looks off. 2. Summarize what you see in chat. Be concrete:

• "Your file has N records."
• "Each record has fields: A (string, ~X chars), B (string, up to Y chars), C (number), D (looks like a list/array)."
• "I noticed: <one or two specific observations — duplicates, unusual values, field with very wide variance, etc.>"

ASK the user (one chat turn, numbered): 1. "Is this what you expected, or are there fields I should ignore / rename?" 2. "Are there any records that aren't representative? (e.g. test rows, debug entries)" 3. "Any fields that should be unique IDs? (Pinecone needs a _id per document; if your data already has a unique key like slug or uuid, we'll map it.)"

Wait for the response. If the user adjusts your understanding, re-read what you need and re-summarize. Don't move on until they've confirmed you understand the data.

Stage 2 — Surface processing decisions

Goal: Most "unprocessed data" needs some transformation before it fits the FTS API. Surface the decisions so the user knows what you're going to do.

For each issue you saw in Stage 1, say what's happening and ask the user how to handle it. Don't decide silently. Common cases:

Long FTS text fields

If any text field exceeds 100 KB (or roughly 10,000 tokens ≈ ~5,000 English words):

• Tell the user: "Field body has records up to N KB. The Pinecone FTS limit is 100 KB / 10,000 tokens per field. We'll need to chunk longer ones."
• ASK: "Want to chunk by paragraph (the standard for prose), by fixed character count, or by semantic units I can detect? When I chunk, the first sub-document keeps the original _id and subsequent ones get a suffix like doc-42#p2, doc-42#p3 (the convention used by scripts/ingest.py). Sound OK?"

Type mismatches (CSV/Excel/Postgres common cases)

If types don't match what an FTS schema would want:

• Numbers stored as strings: "Your year field is a string like "2024". The schema needs a number. I'll coerce — but if any value can't be parsed, I'll abort and show you the offending row."
• Booleans as strings: same. "Convert "true" / "false" to booleans?"
• Comma-separated tags: "tags is a string 'classic,american'. The schema would index tags as a list. I'll split on , — speak up if your data uses a different separator."
• Dates / timestamps: "Pinecone has no date type. We'll either store as ISO-8601 strings (filterable for exact match), or convert to epoch milliseconds (filterable as numeric). Which do you want?"

Missing fields

If some records are missing fields others have:

• "About M of N records are missing intro. Do you want to: skip those records / store them with the field omitted / set a default?"

Duplicates

If _id (or your nominated unique key) repeats:

• "I found K duplicate IDs. Default behavior is last-wins (the later record overwrites). Or I can: abort if duplicates exist / merge fields across duplicates / first-wins. Which?"

Reserved field names

If any field starts with _ or $:

• "Your data has a field called _started_at — Pinecone reserves leading-underscore names for system fields. I'll rename to started_at unless you want a different name."

ASK (one chat turn): present each decision as a numbered question. Wait for answers. Don't proceed until every decision has a confirmed answer.

Stage 3 — Propose the schema

Goal: Translate the user's data + their answers from Stage 2 into a concrete Pinecone schema. Show it in chat as readable Python, with a one-line rationale per field.

Action: Write a chat message like:

Based on what we've discussed, here's the schema I'd create:

>

```python

SchemaBuilder()

# Searchable text — long prose, stemming on so "running" matches "ran"

.add_string_field("body", full_text_search={"language":"en", "stemming":True})

# Searchable text — short titles, stemming off (proper nouns shouldn't over-match)

.add_string_field("title", full_text_search={})

# Filter only — exact-match category like "fiction"

.add_string_field("category", filterable=True)

# Numeric range filter (e.g. year > 2024)

.add_integer_field("year", filterable=True)

# Tag filter — list membership ($in)

.add_string_list_field("tags", filterable=True)

.build()

```

>

A few notes:

- No dense_vector field — you said you don't have embeddings yet. We can add one later, but it requires creating a new index because schemas are immutable. Want to add a placeholder now and keep the door open?

- `year` uses `add_integer_field` but Pinecone stores it as float on the wire. The naming is confusing but normal — there's no separate integer type.

- `tags` will become ["a","b","c"] after the comma-split we discussed.

>

Schemas are immutable in 2026-01.alpha — once we create this, changing it means re-creating the index and re-ingesting all the data.

ASK (one question this time): "Look right? Want to adjust anything before I create the index?"

Wait. Do not proceed until the user explicitly approves (yes, looks good, go ahead, ship it, etc.). If they ask for changes, revise the schema in chat, re-show, ask again.

Stage 4 — Create the index

Once approved: 1. Write the Python (create.py or inline) using the approved schema. 2. Run it. Poll until pc.preview.indexes.describe(name).status.ready: True. 3. Tell the user when it's ready: "Index <name> created and ready. Now ingesting your data."

If creation fails for a reason you didn't anticipate (e.g. name conflict, region mismatch, CMEK restriction), tell the user the specific error and how to fix — don't auto-retry under a different name without asking.

Stage 5 — Process and ingest

Goal: Actually transform the data per Stage 2 decisions and bulk-load.

Action: 1. Write a small processing script that applies the agreed transformations: type coercion, chunking, dedup, list splitting, etc. Save the result to processed.jsonl. 2. Show the user a summary: "Wrote N processed records (was M raw; X chunked / Y deduped / Z dropped). Sample record: <first record>." 3. ASK (only if any record was dropped or substantially changed): "Look right?" If they confirm, proceed. 4. Invoke scripts/ingest.py --data processed.jsonl --index <name> --sentinel-field <your-longest-fts-field>. The script handles batch_upsert + error inspection + readiness polling. Don't reimplement that loop. 5. Watch its output. If it fails, tell the user what it complained about (field type mismatch, payload size, etc.) — don't just say "ingest failed."

Stage 6 — Verify together

Goal: Confirm the data is actually searchable. Don't trust polling alone — run a real query.

Action: 1. Pick a sentinel query you know should match a known record. E.g. for a corpus of book reviews, the first word of the first record's body; for a corpus with named entities, a known title or proper noun against the relevant FTS field. 2. Run a documents.search(...) call against the index — score_by=[{"type":"text", "field":"<fts_field>", "query":"<sentinel>"}], include_fields=["*"], top_k=3. See the Querying section in SKILL.md for the canonical shape. 3. Show the user the results: "Search returned K matches. Top match was <id> with score <x>." 4. If it didn't match what you expected, debug with the user — don't silently rerun. The async-indexing window may not have closed yet, OR your processing dropped a field, OR the user's expectation was off.

Stage 7 — Hand off

Goal: Make sure the user can use the index without you.

Tell them: 1. The index name and schema in one line. 2. How to query — give them a copy-pasteable idx.documents.search(...) snippet shaped to their schema (one score_by clause + the include_fields they care about). Refer them to the Querying section in SKILL.md or references/querying.md for variations. 3. How to ingest more — scripts/ingest.py with the same --sentinel-field they should use. 4. How to delete — pc.preview.indexes.delete("<name>") when done. 5. What's in the way of changing the schema — recreate + re-ingest, no schema migration.

Optionally save a small README.md in the working directory with the same info, so they have it when they come back.

Anti-patterns — don't do these

• Don't decide silently. Every decision in Stage 2 should be surfaced. If you assume a separator, a coercion, a dedup policy, you'll be wrong sometimes and the user won't know to push back.
• Don't call `indexes.create()` without explicit approval — schemas are immutable.
• Don't write a giant pre-flight script that does Stages 1-2 in code without ever showing the user. The point is the conversation, not the automation.
• Don't skip Stage 6. Polling says the index is "ready"; only a real query confirms the documents are there in the shape you expected.
• Don't add a dense_vector field "just in case." It commits the user to a specific embedding dimension forever — and if they don't have embeddings to ingest, the field is useless.
• Don't promise reversibility. Whenever you say "we can change this later," follow up with: "...by creating a new index and re-ingesting. There's no schema migration."

Non-interactive runtimes

If you're running in a non-interactive harness (CI bot, batch processor, eval sandbox where the user can't respond), make best-effort decisions, document each one in chat or comments, and proceed. Document the assumptions clearly so a human reviewing later can spot them: "I picked stemming=on for body because it's long prose. I assumed comma-separated tags. I dropped duplicate _ids with last-wins." When in doubt, prefer the conservative choice (don't chunk if you're unsure how, don't coerce if the values look ambiguous).

Querying

All reads on a Pinecone preview document index go through idx.documents.search(...) (ranked) or idx.documents.fetch(...) (direct, ID-only). The interesting shape is the single score_by clause and the filter={...} predicate — everything else is plumbing.

The one-scoring-type rule

A single documents.search request ranks by one scoring type. score_by accepts a list, but every entry must share a type:

• Multiple text clauses (one per field) — that's how multi-field BM25 works.
• A single query_string clause (which can target multiple fields via fields: [...] or inline field:term syntax inside the Lucene expression).
• dense_vector clauses must appear alone.
• sparse_vector clauses must appear alone.
• You cannot blend types — no text + query_string, no text + dense_vector, no cross-type mix. The server rejects it.

To compose lexical and dense / sparse signals, put the lexical signal in filter via the text-match operators ($match_phrase / $match_all / $match_any) and let the vector clause in score_by do the ranking. That's the supported hybrid pattern in 2026-01.alpha.

score_by signal types

1. text — BM25 token-OR on a single text field

resp = idx.documents.search(
    namespace=NAMESPACE,
    top_k=5,
    score_by=[{"type": "text", "field": "body", "query": "beautifully written"}],
    include_fields=["*"],
)

Tokenizes query with the field's analyzer, scores each matching document with a BM25 ranker over the inverted index, returns the top top_k. Multiple terms use OR semantics — documents matching any token participate, those matching more / rarer tokens score higher. Phrase constraints (adjacent words in order) are not supported here — use query_string with quotes, or a $match_phrase filter, for phrase semantics.

field is a single string (singular) naming an FTS-enabled string field — text clauses are scoped to one field at a time. For multi-field BM25, pass several text clauses (one per field) or use a query_string clause with a fields array (see Multi-field BM25 below).

2. query_string — Lucene syntax (boolean / phrase / boost / slop / prefix / cross-field)

resp = idx.documents.search(
    namespace=NAMESPACE,
    top_k=5,
    score_by=[{
        "type": "query_string",
        "query": 'body:(classic AND ("masterpiece" OR timeless)) NOT body:boring',
    }],
    include_fields=["*"],
)

Supported operators (full table in the public-preview docs, summarized here):

Cross-field clauses are unique to query_string — they let one expression target multiple text-searchable fields with their own sub-clauses. Optionally pass a top-level fields array on the clause to restrict scope; omitted, the query runs against every text-searchable field in the schema.

score_by=[{
    "type": "query_string",
    "fields": ["title", "body"],            # optional; restricts the query
    "query": 'title:(quantum)^2 OR body:("machine learning")',
}]

Single-term prefix wildcards (auto*) are not supported. Use phrase prefix instead: "machine lea"* (phrase must contain at least two terms; only the last is matched as prefix).

3. dense_vector — score against a stored dense vector

resp = idx.documents.search(
    namespace=NAMESPACE,
    top_k=5,
    score_by=[{"type": "dense_vector", "field": "embedding", "values": query_vector}],
    include_fields=["title", "body"],
)

field is a single string (singular) naming a dense_vector field. values is a list[float] matching the field's declared dimension. Typically produced by embedding the user's query through the same (or a compatible) model at runtime. For text embedders with a passage/query distinction (e.g. multilingual-e5-large), use input_type="query" on the query side. Must appear alone in score_by.

4. sparse_vector — score against a stored sparse vector

resp = idx.documents.search(
    namespace=NAMESPACE,
    top_k=5,
    score_by=[{
        "type": "sparse_vector",
        "field": "sparse_embedding",
        "sparse_values": {"indices": q.sparse_indices, "values": q.sparse_values},
    }],
    include_fields=["title", "body"],
)

Stored and queried as {"indices": [...], "values": [...]}. Hosted sparse models (e.g. pinecone-sparse-english-v0) return embeddings with .sparse_indices and .sparse_values ready to drop in. Must appear alone in score_by.

Multi-field BM25

Two equivalent ways to score across multiple text fields in one request:

Option A — multiple `text` clauses (one per field):

score_by=[
    {"type": "text", "field": "title", "query": q},
    {"type": "text", "field": "intro", "query": q},
    {"type": "text", "field": "body",  "query": q},
]

Option B — one `query_string` cross-field expression:

score_by=[{
    "type": "query_string",
    "query": f'title:({q}) OR intro:({q}) OR body:({q})',
}]

Both reward documents that match in multiple fields. `2026-01.alpha` weights every contributing field equally — there is no per-clause weight parameter. To approximate weighting, use Option B with ^N term boosts inside the query string (title:({q})^3 OR body:({q})).

Filtering

Filters run before scoring — they shrink the candidate set, then the chosen score_by ranks survivors. Two families of operators.

Text-match filters (on text-searchable fields)

These operate on full_text_search-enabled fields and reuse the field's tokenizer / stemmer. Each value is a single string (max 128 tokens). Not available inside query_string — they live in filter.

filter={"body": {"$match_phrase": "machine learning"}}      # exact phrase
filter={"body": {"$match_all":   "machine learning"}}      # both tokens, any order
filter={"body": {"$match_any":   "AI robotics"}}           # either token

These are the supported way to compose lexical pre-filtering with dense_vector (or sparse_vector) scoring — see "Cross-modal hybrid" below.

Scoring-only operators don't go in `filter`. Phrase slop ("…"~N), term boost (^N), and phrase prefix ("… word"*) influence ranking, so they're available in query_string score_by but not in filter.

Metadata filters (on filterable: true fields)

Standard comparison and membership operators — work on string, string_list, float, and boolean filterable fields.

Composing with $and / $or / $not

Multiple keys at the top level of a filter object are implicitly AND-ed. Use $and, $or, $not for explicit / nested composition. Text-match and metadata filters compose freely:

filter={
    "$and": [
        {"body": {"$match_all": "federal reserve"}},     # text-match operator
        {"category": {"$eq": "finance"}},                # metadata operator
        {"year": {"$gte": 2024}},
        {"$not": {"tags": {"$in": ["opinion"]}}},
    ],
}

Cross-modal hybrid: dense ranking + text-match filter

The supported way to compose lexical and dense signals in one request: dense (or sparse) score_by, plus a text-match filter that hard-restricts the candidate set to documents whose lexical field contains the right tokens / phrase.

results = idx.documents.search(
    namespace=NAMESPACE,
    top_k=10,
    filter={"body": {"$match_phrase": "beautifully written"}},
    score_by=[{
        "type": "dense_vector",
        "field": "review_embedding",
        "values": embed("a moving family epic"),
    }],
    include_fields=["*"],
)

Read it top-down: only docs whose body contains the exact phrase "beautifully written", ranked by dense-vector similarity to the embedding of "a moving family epic." One round trip, server-side hard filter, dense rerank.

When to use which text-match operator inside a hybrid query:

include_fields modes

include_fields controls what each match object carries back in the response.

Always pass `include_fields` explicitly on documents.search. Some SDK builds default to []; some return 400 / 422 if it's missing. Being explicit avoids surprises and makes the call's intent obvious.

User metadata fields literally named score are returned alongside the system-owned _score match score — the leading underscore prevents collisions.

Reading match objects

Match objects carry:

• _id (string) — document ID.
• _score (float) — system match score; higher is better.
• The fields requested via include_fields.

The score field name is reserved for user metadata; the system match score is always _score. Older SDK / backend builds may still emit unprefixed score; reading via getattr(match, "_score", getattr(match, "score", None)) covers both.

documents.fetch — direct retrieval, ID-only

Fetch is ID-only in 2026-01.alpha. It does not accept a filter. To retrieve documents matching a metadata expression, search first to get IDs, then fetch:

fetched = idx.documents.fetch(
    namespace=NAMESPACE,
    ids=["doc-1", "doc-2", "does-not-exist"],
    include_fields=["*"],
)
for doc_id, doc in fetched.documents.items():
    print(doc_id, doc.to_dict())

Missing IDs are silently omitted from the response (no error). ids accepts 1–1000 entries per call.

documents.delete — by ID or delete_all

# By IDs (1–1000 per call). Non-existent IDs are silently ignored.
idx.documents.delete(
    namespace=NAMESPACE,
    ids=["doc-1", "doc-2"],
)

# Wipe the entire namespace.
idx.documents.delete(
    namespace=NAMESPACE,
    delete_all=True,
)

Delete does not accept a filter. To delete documents matching a metadata expression, search first to collect IDs, then pass them to delete. Deletes are permanent within the namespace.

Worked cross-modal example — "pick your signal" pattern

One index with two FTS text fields and one multimodal dense vector field. The dense field holds an embedding that lives in a shared text/image space (e.g. a Gemini multimodal embedding of each document's representative image). Because text and image share the space, a typed description can be embedded as text and scored against the stored image vectors.

Schema

schema = (
    SchemaBuilder()
    .add_string_field("title", full_text_search={"language": "en"})
    .add_string_field("body",  full_text_search={"language": "en", "stemming": True})
    .add_dense_vector_field("image_embedding", dimension=DIM, metric="cosine")
    .build()
)

Three query modes against the same index

1. Pure text — multi-field BM25 token-OR.

resp = idx.documents.search(
    namespace=NS,
    top_k=10,
    score_by=[
        {"type": "text", "field": "title", "query": "transformer architecture"},
        {"type": "text", "field": "body",  "query": "transformer architecture"},
    ],
    include_fields=["title", "body"],
)

2. Exact phrase via `query_string`.

resp = idx.documents.search(
    namespace=NS,
    top_k=10,
    score_by=[{
        "type": "query_string",
        "query": 'body:("attention is all you need")',
    }],
    include_fields=["title", "body"],
)

3. Pure dense — semantic query against stored vectors.

q_emb = embed("a paper introducing self-attention for sequence modeling")
resp = idx.documents.search(
    namespace=NS,
    top_k=10,
    score_by=[{"type": "dense_vector", "field": "image_embedding", "values": q_emb}],
    include_fields=["title", "body"],
)

4. Hybrid — `$match_all` filter narrows; dense ranks.

q_emb = embed("self-attention for sequence modeling")
resp = idx.documents.search(
    namespace=NS,
    top_k=10,
    filter={"body": {"$match_all": "transformer"}},
    score_by=[{"type": "dense_vector", "field": "image_embedding", "values": q_emb}],
    include_fields=["title", "body"],
)

The same index supports all four modes; which one you want depends on whether the user's intent is keyword-driven (Mode 1), phrase-driven (Mode 2), appearance-driven (Mode 3), or "constrain by keyword, rank by appearance" (Mode 4). That's the pick-your-signal pattern — build the index once, vary the query shape per user intent.

Schema design

Everything a Pinecone preview document index needs is declared up-front via SchemaBuilder. The schema pins which fields are searchable, which are filterable metadata, which hold vectors, and what their dimensions / metrics are. Schemas are fixed at index creation in `2026-01.alpha` — adding, removing, or retyping fields afterwards is not supported. Plan carefully.

SchemaBuilder overview

from pinecone.preview import SchemaBuilder

schema = (
    SchemaBuilder()
    .add_string_field("title", full_text_search={"language": "en"})
    .add_string_field("body",  full_text_search={"language": "en", "stemming": True})
    .add_string_field("category", filterable=True)
    .add_integer_field("year", filterable=True)              # emits `"type": "float"` on the wire
    .add_dense_vector_field("embedding",       dimension=1024, metric="cosine")
    .add_sparse_vector_field("sparse_embedding", metric="dotproduct")
    .build()            # terminal: returns the schema object you pass to indexes.create
)

.build() is the terminal call — every chain ends with it. The resulting schema is passed to pc.preview.indexes.create(name=..., schema=schema, read_capacity=...). read_capacity defaults to {"mode": "OnDemand"} (auto-scaled shared reads); pass {"mode": "Dedicated", "dedicated": {...}} only if you specifically want provisioned read nodes.

Field types at a glance

Every field can also include an optional description string — surfaced by DescribeIndex and useful for agentic workflows where an LLM inspects the schema to decide how to query.

Reserved field names

Field names must be unique, non-empty strings. Two hard rules:

• Must not start with `_` — reserved for system-managed fields (_id, _score).
• Must not start with `$` — reserved for filter operators.
• Limited to 64 bytes (bytes, not characters — non-ASCII names take extra space).

_id is required on every document. _score is the system match-score field name returned by documents.search. A user metadata field literally named score is allowed and won't collide with _score.

String fields — text vs. metadata

A single string field is either full-text-search (BM25 / Lucene scoring + text-match filters) or filterable metadata (exact-match), never both. If you need both surfaces for the same logical content, duplicate it into two differently-configured fields.

Full-text-searchable string

.add_string_field("body", full_text_search={"language": "en", "stemming": True})

full_text_search takes a dict. Pass {} for all server defaults; populate it with any of:

• language (string, default "en") — selects the analyzer (tokenizer + stemmer + stopword set). Supported short codes: ar, da, de, el, en, es, fi, fr, hu, it, nl, no, pt, ro, ru, sv, ta, tr. Full names are also accepted (e.g. "english", "french", "arabic"). Stop-word lists are available for most languages but a few are tokenize/stem only (no stop_word filtering even when stop_words: true is set) — ar, da, de are notable cases; en, es, fr etc. have full stop-word support.
• stemming (boolean, default false) — if true, applies the language's stemmer so running matches runs.
• stop_words (boolean, default false) — if true, the analyzer's stopword set is filtered out at index and query time.
• lowercase (boolean, default true, server-applied) — case-insensitive matching.
• max_token_length (int, default 40, server-applied) — discards excessively long tokens.

Heuristic on stemming: turn it on for long prose fields where morphological variants of a root should match (running ~ runs ~ ran); leave off for short / identifier fields like titles, tags, or proper nouns where stemming would over-match (a book titled Running probably shouldn't also match the query ran). Typical pattern: stemming on for body, off for title / proper-noun fields.

Enables, on field_name:

• BM25 token scoring with score_by=[{"type": "text", "field": "field_name", "query": "..."}].
• Lucene scoring with score_by=[{"type": "query_string", "query": "field_name:(a AND (b OR c)) NOT field_name:d"}].
• Phrase / token filters: filter={"field_name": {"$match_phrase": "..."}}, {"$match_all": "..."}, {"$match_any": "..."}.

Filterable-only string

.add_string_field("category", filterable=True)

Stored verbatim, not tokenized, not text-scored. Enables exact-match filtering: {"category": {"$eq": "fiction"}}, {"category": {"$in": ["fiction", "biography"]}}, {"category": {"$exists": true}}.

Numeric, boolean, and array metadata

.add_integer_field("year", filterable=True)                              # wire type: "float"
.add_custom_field("featured", {"type": "boolean", "filterable": True})  # no add_boolean_field helper in v9
.add_string_list_field("tags", filterable=True)

• `float` is the only numeric wire type — there is no separate integer type. The SchemaBuilder helper is misleadingly named add_integer_field but emits {"type": "float", "filterable": ...}. Supports $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists.
• `boolean` has no dedicated builder helper in pinecone v9 — declare via add_custom_field("name", {"type": "boolean", "filterable": True}). Supports $eq and $exists.
• `string_list` supports $in / $nin membership semantics — handy for tag-style metadata.

All filter operators compose under $and, $or, $not. Multiple keys at the top level of filter are combined with implicit AND.

SchemaBuilder helper-name pitfall (pinecone v9): add_integer_field() produces {"type": "float"}, not a separate integer type. The class name in describe() responses is also PreviewIntegerField, but the wire/server type is "float". Use add_integer_field for any numeric metadata; use add_custom_field with an explicit {"type": "boolean", ...} dict for booleans.

Forward-looking note. In the public preview, metadata fields you send at upsert time are auto-indexed for filtering even if not declared in the schema. In a future release, only schema-declared fields with filterable: true will be indexed. Declare your metadata fields in the schema today to be future-proof.

Metadata size limit. Filterable metadata on a single document is capped at 40 KB combined (everything that's not in an FTS-enabled string field). FTS-enabled string fields don't count toward this — they have their own per-field limit (100 KB / 10,000 tokens, see references/ingestion.md).

Dense vector fields

.add_dense_vector_field("embedding", dimension=1024, metric="cosine")

• dimension must match whatever embedding model you'll store. If the model is chosen at runtime, query its default dimension first (e.g. pc.inference.get_model(model="multilingual-e5-large").default_dimension) and pass that in.
• metric is one of "cosine", "dotproduct", "euclidean". Pick the metric the embedding provider recommends — most text embedders use cosine.
• Scored at query time with score_by=[{"type": "dense_vector", "field": "embedding", "values": [...]}].

At most one `dense_vector` field per index in 2026-01.alpha. If you need two semantically distinct dense signals, you need two indexes.

Sparse vector fields

.add_sparse_vector_field("sparse_embedding", metric="dotproduct")

• No dimension — sparse vectors are variable-length.
• metric="dotproduct" is the standard choice for learned sparse embeddings (e.g. pinecone-sparse-english-v0).
• Stored and queried as {"indices": [...], "values": [...]}; query side: score_by=[{"type": "sparse_vector", "field": "sparse_embedding", "sparse_values": {"indices": [...], "values": [...]}}].

At most one `sparse_vector` field per index in 2026-01.alpha.

When to add a dense field at all

This is the key design question when the index already has FTS fields. Only add a dense vector field when it represents a modality or signal that FTS cannot express. Examples of justified dense fields:

• An image embedding over pictures associated with each document — visual appearance is not text.
• An audio embedding over voice clips or music — timbre and melody are not text.
• An external ranking-model score pre-computed and stored as a 1-D "vector" for sort purposes.
• A semantic text embedding over a different corpus than the one in the FTS field — e.g. the FTS field holds the product description, the dense field holds an embedding of the seller's support-ticket history for that product. Different data, different signal.

Anti-pattern: re-encoding text that already lives in an FTS field on the same index. Indexing the body string as FTS and embedding that same body into a dense text vector on the same index is redundant modeling, not an additive signal. FTS already gives you lexical retrieval; adding a dense re-encoding only pays off when the lexical signal is demonstrably insufficient (typically: very large corpus, very semantic queries, and you've measured the gap).

Multi-field text design heuristics

When a document has a natural hierarchy (title → intro → body, or summary → transcript, or headline → lede → article), splitting across FTS fields enables two things you can't get from one blob:

1. Per-field scoring. A match on title is almost always a stronger signal than a match on body. With separate fields you can search just the title, just the body, or blend them at query time by listing each as its own score_by entry (see references/querying.md — multi-field BM25). 2. Multi-field blended relevance. Passing score_by=[{text, title, q}, {text, intro, q}, {text, body, q}] rewards documents that match in multiple fields. (2026-01.alpha weights every contributing field equally — no per-clause weight parameter.)

Keep it a single field when:

• The content has no natural subdivision (a tweet, a log line, a chat message).
• You will never want per-field weighting at query time.
• Your documents are short enough that inter-field distinctions are noise.

Schemas are fixed at creation

2026-01.alpha does not support schema migration. You cannot:

• Add a new field after creation.
• Remove an existing field.
• Change a field's type or sub-config (e.g. flip a filterable string to FTS, toggle stemming, change dense vector dimension).

The supported workaround is to create a new index with the desired schema and reindex documents (the document set is small enough at preview-launch scale that this is usually painless). Existing pre-public-preview indexes from earlier API versions cannot be backfilled with a 2026-01.alpha schema.

description for agentic / LLM-driven workflows

Each field accepts an optional description string:

.add_string_field(
    "body",
    full_text_search={"language": "en", "stemming": True},
    description="Full article text. Use for keyword searches, narrative phrases, and topical queries.",
)

Returned by DescribeIndex. Useful when an LLM is choosing how to query: it can read the descriptions and pick the right field + operator without hard-coded prompt engineering.