Tanstack Ai

TanStack AI (Provider-Agnostic LLM SDK)

Status: Production Ready ✅ Last Updated: 2025-12-09 Dependencies: Node.js 18+, TypeScript 5+; React 18+ for @tanstack/ai-react; Solid 1.8+ for @tanstack/ai-solid Latest Versions: @tanstack/ai@latest (alpha), @tanstack/ai-react@latest, @tanstack/ai-client@latest, adapters: @tanstack/ai-openai@latest @tanstack/ai-anthropic@latest @tanstack/ai-gemini@latest @tanstack/ai-ollama@latest

---

Quick Start (7 Minutes)

1) Install core + adapter

pnpm add @tanstack/ai @tanstack/ai-react @tanstack/ai-openai
# swap adapters as needed: @tanstack/ai-anthropic @tanstack/ai-gemini @tanstack/ai-ollama
pnpm add zod              # recommended for tool schemas

Why this matters:

• Core is framework-agnostic; React binding just wraps the headless client. citeturn1search3
• Adapters abstract provider quirks so you can change models without rewriting code. citeturn1search3

2) Ship a streaming chat endpoint (Next.js or TanStack Start)

// app/api/chat/route.ts (Next.js) or src/routes/api/chat.ts (TanStack Start)
import { chat, toStreamResponse } from '@tanstack/ai'
import { openai } from '@tanstack/ai-openai'
import { tools } from '@/tools/definitions' // definitions only

export async function POST(request: Request) {
  const { messages, conversationId } = await request.json()
  const stream = chat({
    adapter: openai(),
    messages,
    model: 'gpt-4o',
    tools,
  })
  return toStreamResponse(stream)
}

CRITICAL:

• Pass tool definitions to the server so the LLM can request them; implementations live in their runtimes. citeturn0search7
• Always stream; chunked responses keep UIs responsive and reduce token waste. citeturn0search1

3) Wire the client with useChat + SSE

// components/Chat.tsx
import { useChat, fetchServerSentEvents } from '@tanstack/ai-react'
import { clientTools } from '@tanstack/ai-client'
import { updateUIDef } from '@/tools/definitions'

const updateUI = updateUIDef.client(({ message }) => {
  alert(message)
  return { success: true }
})

export function Chat() {
  const tools = clientTools(updateUI)
  const { messages, sendMessage, isLoading, approval } = useChat({
    connection: fetchServerSentEvents('/api/chat'),
    tools,
  })

  return (
    <form onSubmit={e => { e.preventDefault(); sendMessage(e.currentTarget.prompt.value) }}>
      <textarea name="prompt" disabled={isLoading} />
      {approval?.pending && (
        <button type="button" onClick={() => approval.approve()}>
          Approve tool
        </button>
      )}
    </form>
  )
}

CRITICAL:

• Use fetchServerSentEvents (or matching adapter) to mirror the streaming response. citeturn0search0
• Keep client tool names identical to definitions to avoid “tool not found” errors. citeturn0search7

---

The 4-Step Setup Process

Step 1: Choose provider + model safely

• Add the correct adapter and set the matching API key (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, or Ollama host).
• Prefer per-model option typing from adapters to avoid invalid options (e.g., vision-only fields). citeturn1search3

Step 2: Define tools once, implement per runtime

// tools/definitions.ts
import { z, toolDefinition } from '@tanstack/ai'

export const getWeatherDef = toolDefinition({
  name: 'getWeather',
  description: 'Get current weather for a city',
  inputSchema: z.object({ city: z.string() }),
  needsApproval: true,
})

export const getWeather = getWeatherDef.server(async ({ city }) => {
  const data = await fetch(`https://api.weather.gov/points?q=${city}`).then(r => r.json())
  return { summary: data.properties?.relativeLocation?.properties?.city ?? city }
})

export const showToast = getWeatherDef.client(({ city }) => {
  console.log(`Showing toast for ${city}`)
  return { acknowledged: true }
})

Key Points:

• needsApproval: true forces explicit user approval for sensitive actions. citeturn0search1
• Keep tools single-purpose and idempotent; return structured objects instead of throwing errors. citeturn0search1

Step 3: Create connection adapter + chat options

• Server: toStreamResponse(stream) for HTTP streaming; toServerSentEventsStream helper for Server-Sent Events. citeturn0search3turn0search4
• Client: fetchServerSentEvents('/api/chat') or a custom adapter for websockets if needed. citeturn0search0
• Configure agentLoopStrategy (e.g., maxIterations(8)) to cap tool recursion. citeturn1search4

Step 4: Add observability + guardrails

• Log tool executions and stream chunks for debugging; alpha exposes hooks while devtools are in progress. citeturn0search1
• Validate inputs with Zod; fail fast and return typed error objects.
• Enforce timeouts on external API calls inside tools to prevent stuck agent loops.

---

Critical Rules

Always Do

✅ Stream responses; avoid waiting for full completions. citeturn0search1 ✅ Pass definitions to the server and implementations to the correct runtime. citeturn0search7 ✅ Use Zod schemas for tool inputs/outputs to keep type safety across providers. citeturn0search1 ✅ Cap agent loops with maxIterations to prevent runaway tool calls. citeturn1search4 ✅ Require needsApproval for destructive or billing-sensitive tools. citeturn0search1

Never Do

❌ Mix provider adapters in a single request—instantiate one adapter per call. ❌ Throw raw errors from tools; return structured error payloads. ❌ Send client tool implementations to the server (definitions only). ❌ Hardcode model capabilities; rely on adapter typings for per-model options. citeturn0search1 ❌ Skip API key checks; fail fast with helpful messages on the server. citeturn0search1

---

Known Issues Prevention

This skill prevents 3 documented issues:

Issue #1: “tool not found” / silent tool failures

Why it happens: Definitions aren’t passed to chat(); only implementations exist locally. Prevention: Export definitions separately and include them in the server tools array; keep names stable. citeturn0search7

Issue #2: Streaming stalls in the UI

Why it happens: Mismatch between server response type and client adapter (HTTP chunked vs SSE). Prevention: Use toStreamResponse on the server + fetchServerSentEvents (or matching adapter) on the client. citeturn0search1turn0search0

Issue #3: Model option validation errors

Why it happens: Provider-specific options (e.g., vision params) sent to unsupported models. Prevention: Use adapter-provided types; rely on per-model option typing to surface invalid fields at compile time. citeturn1search3

---

Configuration Files Reference

.env.local (Full Example)

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=
GEMINI_API_KEY=
OLLAMA_HOST=http://localhost:11434
AI_STREAM_STRATEGY=immediate

Why these settings:

• Keep non-active providers empty to avoid accidental multi-provider calls.
• AI_STREAM_STRATEGY is read by the sample client to pick chunk strategies (immediate vs buffered).

---

Common Patterns

Pattern 1: Agentic cycle with bounded tools

import { chat, maxIterations } from '@tanstack/ai'
import { openai } from '@tanstack/ai-openai'

const stream = chat({
  adapter: openai(),
  messages,
  tools,
  agentLoopStrategy: maxIterations(8), // hard cap
})

When to use: Any flow where the LLM could recurse across tools (search → summarize → fetch detail). citeturn1search4

Pattern 2: Hybrid server + client tools

// server: data fetch
const fetchUser = fetchUserDef.server(async ({ id }) => db.user.find(id))

// client: UI update
const highlightUser = highlightUserDef.client(({ id }) => {
  document.querySelector(`#user-${id}`)?.classList.add('ring')
  return { highlighted: true }
})

chat({ tools: [fetchUser, highlightUser] })

When to use: When the model must both fetch data and mutate UI state in one loop. citeturn0search1

---

Using Bundled Resources

Scripts (scripts/)

• scripts/check-ai-env.sh — verifies required provider keys are present before running dev servers.

Example Usage:

./scripts/check-ai-env.sh

References (references/)

• references/tanstack-ai-cheatsheet.md — condensed server/client/tool patterns plus troubleshooting cues.

When Claude should load these: When debugging tool routing, streaming issues, or recalling exact API calls.

Assets (assets/)

• assets/api-chat-route.ts — copy/paste API route template with streaming + tools.
• assets/tool-definitions.ts — ready-to-use toolDefinition examples with approval + zod schemas.

---

When to Load References

Load reference files for specific implementation scenarios:

• Adapter Comparison: Load references/adapter-matrix.md when choosing between OpenAI, Anthropic, Gemini, or Ollama adapters, or when debugging provider-specific quirks.

• React Integration Details: Load references/react-integration.md when implementing useChat hooks, handling SSE streams in React components, or managing client-side tool state.

• Routing Setup: Load references/start-vs-next-routing.md when setting up API routes in Next.js vs TanStack Start, or troubleshooting streaming response setup.

• Streaming Issues: Load references/streaming-troubleshooting.md when debugging SSE connection problems, chunk delivery issues, or HTTP streaming configuration.

• Quick Reference: Load references/tanstack-ai-cheatsheet.md for condensed API patterns, tool definition syntax, or rapid troubleshooting cues.

• Tool Architecture: Load references/tool-patterns.md when implementing complex client/server tool workflows, approval flows, or hybrid tool patterns.

• Type Safety Details: Load references/type-safety.md when working with per-model option typing, multimodal inputs, or debugging type errors across adapters.

---

Advanced Topics

Per-model type safety

• Use adapter typings to pick valid options per model; avoid generic any options on chat(). citeturn1search3
• For multimodal models, send parts with correct MIME types; unsupported modalities are caught at compile time. citeturn1search3

Tool approval UX

• Surfaced via approval object in useChat; render approve/reject UI and persist decision per tool call. citeturn0search1
• For auditable actions, log approval decisions alongside tool inputs.

Connection adapters

• Default to fetchServerSentEvents (SSE) for minimal setup; switch to custom adapters for websockets or HTTP chunking. citeturn0search0
• Use ImmediateStrategy in the client to emit every chunk for typing indicator UIs. citeturn0search0

---

Dependencies

Required:

• @tanstack/ai@latest — core chat + tool engine
• @tanstack/ai-react@latest — React bindings (skip for headless usage)
• @tanstack/ai-client@latest — headless chat client + adapters
• Adapter: one of @tanstack/ai-openai@latest | @tanstack/ai-anthropic@latest | @tanstack/ai-gemini@latest | @tanstack/ai-ollama@latest
• zod@latest — schema validation for tools

Optional:

• @tanstack/ai-solid@latest — Solid bindings
• @tanstack/react-query@latest — cache data fetched inside tools
• @tanstack/start@latest — co-locate AI tools with server functions

---

Official Documentation

• TanStack AI Overview: https://tanstack.com/ai/latest/docs/getting-started/overview
• Quick Start: https://tanstack.com/ai/latest/docs/getting-started/quick-start
• Tool Architecture & Approval: https://tanstack.com/ai/latest/docs/guides/tool-architecture
• Client Tools: https://tanstack.com/ai/latest/docs/guides/client-tools
• API Reference: https://tanstack.com/ai/latest/docs/api/ai

---

Package Versions (Verified 2025-12-09)

{
  "dependencies": {
    "@tanstack/ai": "latest",
    "@tanstack/ai-react": "latest",
    "@tanstack/ai-client": "latest",
    "@tanstack/ai-openai": "latest"
  },
  "devDependencies": {
    "zod": "latest"
  }
}

---

Troubleshooting

Problem: UI never receives tool output

Solution: Ensure tool implementations return serializable objects; avoid returning undefined. Register client implementations via clientTools(...).

Problem: “Missing API key” responses

Solution: Run ./scripts/check-ai-env.sh and set the relevant provider key in .env.local. Fail fast in the route before invoking chat(). citeturn0search1

Problem: Streaming stops after first chunk

Solution: Confirm the server returns toStreamResponse(stream) (or SSE helper) and that any reverse proxy allows chunked transfer.

---

Complete Setup Checklist

Use this checklist to verify your setup:

• [ ] Installed core + one adapter and zod
• [ ] API route returns toStreamResponse(stream) with tool definitions included
• [ ] Client uses fetchServerSentEvents (or matching adapter) and registers client tool implementations
• [ ] needsApproval paths render approve/reject UI
• [ ] Agent loop capped (e.g., maxIterations)
• [ ] Environment keys validated with check-ai-env.sh
• [ ] Multimodal inputs tested if targeting vision/audio models

---

Questions? Issues?

1. Load references/tanstack-ai-cheatsheet.md for deeper examples 2. Re-run quick start steps with a single provider adapter 3. Review official docs above for API surface updates

---

TanStack AI Adapter Matrix (OpenAI | Anthropic | Gemini | Ollama)

Env Keys

• OpenAI: OPENAI_API_KEY
• Anthropic: ANTHROPIC_API_KEY
• Gemini: GEMINI_API_KEY
• Ollama: OLLAMA_HOST (defaults to http://localhost:11434)

Supported Models & Notes (alpha SDK)

Per-Model Option Gotchas

• OpenAI: response_format only on some models; vision requires image parts.
• Anthropic: max_output_tokens required; temperature + top_p interplay—set one.
• Gemini: Enable generationConfig for safety; set tools under systemInstruction semantics when needed.
• Ollama: Options vary by model; avoid sending provider-specific fields from other adapters.

Adapter Usage Snippets

import { openai } from '@tanstack/ai-openai'
import { anthropic } from '@tanstack/ai-anthropic'
import { gemini } from '@tanstack/ai-gemini'
import { ollama } from '@tanstack/ai-ollama'

const provider = openai({ apiKey: process.env.OPENAI_API_KEY })
// swap anthropic(), gemini(), or ollama({ baseUrl: process.env.OLLAMA_HOST })

Capability Checklist

• Tools: OpenAI ✅ | Anthropic ✅ | Gemini ✅ | Ollama ⚠️ (model-dependent)
• Multimodal: OpenAI vision ✅ | Anthropic vision ✅ | Gemini ✅ | Ollama ⚠️
• Streaming: All four via adapter helpers
• Native JSON mode: OpenAI & Gemini support; Anthropic via tool outputs; Ollama depends on model

Switching Providers Safely

1. Keep tool definitions provider-agnostic; avoid provider-specific enum values. 2. Gate model-specific options with TypeScript narrowing (e.g., helper isOpenAIAdapter). 3. Verify env key exists before instantiating adapter; fail fast with HTTP 400. 4. Run smoke test per provider: send messages: [{ role: 'user', content: 'ping' }] and ensure stream arrives.

React Integration Patterns (TanStack AI)

Minimal hook usage

const { messages, sendMessage, isLoading } = useChat({
  connection: fetchServerSentEvents('/api/chat'),
  tools: clientTools(showToast),
})

Local UI state + optimistic send

const [input, setInput] = useState('')
const { sendMessage, isLoading } = useChat({...})

function onSubmit(e) {
  e.preventDefault()
  sendMessage(input, { experimental_optimisticResponse: true })
  setInput('')
}

Rendering tool approvals

const { approval } = useChat({...})
return approval?.pending ? (
  <>
    <p>{approval.toolCall.name} wants to run.</p>
    <button onClick={() => approval.approve()}>Approve</button>
    <button onClick={() => approval.reject()}>Reject</button>
  </>
) : null

Error boundaries

Wrap chat UI with an error boundary to catch stream/tool errors and allow retry.

Suspense-friendly

useChat can coexist with React Suspense data fetching; keep chat state isolated to avoid tearing.

Devtools (DIY for now)

Add console logging on onChunk to inspect partial deltas during development:

useChat({
  ...,
  onChunk(chunk) {
    console.debug('chunk', chunk.delta)
  },
})

Keyboard UX

• Enter to send, Shift+Enter for newline.
• Disable send while isLoading to avoid concurrent runs, or queue messages manually.

Styling tips

• Keep messages array as single source of truth; derive UI (roles, tool outputs) from it.
• Render tool call results inline: when a message has toolResult, show a compact card with status and payload.

TanStack Start vs Next.js Routing (Streaming Chat)

TanStack Start

src/routes/api/chat.ts

import { chat, toStreamResponse } from '@tanstack/ai'
import { openai } from '@tanstack/ai-openai'
import { tools } from '@/tools/definitions'

export async function POST({ request }: { request: Request }) {
  const { messages } = await request.json()
  const stream = chat({ adapter: openai(), model: 'gpt-4o', messages, tools })
  return toStreamResponse(stream)
}

Notes:

• File-based routes; exports align with HTTP verbs.
• Runs on the same runtime as Start server functions; streaming works by default.

Next.js (App Router)

app/api/chat/route.ts

import { chat, toStreamResponse } from '@tanstack/ai'
import { openai } from '@tanstack/ai-openai'
import { tools } from '@/tools/definitions'

export const runtime = 'edge' // optional but recommended for streaming

export async function POST(req: Request) {
  const { messages } = await req.json()
  const stream = chat({ adapter: openai(), model: 'gpt-4o', messages, tools })
  return toStreamResponse(stream)
}

Notes:

• Use runtime = 'edge' for lowest latency; for Node runtime ensure dynamic = 'force-dynamic' if needed.
• In dev, Next auto-handles chunked responses; in prod verify proxy allows streaming.

Client hookup (same for both)

const { messages } = useChat({
  connection: fetchServerSentEvents('/api/chat'),
  tools: clientTools(showToast),
})

Path & Import Differences

• Start: src/routes/... and absolute imports often use $lib aliases.
• Next: app/api/... and @/ alias maps to src/.

Testing

• Run pnpm dev and hit /api/chat with curl -N to confirm streaming:

curl -N -X POST -H "Content-Type: application/json" --data '{"messages":[{"role":"user","content":"ping"}]}' http://localhost:3000/api/chat

Streaming Troubleshooting (TanStack AI)

Fast Checklist

• Server uses toStreamResponse(stream) (or toServerSentEventsStream) and returns immediately.
• Client uses matching adapter (fetchServerSentEvents for SSE).
• Reverse proxy allows streaming: disable body buffering; set Cache-Control: no-transform.
• Content-Type: text/event-stream for SSE; text/plain or application/json for chunked is fine.
• Keep-alive: ensure server sends heartbeats if long gaps; otherwise some hosts close idle connections.

Common Symptoms → Fixes

• Only first chunk shows, then hangs: proxy buffering (Vercel/Netlify edge off), or client using fetch without streaming reader—switch to provided adapter.
• CORS preflight fails: allow Accept: text/event-stream and Cache-Control headers; include Access-Control-Expose-Headers.
• “Unexpected end of JSON input”: client tries to parse JSON from a chunked stream—consume as stream via the adapter.
• Slow first token: cold start or model warmup—add tiny system prompt, or send initial heartbeat chunk.
• Stream closes mid-way: proxy timeout—lower agentLoopStrategy iterations; add server timeout guards around tools.

Reference Snippets

Server (Next.js):

export async function POST(req: Request) {
  const stream = chat({ adapter: openai(), messages, tools })
  return toStreamResponse(stream)
}

Client (React):

const { messages } = useChat({
  connection: fetchServerSentEvents('/api/chat'),
  tools: clientTools(myTool),
})

Deployment Notes

• Vercel: Edge functions stream well; Node functions require config.runtime = 'edge' or dynamic = 'force-dynamic'.
• Netlify: Enable streaming: true or use edge functions; avoid legacy lambda buffering.
• Fly/Render: Check proxy idle timeout; send heartbeat comments (data: :heartbeat).
• Nginx: proxy_buffering off; proxy_http_version 1.1; chunked_transfer_encoding on;.

Debugging Tips

• Log chunk arrival timestamps on client to spot buffering.
• In devtools Network tab, choose “Headers” and confirm Transfer-Encoding: chunked or SSE headers.
• Compare payload size: if whole response appears at once, you’re not actually streaming.

TanStack AI Quick Reference

Minimal Server (streaming)

import { chat, toStreamResponse } from '@tanstack/ai'
import { openai } from '@tanstack/ai-openai'
import { tools } from '@/tools/definitions'

export const POST = async (req: Request) => {
  const { messages, conversationId } = await req.json()
  return toStreamResponse(
    chat({
      adapter: openai(),
      model: 'gpt-4o',
      messages,
      tools,
      agentLoopStrategy: maxIterations(8),
    })
  )
}

Client (React)

import { useChat, fetchServerSentEvents } from '@tanstack/ai-react'
import { clientTools } from '@tanstack/ai-client'
import { showToastDef } from '@/tools/definitions'

const showToast = showToastDef.client(({ message }) => ({ ok: true }))

const { messages, sendMessage, approval } = useChat({
  connection: fetchServerSentEvents('/api/chat'),
  tools: clientTools(showToast),
})

Tool Definition Pattern

import { z, toolDefinition } from '@tanstack/ai'

export const showToastDef = toolDefinition({
  name: 'showToast',
  description: 'Notify the user in the UI',
  inputSchema: z.object({ message: z.string().min(3) }),
  needsApproval: false,
})

Keep names identical across server/client implementations. Definitions go to the server tools list; implementations stay where they run.

Connection Adapters Cheat Sheet

Streaming Checklist

• Response headers allow chunking/SSE (Transfer-Encoding: chunked).
• Client adapter matches server emitter.
• Proxy (Vercel/Netlify) streaming enabled; disable body buffering if needed.

Approval & Safety

• Mark destructive tools with needsApproval: true.
• Surface approval.pending in the UI with Approve / Reject buttons.
• Add timeouts around external calls inside tools to avoid stuck agent loops.

Troubleshooting

• No tool output: definitions missing from chat({ tools }).
• Stream stops after first chunk: proxy buffering or adapter mismatch.
• Model option errors: wrong adapter/model combo—check per-model typings.

Tool Patterns (TanStack AI)

1) Server-only tool (data fetch)

const getUserDef = toolDefinition({...})
export const getUser = getUserDef.server(async ({ id }) => db.user.find(id))
// Pass getUser in server tools array

Use for: DB/API reads, writes that must stay server-side.

2) Client-only tool (UI mutation)

const highlightDef = toolDefinition({...})
export const highlight = highlightDef.client(({ id }) => {
  document.querySelector(`#row-${id}`)?.classList.add('ring')
  return { highlighted: true }
})

Use for: UI updates, notifications, clipboard, local storage.

3) Hybrid (server fetch → client update)

const getUserDef = toolDefinition({...})
export const getUser = getUserDef.server(/* fetch */)
export const showUser = getUserDef.client(/* render or focus */)
// include both in tools

Use for: fetch data then update UI within one agent loop.

4) Long-running tool with polling

export const startJob = jobDef.server(async () => {
  const jobId = await queue.enqueue()
  return { jobId, pollAfterMs: 2000 }
})

Model can call the tool again after pollAfterMs to check status; cap with agentLoopStrategy.

5) Approval-gated tool

const deleteDef = toolDefinition({ needsApproval: true, ... })
export const deleteUser = deleteDef.server(/* destructive op */)
// UI shows approval.pending and calls approval.approve()

Use for: billing, deletes, external side effects.

6) Error-handling pattern

Return structured errors instead of throw:

return { error: { type: 'NotFound', message: 'User missing' } }

Model can summarize or retry; avoids unhandled stream errors.

7) Input validation

Always add Zod schemas on inputs and normalize outputs. Reject early with helpful messages.

8) Naming consistency

Tool name in toolDefinition must match client/server implementations exactly to avoid “tool not found.”

9) Observability hooks

Wrap tool implementations to log duration, input size, and approval status; redact secrets before logging.

Type Safety Cheatsheet (TanStack AI)

Per-model options

Use adapter typings to guard options:

const ai = openai()
chat({
  adapter: ai,
  model: 'gpt-4o',           // type-checked
  maxTokens: 200,            // allowed
  // temperature: 'high',    // ❌ would fail TS
})

Zod for tools

const def = toolDefinition({
  name: 'createNote',
  inputSchema: z.object({
    title: z.string().min(3),
    content: z.string().max(2000),
  }),
})

Benefits: consistent validation across providers; clear error messages back to the model.

Output shapes

Return objects with discriminated unions:

return { ok: true, noteId }
// or
return { ok: false, reason: 'NotFound' as const }

Helps the model branch correctly and avoids exceptions mid-stream.

Multimodal payloads

• OpenAI/Anthropic: send parts array with type: 'image' | 'text'.
• Gemini: ensure correct MIME (image/png, audio/wav).
• Validate file size client-side before sending to avoid provider limits.

Narrowing helpers

When sharing code across adapters, add small type guards:

function isOpenAI(adapter: Adapter): adapter is ReturnType<typeof openai> {
  return 'apiKey' in adapter
}

Gate provider-specific options inside these branches.

Agent loop guards

Type the strategy:

agentLoopStrategy: maxIterations(8)

Prevents accidental infinite loops at runtime.

Linting tips

• Enable no-floating-promises for tool implementations.
• Use @typescript-eslint/consistent-type-imports.
• Strict mode on ("strict": true in tsconfig).