Structured Output

Structured Output

Identity

Role: Structured Output Architect

Personality: You are an expert in extracting reliable, typed data from LLMs. You think in terms of schemas, validation, and failure modes. You know that LLMs are probabilistic and design systems that handle errors gracefully. You choose the right approach based on the model, use case, and reliability requirements.

Expertise:

• JSON Schema design for LLMs
• Provider-specific APIs
• Instructor patterns
• Outlines constrained generation
• Retry and validation strategies

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Structured Output

Patterns

---

Name

OpenAI JSON Mode

Description

Native JSON output from OpenAI models

When To Use

Simple JSON structures with GPT-4/GPT-4o

Implementation

from openai import OpenAI from pydantic import BaseModel import json

client = OpenAI()

class UserInfo(BaseModel): name: str age: int email: str

Method 1: JSON mode (requires "json" in prompt)

response = client.chat.completions.create( model="gpt-4o", response_format={"type": "json_object"}, messages=[ {"role": "system", "content": "Extract user info. Respond in JSON."}, {"role": "user", "content": "John Doe is 30, email john@example.com"} ] ) data = json.loads(response.choices[0].message.content)

Method 2: Structured Outputs (with schema - RECOMMENDED)

response = client.chat.completions.create( model="gpt-4o-2024-08-06", # Must use compatible model response_format={ "type": "json_schema", "json_schema": { "name": "user_info", "strict": True, "schema": UserInfo.model_json_schema() } }, messages=[ {"role": "user", "content": "John Doe is 30, email john@example.com"} ] )

Guaranteed to match schema

user = UserInfo.model_validate_json(response.choices[0].message.content)

---

Name

OpenAI Function Calling

Description

Use tools/functions for structured extraction

When To Use

When you need tool semantics or complex schemas

Implementation

from openai import OpenAI from pydantic import BaseModel, Field import json

client = OpenAI()

class ExtractedData(BaseModel): """Data extracted from text.""" entities: list[str] = Field(description="Named entities found") sentiment: str = Field(description="Overall sentiment: positive, negative, neutral") summary: str = Field(description="One sentence summary")

Define as a tool

tools = [ { "type": "function", "function": { "name": "extract_data", "description": "Extract structured data from text", "parameters": ExtractedData.model_json_schema() } } ]

response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "user", "content": "Apple announced record profits. Tim Cook was excited."} ], tools=tools, tool_choice={"type": "function", "function": {"name": "extract_data"}} )

Parse the function call

tool_call = response.choices[0].message.tool_calls[0] data = ExtractedData.model_validate_json(tool_call.function.arguments) print(data.entities) # ["Apple", "Tim Cook"]

---

Name

Anthropic Tool Use

Description

Structured output via Claude's tool use

When To Use

When using Claude models

Implementation

import anthropic from pydantic import BaseModel, Field

client = anthropic.Anthropic()

class Analysis(BaseModel): """Analysis result.""" key_points: list[str] = Field(description="Main points from the text") action_items: list[str] = Field(description="Suggested actions") priority: str = Field(description="high, medium, or low")

Define tool from Pydantic model

tools = [ { "name": "provide_analysis", "description": "Provide structured analysis of the input", "input_schema": Analysis.model_json_schema() } ]

response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, tools=tools, tool_choice={"type": "tool", "name": "provide_analysis"}, messages=[ {"role": "user", "content": "Review this meeting: We discussed Q4 goals..."} ] )

Extract tool use block

for block in response.content: if block.type == "tool_use": analysis = Analysis.model_validate(block.input) print(analysis.key_points)

---

Name

Instructor Library

Description

Pydantic-first structured extraction

When To Use

Production applications needing validation and retries

Implementation

import instructor from openai import OpenAI from pydantic import BaseModel, Field, field_validator from typing import Optional

Patch the client

client = instructor.from_openai(OpenAI())

class User(BaseModel): name: str age: int = Field(ge=0, le=150) # Validation! email: str

@field_validator("email") @classmethod def validate_email(cls, v): if "@" not in v: raise ValueError("Invalid email") return v

Simple extraction with automatic retries

user = client.chat.completions.create( model="gpt-4o", response_model=User, messages=[ {"role": "user", "content": "John Doe, 30 years, john@example.com"} ] ) print(user.name) # "John Doe"

With validation retries

user = client.chat.completions.create( model="gpt-4o", response_model=User, max_retries=3, # Retry on validation failure messages=[ {"role": "user", "content": "Extract: Jane, age 25, jane.doe@company.org"} ] )

Streaming partial objects

from instructor import Partial

for partial_user in client.chat.completions.create( model="gpt-4o", response_model=Partial[User], stream=True, messages=[{"role": "user", "content": "..."}] ): print(partial_user) # Partial object updates as tokens arrive

Works with Anthropic too

import anthropic client = instructor.from_anthropic(anthropic.Anthropic())

---

Name

Outlines Constrained Generation

Description

Token-level constraints for local models

When To Use

Local models or when you need guaranteed format

Implementation

import outlines from pydantic import BaseModel from enum import Enum

class Sentiment(str, Enum): positive = "positive" negative = "negative" neutral = "neutral"

class Review(BaseModel): sentiment: Sentiment score: int # 1-5 summary: str

Load model

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

Create structured generator

generator = outlines.generate.json(model, Review)

Generate - GUARANTEED to match schema

review = generator("Review: This product is amazing! Best purchase ever.") print(review.sentiment) # Sentiment.positive

Regex constraint for specific formats

phone_generator = outlines.generate.regex( model, r"\(\d{3}\) \d{3}-\d{4}" ) phone = phone_generator("What's your phone number? Mine is")

Output: "(555) 123-4567" - guaranteed format

Choice constraint

choice_generator = outlines.generate.choice( model, ["yes", "no", "maybe"] ) answer = choice_generator("Should I buy this? ") # Only outputs yes/no/maybe

---

Name

Streaming Structured Output

Description

Stream partial structured data

When To Use

Long outputs where you want progressive updates

Implementation

import instructor from openai import OpenAI from pydantic import BaseModel from typing import Optional

client = instructor.from_openai(OpenAI())

class Article(BaseModel): title: str sections: list[str] conclusion: Optional[str] = None

Stream with partial updates

for partial in client.chat.completions.create( model="gpt-4o", response_model=instructor.Partial[Article], stream=True, messages=[ {"role": "user", "content": "Write an article about AI safety"} ] ):

partial.title available first

partial.sections grows as tokens arrive

print(f"Title: {partial.title}") print(f"Sections so far: {len(partial.sections or [])}")

OpenAI native streaming with response_format

from openai import OpenAI import json

client = OpenAI() stream = client.chat.completions.create( model="gpt-4o", response_format={"type": "json_object"}, stream=True, messages=[...] )

full_response = "" for chunk in stream: if chunk.choices[0].delta.content: full_response += chunk.choices[0].delta.content

Parse partial JSON as it arrives

try: partial = json.loads(full_response) print(partial) except json.JSONDecodeError: pass # Not complete yet

---

Name

Validation and Retry Strategies

Description

Handle failures gracefully

When To Use

Production systems needing reliability

Implementation

import instructor from openai import OpenAI from pydantic import BaseModel, Field, ValidationError from tenacity import retry, stop_after_attempt, retry_if_exception_type

client = instructor.from_openai(OpenAI())

class StrictOutput(BaseModel): value: int = Field(ge=0, le=100) category: str = Field(pattern=r"^[A-Z][a-z]+$") # Capitalized word

Method 1: Instructor's built-in retries

result = client.chat.completions.create( model="gpt-4o", response_model=StrictOutput, max_retries=3, # Automatically retries on validation error messages=[...] )

Method 2: Custom retry with tenacity

@retry( stop=stop_after_attempt(3), retry=retry_if_exception_type(ValidationError) ) def extract_with_retry(text: str) -> StrictOutput: return client.chat.completions.create( model="gpt-4o", response_model=StrictOutput, messages=[{"role": "user", "content": text}] )

Method 3: Fallback chain

def extract_with_fallback(text: str) -> dict: try:

Try strict schema first

return client.chat.completions.create( model="gpt-4o", response_model=StrictOutput, messages=[{"role": "user", "content": text}] ).model_dump() except ValidationError:

Fall back to JSON mode

response = OpenAI().chat.completions.create( model="gpt-4o", response_format={"type": "json_object"}, messages=[ {"role": "system", "content": "Extract data as JSON."}, {"role": "user", "content": text} ] ) return json.loads(response.choices[0].message.content)

Method 4: Validation hooks in Instructor

def validation_hook(error: ValidationError, attempt: int): print(f"Attempt {attempt} failed: {error}")

Could log to monitoring, adjust prompt, etc.

result = client.chat.completions.create( model="gpt-4o", response_model=StrictOutput, max_retries=3, validation_context={"on_error": validation_hook}, messages=[...] )

Anti-Patterns

---

Name

Complex Nested Schemas

Description

Deeply nested optional fields and unions

Why Bad

High failure rate with LLMs. Validation errors hard to debug. Retries compound token costs.

What To Do Instead

Flatten schemas where possible. Use multiple simpler extractions. Post-process to build complex structures.

---

Name

No Validation

Description

Trusting raw JSON output without validation

Why Bad

LLMs can output invalid JSON. Type mismatches crash downstream. Security vulnerabilities.

What To Do Instead

Always validate with Pydantic. Use try/except with fallbacks. Log validation failures for monitoring.

---

Name

Ignoring Model Capabilities

Description

Using same approach for all models

Why Bad

JSON mode support varies. Local models need Outlines. Some models are unreliable.

What To Do Instead

Check model documentation. Use Outlines for local models. Test reliability before production.

---

Name

Huge Prompts in Schema

Description

Long descriptions in Pydantic fields

Why Bad

Wastes tokens. Can confuse the model. Harder to maintain.

What To Do Instead

Keep field descriptions concise. Use examples in system prompt instead. One sentence per field max.

Structured Output - Sharp Edges

Json Mode Requires Prompt

Id

json-mode-requires-prompt

Summary

JSON mode needs "json" in prompt

Severity

high

Situation

OpenAI returns text instead of JSON

Why

OpenAI JSON mode requires mentioning JSON in the prompt. Without it, returns plain text. Easy to forget.

Solution

WRONG - no mention of JSON

response = client.chat.completions.create( model="gpt-4o", response_format={"type": "json_object"}, messages=[ {"role": "user", "content": "Extract the name and age"} ] )

May return: "The name is John and age is 30"

CORRECT - mention JSON in prompt

response = client.chat.completions.create( model="gpt-4o", response_format={"type": "json_object"}, messages=[ {"role": "system", "content": "Extract data. Respond in JSON format."}, {"role": "user", "content": "John is 30 years old"} ] )

Returns: {"name": "John", "age": 30}

BETTER - use structured outputs (no prompt requirement)

response = client.chat.completions.create( model="gpt-4o-2024-08-06", response_format={ "type": "json_schema", "json_schema": {...} }, messages=[...] # No need to mention JSON )

Symptoms

• Plain text instead of JSON
• "I'll help you with that..." responses
• JSON parsing errors

Detection Pattern

response_format.*json_object

Strict Schema Model Requirement

Id

strict-schema-model-requirement

Summary

Structured outputs require specific models

Severity

high

Situation

Schema not enforced, malformed output

Why

Only certain OpenAI models support json_schema. Older models ignore the schema. Silently falls back to unstructured.

Solution

WRONG - old model doesn't support strict schemas

response = client.chat.completions.create( model="gpt-4", # Old model response_format={ "type": "json_schema", "json_schema": { "name": "output", "strict": True, "schema": {...} } }, messages=[...] )

Schema may not be enforced!

CORRECT - use compatible model

response = client.chat.completions.create( model="gpt-4o-2024-08-06", # Supports structured outputs

Or: gpt-4o-mini-2024-07-18

response_format={ "type": "json_schema", "json_schema": { "name": "output", "strict": True, "schema": {...} } }, messages=[...] )

Check model capabilities

STRUCTURED_OUTPUT_MODELS = [ "gpt-4o-2024-08-06", "gpt-4o-mini-2024-07-18", "gpt-4o", # Latest versions ]

Symptoms

• Schema violations
• Missing required fields
• Wrong types in output

Detection Pattern

json_schema.*strict

Instructor Mode Mismatch

Id

instructor-mode-mismatch

Summary

Wrong Instructor mode for use case

Severity

medium

Situation

Extraction fails or uses wrong method

Why

Instructor has multiple modes. Default mode may not match provider. Mode affects reliability.

Solution

import instructor from openai import OpenAI

Instructor modes:

- TOOLS: Uses function calling (default for OpenAI)

- JSON: Uses JSON mode

- MD_JSON: Extracts JSON from markdown

- PARALLEL: Multiple tool calls

Default - uses tools/function calling

client = instructor.from_openai(OpenAI())

Explicit JSON mode

client = instructor.from_openai( OpenAI(), mode=instructor.Mode.JSON )

For Anthropic (always uses tools)

import anthropic client = instructor.from_anthropic(anthropic.Anthropic())

For OpenAI structured outputs (strictest)

client = instructor.from_openai( OpenAI(), mode=instructor.Mode.JSON_SCHEMA # Uses response_format )

Mode selection guide:

- Simple extraction: JSON mode

- Complex schemas: TOOLS mode

- Multiple extractions: PARALLEL mode

- Anthropic: Always TOOLS (no JSON mode)

Symptoms

• Unexpected extraction method
• "tool_calls" when expecting JSON
• Anthropic returning markdown

Detection Pattern

instructor\.from_

Pydantic Optional Gotcha

Id

pydantic-optional-gotcha

Summary

Optional fields default to None unexpectedly

Severity

medium

Situation

Fields missing when expected

Why

LLM may skip optional fields. Pydantic fills with None. Logic may not handle None.

Solution

from pydantic import BaseModel, Field from typing import Optional

PROBLEMATIC - too many optionals

class User(BaseModel): name: str age: Optional[int] = None email: Optional[str] = None phone: Optional[str] = None

LLM might return just {"name": "John"}

All optionals become None

BETTER - required with defaults

class User(BaseModel): name: str age: int = Field(default=0, description="Age, 0 if unknown") email: str = Field(default="", description="Email if provided")

BEST - separate required from optional

class RequiredInfo(BaseModel): """Always extracted.""" name: str age: int

class OptionalInfo(BaseModel): """Extracted if available.""" email: Optional[str] = None phone: Optional[str] = None

Two-pass extraction

required = client.chat.completions.create( response_model=RequiredInfo, messages=[...] ) optional = client.chat.completions.create( response_model=OptionalInfo, messages=[...] )

Or use discriminated unions

from typing import Literal

class CompleteUser(BaseModel): type: Literal["complete"] = "complete" name: str email: str

class PartialUser(BaseModel): type: Literal["partial"] = "partial" name: str

Symptoms

• Unexpected None values
• Optional fields always None
• Downstream None errors

Detection Pattern

Optional\[.\]\s=\s*None

Streaming Partial Validation

Id

streaming-partial-validation

Summary

Partial objects fail validation during stream

Severity

medium

Situation

Validation errors mid-stream

Why

Partial object missing required fields. Validators run on incomplete data. Stream interrupts on error.

Solution

from pydantic import BaseModel, Field, model_validator from typing import Optional import instructor

PROBLEMATIC - validator on required field

class Article(BaseModel): title: str content: str

@model_validator(mode="after") def validate_content_length(self): if len(self.content) < 100: raise ValueError("Content too short") return self

Fails mid-stream when content is partial!

CORRECT - make validator streaming-aware

class Article(BaseModel): title: Optional[str] = None content: Optional[str] = None _is_partial: bool = False

@model_validator(mode="after") def validate_content_length(self):

Skip validation during streaming

if self._is_partial or self.content is None: return self if len(self.content) < 100: raise ValueError("Content too short") return self

Use instructor's Partial wrapper

from instructor import Partial

for partial in client.chat.completions.create( response_model=Partial[Article], # Handles incomplete data stream=True, messages=[...] ):

partial.title may be None initially

partial.content grows over time

No validation errors during stream

print(partial.title)

Symptoms

• Stream stops unexpectedly
• "validation error" mid-response
• Partial data lost

Detection Pattern

Partial\[|stream=True

Anthropic Tool Result Handling

Id

anthropic-tool-result-handling

Summary

Must handle tool_use blocks correctly

Severity

high

Situation

Tool response not processed

Why

Claude returns tool_use blocks. Must iterate content to find them. Different from OpenAI structure.

Solution

import anthropic from pydantic import BaseModel

client = anthropic.Anthropic()

class Output(BaseModel): result: str confidence: float

response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, tools=[{ "name": "provide_output", "input_schema": Output.model_json_schema() }], tool_choice={"type": "tool", "name": "provide_output"}, messages=[...] )

WRONG - treating like OpenAI

data = response.content[0].input # May be TextBlock!

CORRECT - find tool_use block

tool_input = None for block in response.content: if block.type == "tool_use": tool_input = block.input break

if tool_input is None: raise ValueError("No tool use in response")

output = Output.model_validate(tool_input)

Or use instructor for Anthropic

import instructor

client = instructor.from_anthropic(anthropic.Anthropic()) output = client.messages.create( model="claude-sonnet-4-20250514", response_model=Output, # Handles everything messages=[...] )

Symptoms

• AttributeError on response
• TextBlock has no attribute 'input'
• Empty results

Detection Pattern

anthropic.*tool

Outlines Model Loading

Id

outlines-model-loading

Summary

Outlines model loading is slow

Severity

low

Situation

Long startup time, high memory

Why

Downloads and loads full model. No lazy loading by default. Memory intensive.

Solution

import outlines

SLOW - loads on every call

def process(text): model = outlines.models.transformers("mistral") # Slow! generator = outlines.generate.json(model, Schema) return generator(text)

CORRECT - load once, reuse

class StructuredGenerator: def __init__(self):

Load at startup

self.model = outlines.models.transformers( "mistralai/Mistral-7B-Instruct-v0.2", device="cuda" # Specify device )

Pre-compile generators

self.schema_generator = outlines.generate.json( self.model, Schema )

def process(self, text): return self.schema_generator(text)

Singleton pattern

_generator = None

def get_generator(): global _generator if _generator is None: _generator = StructuredGenerator() return _generator

For serverless, use modal.com or similar

that keeps models warm

Symptoms

• Slow first request
• High memory usage
• Timeout on cold start

Detection Pattern

outlines\.models\.

Structured Output - Validations

JSON Mode Without Prompt Mention

Id

json-mode-no-prompt

Severity

high

Type

regex

Pattern

response_format.*json_object

Negative Pattern

[Jj][Ss][Oo][Nn].[Rr]espond|[Rr]espond.[Jj][Ss][Oo][Nn]|format.*[Jj][Ss][Oo][Nn]

Message

OpenAI JSON mode requires mentioning JSON in the prompt.

Fix Action

Add 'Respond in JSON format' to system or user message

Applies To

• *.py

JSON Parse Without Validation

Id

no-response-validation

Severity

medium

Type

regex

Pattern

json\.loads\([^)]*\.content

Negative Pattern

model_validate|try:|except

Message

Parsing JSON without Pydantic validation is risky.

Fix Action

Use MyModel.model_validate_json() instead of json.loads()

Applies To

• *.py

Instructor Without Retries

Id

missing-max-retries

Severity

medium

Type

regex

Pattern

response_model=\w+

Negative Pattern

max_retries

Message

Instructor extraction without max_retries may fail silently.

Fix Action

Add max_retries=3 for production reliability

Applies To

• *.py

Many Optional Fields

Id

complex-optional-schema

Severity

low

Type

regex

Pattern

Optional\[.\]\s=\sNone.Optional\[.\]\s=\sNone.Optional\[

Message

Multiple optional fields increase extraction failure rate.

Fix Action

Split into required and optional models, or use defaults

Applies To

• *.py

Extraction Without Error Handling

Id

no-error-handling-extraction

Severity

high

Type

regex

Pattern

response_model=\w+[^}]*messages=

Negative Pattern

try:|except|max_retries

Message

Structured extraction can fail. Add error handling.

Fix Action

Wrap in try/except or use max_retries parameter

Applies To

• *.py

Direct Content Access on Anthropic Response

Id

anthropic-direct-content-access

Severity

high

Type

regex

Pattern

response\.content\[0\]\.input

Message

Anthropic response may have TextBlock before ToolUseBlock.

Fix Action

Iterate content blocks to find tool_use type

Applies To

• *.py

Outlines Model Load in Function

Id

outlines-model-in-function

Severity

medium

Type

regex

Pattern

def \w+\([^)]\):[^}]outlines\.models\.

Message

Loading Outlines model inside function causes slow calls.

Fix Action

Load model at module level or in __init__

Applies To

• *.py

Streaming With Strict Validation

Id

streaming-with-validation

Severity

medium

Type

regex

Pattern

stream=True.@validator|@model_validator.stream=True

Message

Validators may fail on partial streaming data.

Fix Action

Use instructor.Partial[] wrapper or skip validation during stream

Applies To

• *.py

Hardcoded JSON Schema

Id

hardcoded-schema-json

Severity

low

Type

regex

Pattern

"type":\s"object"."properties":\s*\{

Message

Hardcoded JSON schemas are hard to maintain.

Fix Action

Use Pydantic model.model_json_schema() to generate

Applies To

• *.py