Ai Generating Data

Generate Synthetic Training Data

Guide the user through generating high-quality synthetic training data with DSPy. This solves the "I do not have data" problem that blocks every other AI workflow.

When NOT to generate synthetic data

• You have enough real data — 200+ labeled examples is usually enough for optimization. Real data is always better than synthetic.
• Exact-match tasks — if your task has a known correct answer (math, lookup, structured extraction from templates), write a script to generate test cases programmatically instead of using an LM.
• The LM does not understand your domain — synthetic data inherits the generator LM's biases. For highly specialized domains (medical, legal, niche industry), a few real expert-labeled examples outweigh hundreds of synthetic ones.

Step 1: Understand the data gap

Ask the user: 1. What does your AI do? (classification, extraction, Q&A, generation?) 2. How many real examples do you have? (zero, a handful, or hundreds with gaps?) 3. What is the gap? (no data at all, missing categories, edge cases, privacy constraints?) 4. What format are the inputs/outputs? (text in/category out, text in/JSON out, etc.)

Step 2: Define what an example looks like

Your generator's outputs should match your task's inputs and expected outputs.

import dspy

# Your task — what the AI will do in production
class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into a category."""
    ticket_text: str = dspy.InputField()
    category: str = dspy.OutputField()

# Generator — produces examples for your task
class GenerateTicketExample(dspy.Signature):
    """Generate a realistic support ticket with its correct category."""
    category: str = dspy.InputField(desc="the target category to generate an example for")
    ticket_text: str = dspy.OutputField(desc="a realistic support ticket for this category")

The generator's output fields become inputs to your task. Think of it as: "given what I want the answer to be, generate a realistic input."

Step 3: Write seed examples

Start with 5-10 hand-written examples. These anchor the generator's understanding of what "realistic" means for your domain.

seeds = [
    dspy.Example(
        ticket_text="I was charged twice for my subscription this month. Order #4521.",
        category="billing"
    ).with_inputs("ticket_text"),
    dspy.Example(
        ticket_text="The app crashes when I try to upload a profile photo on Android.",
        category="bug"
    ).with_inputs("ticket_text"),
    dspy.Example(
        ticket_text="How do I export my data to CSV? I cannot find the option anywhere.",
        category="how-to"
    ).with_inputs("ticket_text"),
    dspy.Example(
        ticket_text="I would love to see dark mode added. The white background hurts my eyes.",
        category="feature-request"
    ).with_inputs("ticket_text"),
    dspy.Example(
        ticket_text="My account got locked after too many login attempts. Please help.",
        category="account"
    ).with_inputs("ticket_text"),
]

Even 5 seeds dramatically improve generation quality over zero.

Step 4: Generate in batches

Pick the strategy that fits your gap:

Category-driven generation

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

categories = ["billing", "bug", "how-to", "feature-request", "account"]
examples = []

generator = dspy.Predict(GenerateTicketExample)
for category in categories:
    for i in range(50):
        result = generator(category=category)
        examples.append(
            dspy.Example(ticket_text=result.ticket_text, category=category)
            .with_inputs("ticket_text")
        )

print(f"Generated {len(examples)} examples")

Scenario-driven generation (for edge cases)

class GenerateScenarioTicket(dspy.Signature):
    """Generate a support ticket matching a specific scenario."""
    category: str = dspy.InputField()
    scenario: str = dspy.InputField(desc="the specific scenario to generate")
    ticket_text: str = dspy.OutputField()

gen = dspy.Predict(GenerateScenarioTicket)
scenarios = [
    ("billing", "customer charged in wrong currency"),
    ("billing", "refund for a cancelled subscription"),
    ("bug", "issue only happens on slow network connections"),
    ("how-to", "customer is non-technical and confused by jargon"),
]
for category, scenario in scenarios:
    result = gen(category=category, scenario=scenario)
    examples.append(dspy.Example(ticket_text=result.ticket_text, category=category).with_inputs("ticket_text"))

Programmatic generation with Faker

For structured fields (names, addresses, dates, phone numbers), Faker generates hundreds of thousands of examples instantly with zero LM cost. In one production case, 500K synthetic name records were generated with Faker + custom cultural providers, then used to fine-tune models to 96% accuracy.

from faker import Faker
from faker.providers import BaseProvider

fake = Faker()

# Custom provider for domain-specific data
class TicketProvider(BaseProvider):
    def order_id(self):
        return f"ORD-{self.random_int(1000, 99999)}"

    def product_name(self):
        return self.random_element(["Pro Plan", "Starter", "Enterprise", "Team"])

fake.add_provider(TicketProvider)

# Generate structured training records at scale
examples = []
for _ in range(10_000):
    examples.append(dspy.Example(
        ticket_text=f"Hi, I'm {fake.name()}. Order {fake.order_id()} for {fake.product_name()} "
                    f"was charged to {fake.email()} but I need it on a different card.",
        category="billing"
    ).with_inputs("ticket_text"))

When to use Faker vs LM generation:

• Faker — fields with known formats (names, emails, dates, IDs, addresses). Fast, free, structurally correct.
• LM generation — open-ended text, realistic tone, complex scenarios, domain-specific language.
• Both together — Faker for structured scaffolding, LM to add realistic surrounding context.

Diversity trick

Add a random sindex field to push the LM toward varied outputs:

import random

class GenerateDiverse(dspy.Signature):
    """Generate a unique and realistic support ticket."""
    category: str = dspy.InputField()
    sindex: str = dspy.InputField(desc="a unique seed index for diversity")
    ticket_text: str = dspy.OutputField()

gen = dspy.Predict(GenerateDiverse)
for category in categories:
    for i in range(50):
        result = gen(category=category, sindex=str(random.randint(0, 1_000_000)))
        examples.append(dspy.Example(ticket_text=result.ticket_text, category=category).with_inputs("ticket_text"))

Step 5: Filter for quality

Generated data always contains bad examples. Generate 2-3x what you need, keep ~50%.

Metric-based filtering

program = dspy.ChainOfThought(ClassifyTicket)
filtered = []

for ex in examples:
    pred = program(**ex.inputs())
    if metric(ex, pred):
        filtered.append(ex)

print(f"Kept {len(filtered)}/{len(examples)} ({100*len(filtered)//len(examples)}%)")

LM-based assessment (more robust)

class AssessExample(dspy.Signature):
    """Is this a realistic and correctly labeled example?"""
    ticket_text: str = dspy.InputField()
    category: str = dspy.InputField()
    is_realistic: bool = dspy.OutputField(desc="true if this looks like a real support ticket")
    is_correctly_labeled: bool = dspy.OutputField(desc="true if the category matches the ticket")

assessor = dspy.Predict(AssessExample)
filtered = [ex for ex in examples
    if (r := assessor(ticket_text=ex.ticket_text, category=ex.category)).is_realistic and r.is_correctly_labeled]

Deduplicate

seen = set()
unique = [ex for ex in filtered if (k := ex.ticket_text.strip().lower()) not in seen and not seen.add(k)]
filtered = unique

Step 6: Optimize the generator itself (advanced)

Optimizing the prompt used to generate data dramatically improves downstream quality. This is meta-optimization: better generator prompts produce better data.

class DataGenerator(dspy.Module):
    def __init__(self):
        self.generate = dspy.ChainOfThought(GenerateTicketExample)

    def forward(self, category):
        return self.generate(category=category)

def generator_metric(example, prediction, trace=None):
    classifier = dspy.Predict(ClassifyTicket)
    task_example = dspy.Example(ticket_text=prediction.ticket_text, category=example.category).with_inputs("ticket_text")
    task_pred = classifier(**task_example.inputs())
    return task_pred.category.lower() == example.category.lower()

optimizer = dspy.BootstrapFewShot(metric=generator_metric)
optimized_generator = optimizer.compile(DataGenerator(), trainset=seeds)

Step 7: Use generated data for optimization

from dspy.evaluate import Evaluate

random.shuffle(filtered)
split = int(len(filtered) * 0.8)
trainset, devset = filtered[:split], filtered[split:]

program = dspy.ChainOfThought(ClassifyTicket)
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized = optimizer.compile(program, trainset=trainset)

evaluator = Evaluate(devset=devset, metric=metric, num_threads=4, display_progress=True)
score = evaluator(optimized)
print(f"Score on synthetic dev set: {score:.1f}%")
# Typical: 70-85% on synthetic, validate on real data when available

optimized.save("optimized_program.json")

If you have even a small number of real examples, use them as the dev set instead — real data gives more trustworthy evaluation.

Gotchas

• Claude generates all examples with the same LM config used for the task. Use a stronger model for generation (e.g., a larger model) and a cheaper model for the task. Higher-quality generation data is worth the extra cost — it directly improves the downstream program.
• Claude forgets `.with_inputs()` on generated Examples. Every synthetic dspy.Example must call .with_inputs("field1", ...) to mark input fields. Without this, the optimizer passes all fields (including expected outputs) to the program, inflating scores.
• The `n=N` batch parameter is not supported by all providers. Claude defaults to dspy.Predict(sig, n=20) for batch generation, but Anthropic and some other providers do not support the n parameter. Use the loop pattern as a reliable fallback for any provider.
• Claude generates 50 examples and calls it done. For optimization, you typically need 200+ examples after filtering. Since filtering removes ~50%, generate at least 400-500 raw examples. More is better — generation is cheap compared to the quality improvement.
• Synthetic eval scores are inflated. If both training and evaluation data are synthetic, the eval score overestimates real-world quality. Always validate the final optimized program on real data when available, even if it is only 20-30 hand-labeled examples.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• Improving accuracy to measure and optimize your program after generating data -- see /ai-improving-accuracy
• Fine-tuning once you have enough generated data for weight optimization -- see /ai-fine-tuning
• Kickoff to scaffold a project, then fill data with this skill -- see /ai-kickoff
• Sorting for classification patterns your generated data will train -- see /ai-sorting
• Signatures for defining the generator and task input/output contracts -- see /dspy-signatures
• Install `/ai-do` if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• For end-to-end worked examples (cold start, edge cases, privacy), see examples.md

Worked Examples: Generating Synthetic Data

Example 1: Cold start — ticket classifier with zero real data

You're building a support ticket classifier for a new product. No real tickets exist yet. The PM wants a working prototype by end of week.

Define the task and generator

import dspy

lm = dspy.LM("openai/gpt-4o")
dspy.configure(lm=lm)

# What the AI does in production
class ClassifyTicket(dspy.Signature):
    """Classify a support ticket into a category."""
    ticket_text: str = dspy.InputField()
    category: str = dspy.OutputField(desc="one of: billing, bug, how-to, feature-request, account")

# What generates training examples
class GenerateTicketExample(dspy.Signature):
    """Generate a realistic support ticket for the given category. Make it sound like a real customer wrote it — varied tone, length, and detail level."""
    category: str = dspy.InputField(desc="the target category")
    sindex: str = dspy.InputField(desc="unique seed for diversity")
    ticket_text: str = dspy.OutputField(desc="a realistic support ticket")

Write seed examples (5 is enough)

seeds = [
    dspy.Example(ticket_text="I was charged twice for my subscription this month. Order #4521.", category="billing").with_inputs("ticket_text"),
    dspy.Example(ticket_text="The app crashes when I try to upload a profile photo on Android.", category="bug").with_inputs("ticket_text"),
    dspy.Example(ticket_text="How do I export my data to CSV? Can't find the option.", category="how-to").with_inputs("ticket_text"),
    dspy.Example(ticket_text="Would love to see dark mode. The white background hurts my eyes at night.", category="feature-request").with_inputs("ticket_text"),
    dspy.Example(ticket_text="My account got locked after too many login attempts. Need help ASAP.", category="account").with_inputs("ticket_text"),
]

Generate 200 examples (40 per category)

import random

categories = ["billing", "bug", "how-to", "feature-request", "account"]
generator = dspy.Predict(GenerateTicketExample)
generated = []

for category in categories:
    for i in range(40):
        result = generator(category=category, sindex=str(random.randint(0, 1_000_000)))
        generated.append(
            dspy.Example(ticket_text=result.ticket_text, category=category).with_inputs("ticket_text")
        )

print(f"Generated {len(generated)} examples")
# Generated 200 examples

Filter for quality

class AssessExample(dspy.Signature):
    """Assess whether a generated support ticket is realistic and correctly labeled."""
    ticket_text: str = dspy.InputField()
    category: str = dspy.InputField()
    is_realistic: bool = dspy.OutputField(desc="true if this reads like a real customer ticket")
    is_correctly_labeled: bool = dspy.OutputField(desc="true if the category is correct for this ticket")

assessor = dspy.Predict(AssessExample)
filtered = []

for ex in generated:
    result = assessor(ticket_text=ex.ticket_text, category=ex.category)
    if result.is_realistic and result.is_correctly_labeled:
        filtered.append(ex)

# Deduplicate
seen = set()
unique = []
for ex in filtered:
    key = ex.ticket_text.strip().lower()
    if key not in seen:
        seen.add(key)
        unique.append(ex)

filtered = unique
print(f"Kept {len(filtered)}/200 after filtering")
# Kept 156/200 after filtering

Optimize and evaluate

from dspy.evaluate import Evaluate

# Split
random.shuffle(filtered)
split = int(len(filtered) * 0.8)
trainset = filtered[:split]
devset = filtered[split:]

# Switch to cheaper model for the task
task_lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=task_lm)

def metric(example, prediction, trace=None):
    return prediction.category.strip().lower() == example.category.strip().lower()

# Baseline (no optimization)
program = dspy.ChainOfThought(ClassifyTicket)
evaluator = Evaluate(devset=devset, metric=metric, num_threads=4, display_progress=True)
baseline = evaluator(program)
print(f"Baseline: {baseline:.1f}%")
# Baseline: 72.0%

# Optimize
optimizer = dspy.MIPROv2(metric=metric, auto="light")
optimized = optimizer.compile(program, trainset=trainset)

optimized_score = evaluator(optimized)
print(f"Optimized: {optimized_score:.1f}%")
# Optimized: 89.0%

optimized.save("ticket_classifier.json")

From zero data to 89% accuracy, without a single real ticket.

Example 2: Filling edge case gaps

Your ticket classifier runs at 85% on real data, but error analysis shows it fails on:

• Angry customers with profanity and caps
• Multi-issue tickets (billing AND bug in one message)
• Non-English or mixed-language tickets

Identify the gaps

from dspy.evaluate import Evaluate

evaluator = Evaluate(devset=real_devset, metric=metric, num_threads=4, display_table=20)
score = evaluator(optimized_program)
# Look at the failures in the display table to identify patterns

Generate targeted examples for each gap

class GenerateScenarioTicket(dspy.Signature):
    """Generate a support ticket matching a specific edge case scenario. Make it realistic — these should be the tricky cases that are hard to classify correctly."""
    category: str = dspy.InputField(desc="the correct category for this ticket")
    scenario: str = dspy.InputField(desc="the edge case scenario to generate")
    ticket_text: str = dspy.OutputField(desc="a realistic ticket matching this scenario")

gen = dspy.Predict(GenerateScenarioTicket)

# Define edge case scenarios
edge_cases = [
    # Angry customers
    ("billing", "furious customer using caps and strong language about being overcharged"),
    ("bug", "frustrated user who has reported this bug three times already"),
    ("account", "angry customer locked out before an important deadline"),
    # Multi-issue tickets
    ("billing", "customer reports both a billing error AND a bug in the same ticket"),
    ("bug", "user asks how to do something AND reports a bug they found while trying"),
    ("account", "customer has account access issues AND wants a feature added"),
    # Non-English / mixed language
    ("billing", "ticket written mostly in Spanish with some English technical terms"),
    ("how-to", "ticket in broken English from a non-native speaker"),
    ("feature-request", "ticket mixing French and English"),
    ("bug", "ticket written in informal/slang English that's hard to parse"),
]

edge_examples = []
for category, scenario in edge_cases:
    for i in range(20):
        result = gen(category=category, scenario=scenario)
        edge_examples.append(
            dspy.Example(ticket_text=result.ticket_text, category=category).with_inputs("ticket_text")
        )

print(f"Generated {len(edge_examples)} edge case examples")
# Generated 200 edge case examples

Filter and merge with existing data

# Filter the edge case examples
assessor = dspy.Predict(AssessExample)
filtered_edges = []

for ex in edge_examples:
    result = assessor(ticket_text=ex.ticket_text, category=ex.category)
    if result.is_realistic and result.is_correctly_labeled:
        filtered_edges.append(ex)

print(f"Kept {len(filtered_edges)} edge case examples after filtering")

# Merge with existing training data
augmented_trainset = existing_trainset + filtered_edges
random.shuffle(augmented_trainset)

Re-optimize and compare

program = dspy.ChainOfThought(ClassifyTicket)
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
re_optimized = optimizer.compile(program, trainset=augmented_trainset)

# Evaluate on real data
score = evaluator(re_optimized)
print(f"Before edge cases: 85.0%")
print(f"After edge cases:  {score:.1f}%")
# After edge cases: 91.3%

Targeted synthetic data for specific failure modes is more effective than generating more uniform data.

Example 3: Privacy-safe dataset for medical triage

You're building a medical triage system that categorizes patient complaints. Compliance says you can't use real patient data for AI training. Everything must be synthetic.

Define task and generator with domain-specific detail

import dspy

lm = dspy.LM("openai/gpt-4o")
dspy.configure(lm=lm)

class TriageComplaint(dspy.Signature):
    """Triage a patient complaint by urgency level."""
    complaint: str = dspy.InputField(desc="patient's description of their symptoms")
    urgency: str = dspy.OutputField(desc="one of: emergency, urgent, standard, routine")

class GenerateComplaint(dspy.Signature):
    """Generate a realistic patient complaint for a medical triage system. The complaint should sound like how a real patient would describe their symptoms — using everyday language, not medical terminology. Include realistic but entirely fictional details. Never use real patient data."""
    urgency: str = dspy.InputField(desc="target urgency level: emergency, urgent, standard, or routine")
    scenario: str = dspy.InputField(desc="the medical scenario to generate")
    complaint: str = dspy.OutputField(desc="a realistic patient complaint in the patient's own words")

Define domain-specific scenarios

scenarios = {
    "emergency": [
        "chest pain with shortness of breath",
        "severe allergic reaction with swelling",
        "sudden loss of consciousness",
        "heavy uncontrolled bleeding",
        "signs of stroke — slurred speech, face drooping",
        "difficulty breathing in a child",
    ],
    "urgent": [
        "high fever lasting more than 3 days",
        "deep cut that may need stitches",
        "severe abdominal pain",
        "possible broken bone after a fall",
        "worsening infection with spreading redness",
        "persistent vomiting preventing hydration",
    ],
    "standard": [
        "mild ear infection symptoms",
        "persistent cough for two weeks",
        "minor skin rash that isn't improving",
        "recurring headaches",
        "mild back pain after lifting",
        "urinary tract infection symptoms",
    ],
    "routine": [
        "annual checkup scheduling",
        "prescription refill request",
        "vaccination appointment",
        "follow-up after normal test results",
        "minor seasonal allergy management",
        "request for medical records",
    ],
}

Generate with quality gates

Use dspy.Refine with a reward function to enforce quality during generation:

class AssessMedicalExample(dspy.Signature):
    """Assess a generated patient complaint for quality."""
    complaint: str = dspy.InputField()
    urgency: str = dspy.InputField()
    is_medically_plausible: bool = dspy.OutputField(desc="symptoms match a real medical scenario")
    urgency_is_correct: bool = dspy.OutputField(desc="urgency level is appropriate for these symptoms")
    contains_no_pii: bool = dspy.OutputField(desc="no real names, dates of birth, SSNs, or identifiable info")
    uses_patient_language: bool = dspy.OutputField(desc="written like a patient, not a doctor")

class SafeMedicalGenerator(dspy.Module):
    def __init__(self):
        self.generate = dspy.ChainOfThought(GenerateComplaint)
        self.assess = dspy.Predict(AssessMedicalExample)

    def forward(self, urgency, scenario):
        result = self.generate(urgency=urgency, scenario=scenario)
        return result

assess = dspy.Predict(AssessMedicalExample)

def medical_quality_reward(args, pred):
    """Score a generated complaint on four quality dimensions."""
    score = 1.0
    assessment = assess(complaint=pred.complaint, urgency=args["urgency"])
    if not assessment.is_medically_plausible:
        score -= 0.25
    if not assessment.urgency_is_correct:
        score -= 0.25
    if not assessment.contains_no_pii:
        score -= 0.25
    if not assessment.uses_patient_language:
        score -= 0.25
    return score

generator = dspy.Refine(
    module=SafeMedicalGenerator(),
    N=3,
    reward_fn=medical_quality_reward,
    threshold=0.75,
)

Generate and collect

import random

generated = []
for urgency, scenario_list in scenarios.items():
    for scenario in scenario_list:
        for i in range(15):
            result = generator(urgency=urgency, scenario=scenario)
            if result is not None:
                generated.append(
                    dspy.Example(complaint=result.complaint, urgency=urgency).with_inputs("complaint")
                )

print(f"Generated {len(generated)} examples")
# Generated 342 examples (some skipped when reward threshold not met)

Post-generation privacy audit

Even with quality gates, do a final pass:

class PrivacyAudit(dspy.Signature):
    """Check if text contains any personally identifiable information (PII)."""
    text: str = dspy.InputField()
    contains_pii: bool = dspy.OutputField(desc="true if text contains names, DOBs, SSNs, addresses, phone numbers, or other PII")
    pii_found: str = dspy.OutputField(desc="description of PII found, or 'none'")

auditor = dspy.Predict(PrivacyAudit)
safe = []

for ex in generated:
    result = auditor(text=ex.complaint)
    if not result.contains_pii:
        safe.append(ex)
    else:
        print(f"Removed (PII: {result.pii_found}): {ex.complaint[:60]}...")

print(f"After privacy audit: {len(safe)}/{len(generated)} examples kept")

Optimize and deploy

from dspy.evaluate import Evaluate

random.shuffle(safe)
split = int(len(safe) * 0.8)
trainset = safe[:split]
devset = safe[split:]

# Use cheaper model for the task
task_lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=task_lm)

def metric(example, prediction, trace=None):
    return prediction.urgency.strip().lower() == example.urgency.strip().lower()

program = dspy.ChainOfThought(TriageComplaint)
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized = optimizer.compile(program, trainset=trainset)

evaluator = Evaluate(devset=devset, metric=metric, num_threads=4, display_progress=True)
score = evaluator(optimized)
print(f"Triage accuracy: {score:.1f}%")

optimized.save("triage_program.json")

The result: a medical triage system trained entirely on synthetic data, with no real patient information in the training pipeline. When real data becomes available (with proper consent), mix it in as the dev set for more trustworthy evaluation.