Fine Tuning Data Generator

Fine-Tuning Data Generator

This skill generates high-quality synthetic training data in ChatML format for fine-tuning language models using frameworks like Unsloth, Axolotl, or similar tools.

What Do I Need?

Workflow

Phase 1: Gather Requirements

Start with these essential clarifying questions:

Task Definition:

• What is the model being trained to do? (e.g., customer support, code generation, creative writing)
• What specific domain or subject matter? (e.g., legal, medical, e-commerce, software development)
• How many training examples are needed? (Recommend: 100+ for simple tasks, 500-1000+ for complex)

Quality & Diversity:

• Complexity range: simple to complex mix, or focus on specific difficulty level?
• Diversity: edge cases, error handling, unusual scenarios?
• Tone/style: professional, friendly, technical, concise, detailed?
• Response length preferences?
• Any specific formats: code blocks, lists, tables, JSON?

Dataset Composition:

• Distribution across subtopics: evenly distributed or weighted?
• Include negative examples (what NOT to do)?
• Need validation split? (Recommend 10-20% of total)

See `resources/dataset-strategy.md` for detailed question templates.

Phase 2: Create Generation Plan

Present a plan covering:

• Number and distribution of examples across categories
• Key topics/scenarios to cover
• Diversity strategies (phrasing variations, complexity levels, edge cases)
• System prompt approach (consistent vs. varied)
• Quality assurance approach

Get user approval before generating.

Phase 3: Generate Synthetic Data

Create examples following these quality standards:

Key Principles:

• Realistic scenarios reflecting real-world use cases
• Natural language with varied phrasing and formality levels
• Accurate, helpful responses aligned with desired behavior
• Consistent ChatML formatting throughout
• Balanced difficulty (unless specified)
• Meaningful variety (no repetition)
• Include edge cases and error scenarios

Diversity Techniques:

• Vary query phrasing (questions, commands, statements)
• Include different expertise levels (beginner, intermediate, expert)
• Cover both positive and negative examples
• Mix short and long-form responses
• Include multi-step reasoning when appropriate
• Add context variations

See `resources/generation-techniques.md` for detailed techniques, domain-specific guidance, and batch generation workflow.

Phase 4: Validate & Document

Run validation tools and checks:

# Validate JSON formatting and structure
python scripts/validate_chatml.py training_data.jsonl

# Analyze dataset statistics and diversity
python scripts/analyze_dataset.py training_data.jsonl

# Export statistics
python scripts/analyze_dataset.py training_data.jsonl --export stats.json

Quality Checklist:

• [ ] JSON validation passed (no errors)
• [ ] Analysis shows good diversity metrics
• [ ] Manual sample review passed
• [ ] No duplicate or near-duplicate examples
• [ ] All required fields present
• [ ] Realistic user queries
• [ ] Accurate, helpful responses
• [ ] Balanced category distribution
• [ ] Dataset metadata documented

See `resources/quality-validation.md` for validation details, troubleshooting, and documentation templates.

Phase 5: Integration & Training

Prepare for training with your framework of choice:

Output Files:

• training_data.jsonl - Main training set
• validation_data.jsonl - Optional validation set
• dataset_info.txt - Metadata and statistics

Framework Setup:

• Unsloth: Automatic ChatML detection, efficient 4-bit training
• Axolotl: Specify type: chat_template and chat_template: chatml
• Hugging Face: Use tokenizer's apply_chat_template() method
• Custom: Load from JSONL, handle ChatML formatting

See `resources/framework-integration.md` for setup code, hyperparameters, deployment options, and best practices.

ChatML Format Overview

Each training example is a JSON object with a messages array:

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "How do I reverse a string in Python?"}, {"role": "assistant", "content": "Use slicing: `text[::-1]`"}]}

Roles:

• system: Sets assistant behavior (optional but recommended)
• user: User's input/query
• assistant: Model's expected response

Multi-turn: Add additional user/assistant message pairs for conversations.

See `resources/chatml-format.md` for detailed specification, validation, common issues, and framework-specific notes.

Tool Reference

Scripts in scripts/

validate_chatml.py

Validates ChatML format JSONL files:

python scripts/validate_chatml.py training_data.jsonl
python scripts/validate_chatml.py training_data.jsonl --verbose

Checks:

• Valid JSON formatting
• Required fields (messages, role, content)
• Valid role values (system, user, assistant)
• Proper message order
• Duplicate detection
• Diversity metrics

analyze_dataset.py

Provides comprehensive statistics and analysis:

python scripts/analyze_dataset.py training_data.jsonl
python scripts/analyze_dataset.py training_data.jsonl --export stats.json

Provides:

• Dataset overview (total examples, message counts)
• Message length statistics
• System prompt variations
• User query patterns (questions, commands, code-related, length categories)
• Assistant response patterns (code blocks, lists, headers, length categories)
• Quality indicators (diversity score, balance ratio)
• Token estimates and cost projection

Common Workflows

Small Dataset (100-200 examples)

1. Gather requirements 2. Create generation plan for 1-2 categories 3. Generate in single batch, review quality 4. Validate and document 5. Ready for training

Medium Dataset (500-1000 examples)

1. Gather requirements 2. Create detailed plan with multiple categories 3. Generate in 2-3 batches, reviewing after each 4. Analyze diversity and adjust approach 5. Fill any gaps 6. Final validation and documentation

Large Dataset (2000+ examples)

1. Gather comprehensive requirements 2. Create multi-batch generation plan 3. Batch 1 (50-100): Foundation examples 4. Batch 2 (100-200): Complexity expansion 5. Batch 3 (100-200): Coverage filling 6. Batch 4 (50-100): Polish and validation 7. Run full validation suite 8. Generate comprehensive documentation

Best Practices

Start Small, Iterate

1. Generate 10-20 examples first 2. Review and get feedback 3. Refine approach based on feedback 4. Scale up to full dataset

Quality Over Quantity

• Better to have 500 diverse, high-quality examples than 5,000 repetitive ones
• Each example should teach something new
• Maintain consistent response quality throughout

Diversify Systematically

• Vary query phrasing (questions, commands, statements)
• Cover different expertise levels
• Mix response complexities
• Include edge cases (typically 20-30% of dataset)
• Use batch generation workflow for large datasets

Test Before Deployment

• Test dataset with actual training framework
• Monitor training metrics for issues
• Test fine-tuned model outputs before deployment
• Compare results to base model

Document Everything

• Keep notes on generation parameters
• Save different dataset versions
• Document any modifications made
• Record generation strategies used
• Track model performance metrics

Advanced Features

Batch Generation Strategy

For datasets 500+ examples:

• Generate 50-100 examples at a time
• Review distribution and diversity after each batch
• Adjust generation strategy based on identified gaps
• Prevents repetition and maintains creativity

Common Pitfalls to Avoid

• Over-templating: Creates repetitive patterns (vary naturally)
• Unrealistic Queries: Overly formal/robotic user inputs (use varied phrasing)
• Narrow Coverage: Limited scenarios and phrasing (ensure diversity)
• Inconsistent Quality: Quality degradation over time (use quality checklist)
• JSON Errors: Invalid formatting breaking training (always validate)
• Missing Context: System prompts without detail (provide clear instructions)
• Response Mismatch: Responses don't address queries (verify relevance)

Dataset Size Recommendations

Resources

• Planning & Strategy: `resources/dataset-strategy.md` - Requirements gathering, planning, quality checklists
• Generation Techniques: `resources/generation-techniques.md` - Diversity techniques, domain-specific guidance, batch workflows
• ChatML Specification: `resources/chatml-format.md` - Format details, validation, framework notes
• Example Datasets: `resources/examples.md` - Diverse domain examples, multi-turn patterns
• Quality Validation: `resources/quality-validation.md` - Validation workflow, analysis, troubleshooting
• Framework Integration: `resources/framework-integration.md` - Setup for Unsloth, Axolotl, HuggingFace; deployment options

---

Version: 2.0 | Updated: 2024 | Pattern: Modular Orchestration

Fine-Tuning Data Generator Skill

Generate high-quality synthetic training data in ChatML format for fine-tuning language models.

What This Skill Does

This skill helps you create comprehensive fine-tuning datasets by:

• Asking clarifying questions about your requirements
• Creating a detailed generation plan
• Generating diverse, realistic training examples
• Outputting data in ChatML JSONL format
• Validating the dataset quality

When to Use

Use this skill when you need to:

• Create training data for fine-tuning models with Unsloth, Axolotl, or similar frameworks
• Generate synthetic conversations for specific domains or tasks
• Build datasets for instruction-following, Q&A, code generation, or other tasks
• Ensure consistent ChatML formatting for your training data

Quick Start

1. Request dataset generation:

   "I need to create fine-tuning data for a customer support chatbot in the e-commerce domain"

2. Answer clarifying questions about:

• Number of examples needed
• Domain and task type
• Diversity requirements
• Response style and tone

3. Review the generation plan before Claude creates the data

4. Receive your dataset as JSONL files ready for training

Output Files

• training_data.jsonl - Main training dataset
• validation_data.jsonl - Validation set (optional)
• dataset_info.txt - Statistics and metadata

Validation Tools

Validate ChatML Format

python scripts/validate_chatml.py training_data.jsonl

Analyze Dataset

python scripts/analyze_dataset.py training_data.jsonl
python scripts/analyze_dataset.py training_data.jsonl --export stats.json

Examples

The skill can generate data for various domains:

• Customer Support: Help desk, troubleshooting, order management
• Code Generation: Algorithm implementation, bug fixes, code review
• Technical Writing: Documentation, API specs, tutorials
• Data Analysis: Query writing, data interpretation, visualization
• Creative Writing: Stories, content creation, editing
• Education: Concept explanations, tutoring, learning materials

Resources

• resources/chatml-format.md - Detailed ChatML format specification
• resources/examples.md - Extended examples across domains
• templates/generation-plan.md - Generation plan template

ChatML Format

Each example follows this structure:

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Tips for Best Results

1. Be specific about your use case and domain 2. Request diverse examples covering edge cases 3. Review the first batch before generating large datasets 4. Validate early using the provided scripts 5. Test with actual training to verify quality

Training Framework Compatibility

This skill generates standard ChatML format compatible with:

• Unsloth: Direct compatibility
• Axolotl: Use type: chat_template in config
• Hugging Face: Load with datasets.load_dataset('json', ...)
• Most modern fine-tuning frameworks

Quality Assurance

The skill ensures:

• Valid JSON formatting
• No duplicate examples
• Natural, realistic queries
• Accurate, helpful responses
• Diverse scenarios and phrasing
• Appropriate difficulty distribution

Support

For issues or questions:

• Review the SKILL.md file for detailed instructions
• Check examples in resources/examples.md
• Run validation scripts to identify issues
• Consult ChatML format specification in resources/chatml-format.md

ChatML Format Specification

Overview

ChatML (Chat Markup Language) is a standardized format for representing conversational data in language model fine-tuning. It's widely supported by training frameworks like Unsloth, Axolotl, and others.

File Format

ChatML datasets are typically stored as JSONL (JSON Lines) files, where each line is a valid JSON object representing one training example.

Structure

Basic Structure

Each line in the JSONL file represents one conversation example:

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Required Fields

• messages: Array of message objects representing the conversation
• Each message object contains:
• role: One of "system", "user", or "assistant"
• content: The message text (string)

Role Types

1. system: Sets the behavior/context for the assistant (optional but recommended) 2. user: Represents the user's input/query 3. assistant: Represents the model's expected response

Conversation Patterns

Single-Turn (Simple Q&A)

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}]}

Multi-Turn (Dialogue)

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}, {"role": "user", "content": "What's the population?"}, {"role": "assistant", "content": "Paris has a population of approximately 2.2 million people in the city proper, and about 12 million in the metropolitan area."}]}

Without System Prompt

{"messages": [{"role": "user", "content": "Translate 'hello' to Spanish"}, {"role": "assistant", "content": "The Spanish translation of 'hello' is 'hola'."}]}

Best Practices

System Prompts

Consistent System Prompt (Recommended for most cases):

• Use the same system prompt across all examples
• Defines consistent behavior for the model
• Example: "You are a helpful, harmless, and honest assistant."

Varied System Prompts (Advanced use cases):

• Different system prompts for different task types
• Useful for multi-task models
• Ensure sufficient examples per system prompt variation

Content Guidelines

1. Escape Special Characters: Properly escape quotes, newlines, etc.

   {"role": "user", "content": "She said \"hello\" to me\nWhat should I say back?"}

2. Preserve Formatting: Maintain code blocks, lists, and structure

   {"role": "assistant", "content": "Here's the code:\n\n```python\nprint('hello')\n```\n\nThis prints hello to the console."}

3. Handle Empty Content: Avoid empty content strings

   // BAD
   {"role": "user", "content": ""}

   // GOOD
   {"role": "user", "content": "Hello"}

Complete Example File

training_data.jsonl:

{"messages": [{"role": "system", "content": "You are a Python expert."}, {"role": "user", "content": "How do I reverse a string?"}, {"role": "assistant", "content": "You can reverse a string in Python using slicing: `reversed_string = my_string[::-1]`"}]}
{"messages": [{"role": "system", "content": "You are a Python expert."}, {"role": "user", "content": "What's the difference between a list and a tuple?"}, {"role": "assistant", "content": "Lists are mutable (can be changed) and use square brackets []. Tuples are immutable (cannot be changed) and use parentheses (). Example:\n- List: `my_list = [1, 2, 3]`\n- Tuple: `my_tuple = (1, 2, 3)`"}]}
{"messages": [{"role": "system", "content": "You are a Python expert."}, {"role": "user", "content": "Show me how to read a file"}, {"role": "assistant", "content": "Here's the recommended way to read a file in Python:\n\n```python\nwith open('file.txt', 'r') as f:\n    content = f.read()\n    print(content)\n```\n\nUsing `with` ensures the file is properly closed even if an error occurs."}]}

Validation

Valid JSON Check

Each line must be valid JSON:

# Check if file is valid JSONL
while IFS= read -r line; do
  echo "$line" | jq . > /dev/null || echo "Invalid JSON"
done < training_data.jsonl

Required Fields Check

import json

with open('training_data.jsonl', 'r') as f:
    for i, line in enumerate(f, 1):
        data = json.loads(line)
        assert 'messages' in data, f"Line {i}: missing 'messages'"
        assert isinstance(data['messages'], list), f"Line {i}: 'messages' must be array"
        for msg in data['messages']:
            assert 'role' in msg, f"Line {i}: message missing 'role'"
            assert 'content' in msg, f"Line {i}: message missing 'content'"
            assert msg['role'] in ['system', 'user', 'assistant'], f"Line {i}: invalid role"

Common Issues

Issue 1: Invalid JSON

// BAD - Missing quotes around property names
{messages: [{role: "user", content: "test"}]}

// GOOD
{"messages": [{"role": "user", "content": "test"}]}

Issue 2: Incorrect Nesting

// BAD - messages should be an array
{"messages": {"role": "user", "content": "test"}}

// GOOD
{"messages": [{"role": "user", "content": "test"}]}

Issue 3: Newline Characters

// BAD - Literal newlines break JSONL format
{"messages": [{"role": "user",
"content": "test"}]}

// GOOD - One line per example
{"messages": [{"role": "user", "content": "test"}]}

Issue 4: Unescaped Characters

// BAD
{"messages": [{"role": "user", "content": "She said "hello""}]}

// GOOD
{"messages": [{"role": "user", "content": "She said \"hello\""}]}

Framework-Specific Notes

Unsloth

• Supports standard ChatML format
• Can specify chat template in training config
• Handles tokenization automatically

Axolotl

• Use type: chat_template in dataset config
• Supports various chat templates (chatml, llama2, etc.)
• Can specify custom template if needed

Hugging Face

• Load with datasets library:

  from datasets import load_dataset
  dataset = load_dataset('json', data_files='training_data.jsonl')

Token Counting

Different models tokenize differently. Example lengths:

• Short user query: ~10-20 tokens
• Medium assistant response: ~50-150 tokens
• Long detailed response: ~200-500 tokens
• Code examples: Varies widely (100-1000+ tokens)

Plan your dataset size accounting for context window limits of your target model.

Dataset Size Recommendations

• Small tasks: 100-500 examples
• Medium tasks: 500-2,000 examples
• Complex tasks: 2,000-10,000+ examples
• Domain adaptation: 1,000-5,000 examples

Quality > Quantity. Better to have 500 diverse, high-quality examples than 5,000 repetitive ones.

Dataset Strategy & Planning

Gathering Requirements

Essential Questions

When a user requests fine-tuning data generation, gather these requirements:

Task Definition

• Task Type: What is the model being trained to do?
• Examples: customer support, code generation, creative writing, technical Q&A, instruction following, classification, summarization
• Domain/Topic: What specific domain or subject matter?
• Examples: legal, medical, e-commerce, software development, finance
• Number of Examples: How many training examples are needed?
• Recommendation: minimum 100 for simple tasks, 500-1000+ for complex tasks

Quality & Diversity

• Complexity Range: Simple to complex mix, or focus on specific difficulty level?
• Diversity Requirements:
• Edge cases, error handling, unusual scenarios?
• Variation in query phrasing and response styles?
• Multi-turn conversations or single-turn only?
• Tone/Style: What tone should the assistant use?
• Examples: professional, friendly, concise, detailed, technical
• Response Length: Preferred length for assistant responses?
• Examples: brief answers, detailed explanations, step-by-step guides
• Special Formats: Specific formats to include?
• Examples: code blocks, lists, tables, JSON

Dataset Composition

• Distribution: Evenly distributed across subtopics or weighted toward specific areas?
• Include Negatives: Examples of what NOT to do or incorrect approaches?
• Validation Split: Need separate validation set? (Recommend 10-20% of total)

Creating a Generation Plan

After gathering requirements, present a comprehensive plan:

Plan Components:

• Number of examples and distribution across categories
• Key topics/scenarios to cover
• Diversity strategies (phrasing variations, complexity levels, edge cases)
• System prompt approach (consistent vs. varied)
• Quality assurance approach

Get user approval before generating.

Generation Principles

Quality Standards

• Realistic Scenarios: Reflect real-world use cases
• Natural Language: Varied phrasing, different formality levels, human-like queries
• Accurate Responses: Correct, helpful, aligned with desired behavior
• Consistent Formatting: Proper ChatML structure throughout
• Balanced Difficulty: Mix of simple and complex (unless specified)
• Avoid Repetition: Each example meaningfully different
• Include Edge Cases: Boundary conditions, ambiguous queries, error scenarios

Diversity Techniques

• Vary query phrasing (questions, commands, statements)
• Include different user expertise levels (beginner, intermediate, expert)
• Cover positive and negative examples
• Mix short and long-form responses
• Include multi-step reasoning when appropriate
• Add context variations (different scenarios, parameters, constraints)

Batch Generation Strategy

For large datasets (500+ examples):

• Generate 50-100 examples at a time
• Review distribution and diversity after each batch
• Adjust generation strategy based on gaps or over-representation
• Prevents repetition and maintains creativity

Quality Control Checklist

Before delivering the dataset:

• [ ] All examples are valid JSON
• [ ] No duplicate or near-duplicate examples
• [ ] System prompts are appropriate and consistent (or intentionally varied)
• [ ] User queries are natural and realistic
• [ ] Assistant responses are accurate and helpful
• [ ] Distribution across categories is balanced (or as specified)
• [ ] Edge cases and error scenarios are included
• [ ] Multi-turn examples flow naturally
• [ ] Dataset statistics are documented
• [ ] Validation script passes

Common Pitfalls to Avoid

• Over-templating: Rigid templates create repetitive patterns
• Unrealistic Queries: Overly formal or robotic user inputs
• Inconsistent Quality: Maintain consistent response quality
• Narrow Coverage: Ensure sufficient diversity in scenarios
• JSON Errors: Always validate JSON formatting
• Missing Context: Include necessary context in system prompts
• Response Mismatch: Ensure responses actually address queries

Dataset Size Recommendations

Quality > Quantity: Better to have 500 diverse, high-quality examples than 5,000 repetitive ones.

Tips for Best Results

1. Start Small: Generate 10-20 examples first, review, then scale 2. Iterate: Refine generation approach based on initial batch 3. Use Real Data: If available, use real examples as inspiration (generate synthetic variations) 4. Test Early: Test dataset with actual training to validate quality 5. Version Control: Save different versions as you refine approach 6. Document Decisions: Keep track of generation parameters and strategies

Output Organization

File Structure

• Primary Output: training_data.jsonl (main training set)
• Optional: validation_data.jsonl (eval set if requested)
• Metadata: dataset_info.txt (composition, statistics)

Dataset Statistics to Include

• Total number of examples
• Distribution across categories/topics
• Average user query length
• Average assistant response length
• System prompts used (number of variations)
• Multi-turn vs single-turn ratio
• Estimated token counts

Dataset Info Template

Generated Fine-Tuning Dataset Summary
=====================================

Files Created:
- training_data.jsonl (X examples)
- validation_data.jsonl (Y examples) [if requested]
- dataset_info.txt (metadata)

Dataset Statistics:
- Total training examples: X
- Total validation examples: Y
- Average user query length: Z tokens
- Average assistant response length: W tokens
- System prompts: [number] variation(s)
- Multi-turn conversations: N%
- Single-turn conversations: M%

Category Distribution:
- Category 1: X examples (Y%)
- Category 2: X examples (Y%)
- ...

Next Steps:
1. Review sample examples in training_data.jsonl
2. Load dataset into training framework
3. Adjust hyperparameters based on dataset size
4. Monitor training metrics

Extended ChatML Examples

This document provides diverse examples across different domains and use cases to serve as inspiration for generating synthetic training data.

Domain: Customer Support

Example 1: Order Issue

{"messages": [{"role": "system", "content": "You are a helpful customer support agent for an e-commerce platform. Be empathetic and solution-oriented."}, {"role": "user", "content": "My package was supposed to arrive yesterday but it still hasn't come. Order #12345"}, {"role": "assistant", "content": "I'm sorry to hear your package hasn't arrived yet. Let me check on order #12345 for you.\n\nI can see that your package is currently delayed due to weather conditions in the shipping area. It's now expected to arrive tomorrow by 8 PM. \n\nTo make up for this inconvenience, I've applied a $10 credit to your account that you can use on your next purchase. You'll also receive an email with tracking updates every 4 hours until delivery.\n\nIs there anything else I can help you with regarding this order?"}]}

Example 2: Product Return

{"messages": [{"role": "system", "content": "You are a helpful customer support agent for an e-commerce platform. Be empathetic and solution-oriented."}, {"role": "user", "content": "i bought shoes last week but they dont fit can i return them"}, {"role": "assistant", "content": "Absolutely! Our return policy allows returns within 30 days of purchase. Here's how to return your shoes:\n\n1. Go to 'My Orders' in your account\n2. Find your shoe order and click 'Start Return'\n3. Select 'Wrong size' as the reason\n4. Print the prepaid return label\n5. Pack the shoes in their original box if possible\n6. Drop off at any UPS location\n\nOnce we receive the shoes (usually 5-7 business days), your refund will be processed to your original payment method within 3-5 business days.\n\nWould you like to exchange them for a different size instead? I can arrange a simultaneous exchange to get the right size to you faster!"}]}

Domain: Code Generation

Example 1: Algorithm Implementation

{"messages": [{"role": "system", "content": "You are an expert software engineer. Provide clean, efficient, well-documented code with explanations."}, {"role": "user", "content": "Write a Python function to check if a binary tree is balanced"}, {"role": "assistant", "content": "Here's a solution that checks if a binary tree is balanced (height-balanced, where no two subtrees differ in height by more than 1):\n\n```python\nclass TreeNode:\n    def __init__(self, val=0, left=None, right=None):\n        self.val = val\n        self.left = left\n        self.right = right\n\ndef is_balanced(root: TreeNode) -> bool:\n    \"\"\"\n    Check if a binary tree is height-balanced.\n    \n    Args:\n        root: Root node of the binary tree\n    \n    Returns:\n        True if balanced, False otherwise\n    \"\"\"\n    def check_height(node):\n        # Base case: empty tree is balanced with height 0\n        if not node:\n            return 0\n        \n        # Check left subtree\n        left_height = check_height(node.left)\n        if left_height == -1:  # Left subtree is not balanced\n            return -1\n        \n        # Check right subtree\n        right_height = check_height(node.right)\n        if right_height == -1:  # Right subtree is not balanced\n            return -1\n        \n        # Check if current node is balanced\n        if abs(left_height - right_height) > 1:\n            return -1\n        \n        # Return height of current subtree\n        return max(left_height, right_height) + 1\n    \n    return check_height(root) != -1\n```\n\n**Time Complexity**: O(n) - visits each node once\n**Space Complexity**: O(h) - recursion stack, where h is tree height\n\n**Example usage**:\n```python\n# Create a balanced tree:     1\n#                            / \\\n#                           2   3\nroot = TreeNode(1)\nroot.left = TreeNode(2)\nroot.right = TreeNode(3)\nprint(is_balanced(root))  # True\n\n# Create an unbalanced tree:  1\n#                            /\n#                           2\n#                          /\n#                         3\nroot2 = TreeNode(1)\nroot2.left = TreeNode(2)\nroot2.left.left = TreeNode(3)\nprint(is_balanced(root2))  # False\n```"}]}

Example 2: Bug Fix

{"messages": [{"role": "system", "content": "You are an expert software engineer. Provide clean, efficient, well-documented code with explanations."}, {"role": "user", "content": "This code is supposed to remove duplicates from a list but it's not working:\n\ndef remove_dupes(lst):\n    for item in lst:\n        if lst.count(item) > 1:\n            lst.remove(item)\n    return lst"}, {"role": "assistant", "content": "The issue with this code is that it modifies the list while iterating over it, which can cause items to be skipped. Additionally, `lst.count()` and repeated `lst.remove()` calls make it inefficient.\n\nHere are two better approaches:\n\n**Approach 1: Using set (preserves no order, but fast)**\n```python\ndef remove_dupes(lst):\n    return list(set(lst))\n```\n\n**Approach 2: Preserve order**\n```python\ndef remove_dupes(lst):\n    seen = set()\n    result = []\n    for item in lst:\n        if item not in seen:\n            seen.add(item)\n            result.append(item)\n    return result\n```\n\n**Or more concisely with dict.fromkeys():**\n```python\ndef remove_dupes(lst):\n    return list(dict.fromkeys(lst))\n```\n\n**Examples:**\n```python\nlst = [1, 2, 2, 3, 4, 3, 5]\nprint(remove_dupes(lst))  # [1, 2, 3, 4, 5]\n\nlst2 = ['a', 'b', 'a', 'c']\nprint(remove_dupes(lst2))  # ['a', 'b', 'c']\n```\n\nThe dict.fromkeys() approach is usually best: it preserves order (Python 3.7+) and is concise."}]}

Domain: Technical Writing

Example 1: Documentation Request

{"messages": [{"role": "system", "content": "You are a technical writer who creates clear, comprehensive documentation."}, {"role": "user", "content": "Write API documentation for a user registration endpoint"}, {"role": "assistant", "content": "# User Registration API\n\n## Endpoint\n```\nPOST /api/v1/users/register\n```\n\n## Description\nCreates a new user account in the system.\n\n## Request Headers\n| Header | Type | Required | Description |\n|--------|------|----------|-------------|\n| Content-Type | string | Yes | Must be `application/json` |\n\n## Request Body\n```json\n{\n  \"email\": \"string\",\n  \"password\": \"string\",\n  \"username\": \"string\",\n  \"firstName\": \"string\",\n  \"lastName\": \"string\"\n}\n```\n\n### Parameters\n| Field | Type | Required | Constraints | Description |\n|-------|------|----------|-------------|-------------|\n| email | string | Yes | Valid email format, unique | User's email address |\n| password | string | Yes | Min 8 characters, must include uppercase, lowercase, and number | User's password |\n| username | string | Yes | 3-20 characters, alphanumeric only, unique | Unique username |\n| firstName | string | No | Max 50 characters | User's first name |\n| lastName | string | No | Max 50 characters | User's last name |\n\n## Response\n\n### Success Response (201 Created)\n```json\n{\n  \"id\": \"string\",\n  \"email\": \"string\",\n  \"username\": \"string\",\n  \"firstName\": \"string\",\n  \"lastName\": \"string\",\n  \"createdAt\": \"string (ISO 8601)\",\n  \"token\": \"string\"\n}\n```\n\n### Error Responses\n\n**400 Bad Request** - Invalid input\n```json\n{\n  \"error\": \"VALIDATION_ERROR\",\n  \"message\": \"Invalid email format\",\n  \"fields\": [\"email\"]\n}\n```\n\n**409 Conflict** - Email or username already exists\n```json\n{\n  \"error\": \"DUPLICATE_USER\",\n  \"message\": \"Email already registered\"\n}\n```\n\n## Example Request\n```bash\ncurl -X POST https://api.example.com/api/v1/users/register \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"email\": \"user@example.com\",\n    \"password\": \"SecurePass123\",\n    \"username\": \"johndoe\",\n    \"firstName\": \"John\",\n    \"lastName\": \"Doe\"\n  }'\n```\n\n## Example Response\n```json\n{\n  \"id\": \"usr_1234567890\",\n  \"email\": \"user@example.com\",\n  \"username\": \"johndoe\",\n  \"firstName\": \"John\",\n  \"lastName\": \"Doe\",\n  \"createdAt\": \"2024-01-15T10:30:00Z\",\n  \"token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...\"\n}\n```\n\n## Notes\n- Passwords are hashed using bcrypt before storage\n- The returned token is a JWT valid for 24 hours\n- Email verification is sent automatically upon registration\n- Rate limit: 5 requests per minute per IP address"}]}

Domain: Data Analysis

Example 1: Data Question

{"messages": [{"role": "system", "content": "You are a data analyst expert. Provide clear explanations and actionable insights from data."}, {"role": "user", "content": "I have a CSV with columns: date, product, sales, region. How do I find the top performing product in each region?"}, {"role": "assistant", "content": "Here's how to find the top performing product in each region using Python with pandas:\n\n```python\nimport pandas as pd\n\n# Load your data\ndf = pd.read_csv('sales_data.csv')\n\n# Method 1: Using groupby and idxmax\ntop_products = df.loc[df.groupby('region')['sales'].idxmax()]\nprint(top_products[['region', 'product', 'sales']])\n\n# Method 2: Aggregate first, then get max per region\ntop_products_agg = (\n    df.groupby(['region', 'product'])['sales']\n    .sum()\n    .reset_index()\n    .loc[lambda x: x.groupby('region')['sales'].idxmax()]\n)\nprint(top_products_agg)\n\n# Method 3: Using sort_values and drop_duplicates\ntop_products_sorted = (\n    df.sort_values('sales', ascending=False)\n    .drop_duplicates('region')\n    [['region', 'product', 'sales']]\n)\nprint(top_products_sorted)\n```\n\n**Which method to use:**\n- **Method 1**: Best if you want the single highest sale transaction per region\n- **Method 2**: Best if you want total sales by product, then top product per region\n- **Method 3**: Simplest and most readable\n\n**Example output:**\n```\n   region      product  sales\n0  North   Widget A    15000\n1  South   Widget B    18000\n2  East    Widget C    12000\n3  West    Widget A    16000\n```\n\n**Bonus - Visualization:**\n```python\nimport matplotlib.pyplot as plt\n\ntop_products.plot(kind='bar', x='region', y='sales', title='Top Product by Region')\nplt.ylabel('Sales ($)')\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n```"}]}

Domain: Creative Writing

Example 1: Story Prompt

{"messages": [{"role": "system", "content": "You are a creative writing assistant. Help users develop engaging stories with vivid descriptions and compelling narratives."}, {"role": "user", "content": "Write a short opening paragraph for a sci-fi story about a colony on Mars discovering something unexpected underground"}, {"role": "assistant", "content": "The drilling alarm pierced through the thin Martian atmosphere at 0400 hours, jolting Commander Sarah Chen from her sleep. For six months, the excavation team had been boring through rust-red regolith, searching for ice deposits to sustain New Beijing Colony. Instead, at precisely 847 meters depth, the drill bit had struck something that shouldn't exist: a hollow chamber, and inside it, a perfect geometric structure emitting a faint electromagnetic pulse that matched no known natural phenomenon. As Sarah pulled on her pressure suit and rushed toward the drill site, she couldn't shake the impossible thought crystallizing in her mind—humanity might not be Mars's first visitors after all."}]}

Domain: Educational Content

Example 1: Concept Explanation

{"messages": [{"role": "system", "content": "You are an expert educator. Explain concepts clearly using analogies and examples appropriate for the learner's level."}, {"role": "user", "content": "explain what recursion is in programming like im 10 years old"}, {"role": "assistant", "content": "Imagine you have a big box of toys, and inside that box are smaller boxes, and inside those are even smaller boxes, and so on.\n\nYou want to count all the toys, but they're hidden in all these nested boxes. Here's what you do:\n\n1. Open a box\n2. If you find a toy, count it\n3. If you find another box, open it and go back to step 1\n4. Keep doing this until there are no more boxes\n\n**That's recursion!** It's when something uses itself to solve a problem.\n\nIn programming, it's like a function that calls itself:\n\n```python\ndef count_toys_in_box(box):\n    count = 0\n    for item in box:\n        if item is a toy:\n            count = count + 1\n        if item is a box:\n            count = count + count_toys_in_box(item)  # Here it calls itself!\n    return count\n```\n\nThe function `count_toys_in_box` calls itself whenever it finds another box. Pretty cool, right?\n\n**Real-world example:** Your family tree! You have parents, they have parents (your grandparents), and they have parents (great-grandparents). Each generation is like the function calling itself to go further back in time."}]}

Domain: Legal/Compliance

Example 1: Policy Question

{"messages": [{"role": "system", "content": "You are a helpful assistant knowledgeable about GDPR compliance. Provide accurate information but remind users to consult legal professionals for specific advice."}, {"role": "user", "content": "Do I need to appoint a Data Protection Officer for my startup?"}, {"role": "assistant", "content": "Under GDPR Article 37, you must appoint a Data Protection Officer (DPO) if your organization meets any of these criteria:\n\n**1. Public Authority**\n   - Any public authority or body (except courts)\n\n**2. Core Activities Require Large-Scale Monitoring**\n   - Your main business involves regular and systematic monitoring of individuals at large scale\n   - Examples: behavioral advertising, tracking, profiling, location tracking\n\n**3. Core Activities Involve Special Categories of Data at Large Scale**\n   - Processing sensitive data: health, genetic, biometric, racial, political opinions, etc.\n   - Examples: healthcare apps, background check services\n\n**For most startups:**\n- If you're a typical SaaS, e-commerce, or mobile app company processing standard user data (names, emails, usage data), you likely **do not need** a DPO\n- However, if you do any of these, you probably **do need** one:\n  - Ad-tech platform tracking users across sites\n  - Health/medical app processing patient data\n  - HR platform processing employee data for many companies\n  - Credit scoring or financial profiling\n\n**Alternative to DPO:**\nEven if not required, you should designate someone responsible for privacy compliance internally.\n\n**Important:** This is general guidance. Consult with a privacy lawyer to assess your specific situation, as requirements vary by:\n- Your business model\n- Types of data processed\n- Scale of processing\n- Jurisdictions you operate in\n\nWould you like me to help you assess whether your startup's activities might qualify as \"large-scale monitoring\"?"}]}

Multi-Turn Conversation Examples

Example 1: Debugging Session

{"messages": [{"role": "system", "content": "You are an expert software engineer helping debug code issues."}, {"role": "user", "content": "My Python script keeps throwing a KeyError when I try to access dictionary values"}, {"role": "assistant", "content": "A KeyError occurs when you try to access a dictionary key that doesn't exist. Can you share the code that's causing the error? Specifically:\n\n1. The line where the KeyError occurs\n2. How the dictionary is created\n3. The full error message\n\nThis will help me pinpoint the exact issue."}, {"role": "user", "content": "Here's the code:\n\ndata = {'name': 'John', 'age': 30}\nprint(data['address'])"}, {"role": "assistant", "content": "Found the issue! You're trying to access `data['address']`, but the dictionary only has `'name'` and `'age'` keys. There's no `'address'` key, which causes the KeyError.\n\n**Solutions:**\n\n**1. Use `.get()` method (safest)**\n```python\naddress = data.get('address', 'Not provided')\nprint(address)  # Prints: Not provided\n```\n\n**2. Check if key exists first**\n```python\nif 'address' in data:\n    print(data['address'])\nelse:\n    print('Address not found')\n```\n\n**3. Use try/except**\n```python\ntry:\n    print(data['address'])\nexcept KeyError:\n    print('Address key does not exist')\n```\n\n**I recommend option 1** (`.get()`) because it's concise and lets you provide a default value.\n\nWas this dictionary supposed to have an address field, or should you be handling missing keys?"}]}

Edge Cases and Error Handling

Example 1: Handling Invalid Input

{"messages": [{"role": "system", "content": "You are a helpful assistant. When users provide unclear or invalid input, politely ask for clarification."}, {"role": "user", "content": "fix it"}, {"role": "assistant", "content": "I'd be happy to help fix something! However, I need a bit more information:\n\n1. What specifically needs to be fixed? (code, error, document, etc.)\n2. What is the current problem or error you're experiencing?\n3. Can you share the relevant code, error message, or details?\n\nOnce you provide these details, I'll be able to assist you effectively!"}]}

Example 2: Ambiguous Request

{"messages": [{"role": "system", "content": "You are a helpful programming assistant."}, {"role": "user", "content": "How do I sort?"}, {"role": "assistant", "content": "I can help you with sorting! To provide the best solution, I need to know:\n\n1. **What are you sorting?** (a list, array, database results, files, etc.)\n2. **What programming language?** (Python, JavaScript, Java, SQL, etc.)\n3. **Sort by what criteria?** (alphabetically, numerically, by date, custom criteria?)\n4. **Sort order?** (ascending or descending?)\n\nFor example:\n- \"Sort a list of numbers in Python\"\n- \"Sort array of objects by date in JavaScript\"\n- \"Sort database query results by name in SQL\"\n\nLet me know these details and I'll provide the exact code you need!"}]}

Different Response Styles

Concise Style

{"messages": [{"role": "system", "content": "You are a helpful assistant. Provide concise, direct answers."}, {"role": "user", "content": "What's the difference between let and const in JavaScript?"}, {"role": "assistant", "content": "`let`: Declares a variable that can be reassigned\n```javascript\nlet x = 5;\nx = 10; // ✓ Works\n```\n\n`const`: Declares a constant that cannot be reassigned\n```javascript\nconst y = 5;\ny = 10; // ✗ Error\n```\n\nNote: `const` objects/arrays can have their properties/elements modified, just not reassigned entirely."}]}

Detailed Style

{"messages": [{"role": "system", "content": "You are a thorough educator. Provide comprehensive explanations with examples and context."}, {"role": "user", "content": "What's the difference between let and const in JavaScript?"}, {"role": "assistant", "content": "# let vs const in JavaScript\n\nBoth `let` and `const` are block-scoped variable declarations introduced in ES6 (2015). Here's the detailed breakdown:\n\n## let - Reassignable Variables\n\n`let` declares a variable whose value can be changed:\n\n```javascript\nlet counter = 0;\ncounter = counter + 1;  // ✓ Allowed\ncounter = 100;          // ✓ Allowed\n```\n\n**Use when:** The value needs to change (loop counters, accumulating values, conditional assignments)\n\n## const - Constants\n\n`const` declares a constant reference that cannot be reassigned:\n\n```javascript\nconst PI = 3.14159;\nPI = 3.14;  // ✗ TypeError: Assignment to constant variable\n```\n\n**Important caveat:** While the reference can't change, object/array contents can:\n\n```javascript\nconst user = { name: 'John' };\nuser.name = 'Jane';  // ✓ Allowed - modifying property\nuser = {};           // ✗ Error - reassigning entire object\n\nconst numbers = [1, 2, 3];\nnumbers.push(4);     // ✓ Allowed - modifying array\nnumbers = [];        // ✗ Error - reassigning array\n```\n\n## Best Practices\n\n1. **Default to `const`**: Use unless you know the value will change\n2. **Use `let` for reassignment**: Counters, accumulating values, swapping\n3. **Never use `var`**: It has function scope and hoisting issues\n\n## Comparison with var\n\n```javascript\n// var: function-scoped, hoisted\nfunction example() {\n    console.log(x);  // undefined (hoisted)\n    var x = 5;\n}\n\n// let/const: block-scoped, temporal dead zone\nfunction example2() {\n    console.log(y);  // ReferenceError\n    let y = 5;\n}\n```\n\nIn modern JavaScript, `const` and `let` have completely replaced `var`."}]}

These examples demonstrate the variety and quality needed for effective fine-tuning datasets.

Framework Integration & Deployment

Training Framework Compatibility

Unsloth Integration

Unsloth is optimized for efficient fine-tuning with ChatML format.

Setup:

from datasets import load_dataset
from unsloth import FastLanguageModel

# Load your ChatML dataset
dataset = load_dataset('json', data_files='training_data.jsonl')

# Split into train/val
train_val = dataset['train'].train_test_split(test_size=0.1)

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/mistral-7b",
    load_in_4bit=True,
)

# Configure for ChatML
FastLanguageModel.for_training(model)

# Train with your dataset
trainer = SFTTrainer(
    model=model,
    train_dataset=train_val['train'],
    eval_dataset=train_val['test'],
    dataset_text_field="messages",  # Unsloth automatically handles ChatML
    ...
)

Key Features:

• Automatic ChatML format detection
• Efficient token handling
• Built-in data loading for JSONL

Axolotl Integration

Axolotl supports multiple chat template formats including ChatML.

Setup in `config.yaml`:

datasets:
  - path: training_data.jsonl
    type: chat_template
    chat_template: chatml  # Explicitly specify ChatML format
    
model_name_or_path: mistralai/Mistral-7B

chat_template: chatml  # Or auto-detect

training_hyperparameters:
  lr: 2e-4
  num_epochs: 3
  ...

Supported Formats:

• chatml: Standard ChatML (recommended)
• llama2: Llama 2 format
• alpaca: Alpaca format
• Custom templates supported

Hugging Face Transformers

Standard ChatML works with Hugging Face's data loading.

Setup:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments

# Load dataset
dataset = load_dataset('json', data_files='training_data.jsonl')

# Tokenize
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

def preprocess_function(examples):
    # Convert messages to text format
    texts = []
    for messages in examples['messages']:
        text = tokenizer.apply_chat_template(
            messages, 
            tokenize=False, 
            add_generation_prompt=True
        )
        texts.append(text)
    
    tokenized = tokenizer(
        texts, 
        truncation=True, 
        max_length=2048,
        return_tensors="pt"
    )
    return tokenized

tokenized_dataset = dataset.map(preprocess_function, batched=True)

# Train with Trainer
training_args = TrainingArguments(
    output_dir="./fine-tuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    ...
)

trainer = Trainer(
    model=AutoModelForCausalLM.from_pretrained(...),
    args=training_args,
    train_dataset=tokenized_dataset['train'],
    ...
)

trainer.train()

Important Notes:

• Chat template must match model expectations
• Tokenizer may need custom chat template configuration
• Ensure context window matches model limits

Custom Training Loops

For unsupported frameworks:

import json
from torch.utils.data import Dataset, DataLoader

class ChatMLDataset(Dataset):
    def __init__(self, jsonl_file, tokenizer, max_length=2048):
        self.tokenizer = tokenizer
        self.max_length = max_length
        self.examples = []
        
        with open(jsonl_file, 'r') as f:
            for line in f:
                data = json.loads(line)
                self.examples.append(data['messages'])
    
    def __len__(self):
        return len(self.examples)
    
    def __getitem__(self, idx):
        messages = self.examples[idx]
        
        # Convert to text format using your tokenizer's chat template
        text = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=False
        )
        
        # Tokenize
        encodings = self.tokenizer(
            text,
            max_length=self.max_length,
            truncation=True,
            return_tensors="pt"
        )
        
        # Create labels (same as input for language modeling)
        return {
            'input_ids': encodings['input_ids'].squeeze(),
            'attention_mask': encodings['attention_mask'].squeeze(),
            'labels': encodings['input_ids'].squeeze(),
        }

# Usage
dataset = ChatMLDataset('training_data.jsonl', tokenizer)
dataloader = DataLoader(dataset, batch_size=8, shuffle=True)

Deployment Workflow

Pre-Training Checklist

• [ ] Dataset validated with validate_chatml.py (no errors)
• [ ] Analysis run with analyze_dataset.py (good diversity metrics)
• [ ] Token count estimated (within budget)
• [ ] Validation split prepared (if needed)
• [ ] Framework compatibility verified
• [ ] Training parameters configured
• [ ] Resources allocated (GPU/memory)

Training Best Practices

Hyperparameter Tuning:

• Learning rate: Start with 2e-4 for fine-tuning
• Batch size: 8-16 common, depends on GPU memory
• Epochs: Start with 3, monitor for overfitting
• Warmup steps: 10% of total steps recommended
• Max gradient norm: 1.0 typical

Monitoring:

• Track training loss (should decrease)
• Monitor validation loss (watch for overfitting)
• Sample outputs periodically during training
• Save checkpoints at regular intervals
• Log divergence or instability

Common Issues:

Post-Training Evaluation

After training completes:

1. Quantitative Metrics:

• Perplexity on validation set
• Loss improvement from baseline
• Token accuracy (if applicable)

2. Qualitative Evaluation:

• Generate sample outputs from fine-tuned model
• Compare against training examples
• Test on unseen examples similar to training data
• Check for memorization vs. generalization

3. Real-World Testing:

• Test with actual use cases
• Compare against base model
• Get domain expert feedback
• Measure task-specific metrics (if applicable)

Model Optimization

Quantization

Reduce model size for deployment:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "fine-tuned-model",
    quantization_config=bnb_config,
    device_map="auto"
)

Knowledge Distillation

Train smaller model to match fine-tuned model:

from transformers import Trainer

# Use fine-tuned model as teacher
# Train smaller student model on same dataset
# Add distillation loss to training objective

teacher_model = AutoModelForCausalLM.from_pretrained("fine-tuned-model")
student_model = AutoModelForCausalLM.from_pretrained("smaller-model")

# Custom trainer with distillation loss

Deployment Scenarios

Local Deployment

Run fine-tuned model locally:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("./fine-tuned-model")
tokenizer = AutoTokenizer.from_pretrained("./fine-tuned-model")

# Generate with your data
messages = [
    {"role": "system", "content": "You are helpful..."},
    {"role": "user", "content": "How do I..."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(**inputs, max_length=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

API Deployment

Deploy via API service:

Using vLLM (recommended for inference):

python -m vllm.entrypoints.openai.api_server \
    --model ./fine-tuned-model \
    --port 8000

Then query:

import requests

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "fine-tuned-model",
        "messages": [
            {"role": "system", "content": "You are helpful..."},
            {"role": "user", "content": "How do I..."}
        ],
        "temperature": 0.7
    }
)

Cloud Deployment

• HuggingFace Spaces: Simple WebUI hosting
• AWS SageMaker: Managed training and hosting
• Azure ML: Enterprise-grade ML platform
• Google Cloud Vertex AI: Integrated ML platform

Version Control

Save Training Artifacts

fine-tuned-model/
├── pytorch_model.bin        # Model weights
├── config.json              # Model config
├── tokenizer.model          # Tokenizer
├── training_config.yaml     # Training parameters used
├── training_data.jsonl      # Dataset used
├── validation_data.jsonl    # Validation set
├── dataset_info.txt         # Dataset metadata
├── training_logs/           # Training loss logs
│   └── events.out.tfevents
└── generation_notes.md      # How dataset was created

Reproducibility

Document what you did:

generation_notes.md:

# Dataset Generation Notes

## Parameters Used
- Task Type: Customer Support
- Domain: Technical Support
- Total Examples: 500
- Validation Split: 10%

## Strategy
1. Generated 50 base examples covering core scenarios
2. Expanded with complexity variations (100 examples)
3. Added edge cases and error scenarios (150 examples)
4. Final polish and validation (200 examples)

## Quality Metrics
- Diversity score: 78%
- Multi-turn ratio: 32%
- Total tokens: ~125,000

## Changes Made
- Removed 5 duplicate examples
- Fixed 2 JSON formatting issues
- Added 10 missing error scenario examples

## Model Performance
- Training loss: 2.1 → 0.8
- Validation loss: 2.3 → 0.95
- No sign of overfitting after 3 epochs

Data Generation Techniques

Generation Methods

Diverse Scenario Generation

Create variety by generating across multiple dimensions:

1. Query Phrasing Variations

• Questions: Direct questions, rhetorical questions
• Commands: Imperative statements, requests
• Statements: Declarative with embedded request
• Mixed: Combine multiple forms in longer queries

Example variations for the same task:

Q: How do I reverse a string in Python?
C: Write code to reverse a string in Python
S: I'm working with strings and need to reverse them. What's the best approach?

2. User Expertise Levels

• Beginner: Simple concepts, basic syntax, no assumed knowledge
• Intermediate: Familiar with fundamentals, wants efficiency
• Expert: Looking for advanced techniques, edge cases, performance

Example variations:

Beginner: What is a list in Python?
Intermediate: What's the difference between list and tuple?
Expert: When should I use tuple unpacking vs. destructuring, and what are the performance implications?

3. Context Variations

• Minimal Context: Just the core request
• Rich Context: Include constraints, parameters, examples
• Error Context: Include what failed and why

Example variations:

Minimal: Sort a list of numbers
Rich: Sort a list of 1M numbers in Python with O(n log n) complexity, minimizing memory use
Error: I tried sorting with .sort() but got a TypeError. My list has mixed types.

4. Response Complexity Levels

• Brief: 1-2 sentences, direct answer
• Standard: Full explanation with example
• Detailed: Comprehensive with multiple approaches, trade-offs, best practices

Example variations:

Brief: Use sorted() to sort a list.

Standard: Use sorted() or .sort() method:
- sorted() returns new list
- .sort() modifies in place

Detailed: Python offers three sorting approaches:
1. sorted() - creates new list, good for immutable results
2. .sort() - modifies in place, memory efficient
3. Custom key - for complex sorting criteria
[includes examples and performance notes]

5. Edge Cases & Error Scenarios

• Happy Path: Normal, expected usage
• Boundary Cases: Empty input, single item, maximum size
• Error Cases: Invalid input, conflicting parameters, resource limits
• Special Cases: Null values, special characters, type mismatches

Example edge cases:

Happy: Sort [3, 1, 2] → [1, 2, 3]
Boundary: Sort [] → []
Error: Sort "not a list" → TypeError
Special: Sort [None, 1, 2] → Type error or special handling

Domain-Specific Generation

Tailor examples to your specific domain:

Technical/Programming Domains

• Language-specific patterns: Python syntax differs from JavaScript
• Library/framework concepts: React hooks vs. class components
• Performance considerations: Algorithm complexity, memory usage
• Error patterns: Common mistakes in the domain

Business/Support Domains

• Customer scenarios: Common support requests
• Resolution workflows: Typical solution paths
• Tone matching: Professional, empathetic, solution-oriented
• Policy constraints: Refund limits, warranty coverage

Academic/Educational Domains

• Explanation depth: Balance rigor with accessibility
• Prerequisite knowledge: What should be assumed?
• Worked examples: Step-by-step walkthroughs
• Assessment questions: Test understanding

Systematic Variation Template

For comprehensive coverage, use this template:

For each core scenario:
  For each user expertise level:
    For each query phrasing type:
      For each response complexity level:
        Generate 1 example

        Total: 1 scenario × 3 levels × 3 phrasings × 3 complexities = 27 variations

Benefits:

• Ensures systematic coverage
• Reduces gaps and redundancy
• Maintains consistency across examples
• Scales efficiently for multiple scenarios

Format-Specific Guidance

Code Generation Examples

Key Elements:

• Clean, well-commented code
• Proper error handling
• Performance considerations
• Best practices for the language/framework
• Documentation strings

Diversity:

• Different programming paradigms (OOP, functional, procedural)
• Various complexity levels (simple functions to complex systems)
• Multiple approaches to same problem
• Both complete programs and code snippets

Creative Writing Examples

Key Elements:

• Vivid, sensory-rich descriptions
• Compelling narrative voice
• Character development
• Dialogue that feels natural
• Consistent tone and style

Diversity:

• Different genres and subgenres
• Various narrative perspectives (first, third person)
• Pacing variations (fast-paced to contemplative)
• Multiple writing styles (minimalist to ornate)

Data Analysis Examples

Key Elements:

• Clear problem definition
• Step-by-step analytical approach
• Relevant visualizations or summaries
• Actionable insights
• Code for reproducibility

Diversity:

• Different data types (structured, time-series, text)
• Various analytical methods (descriptive, predictive, exploratory)
• Multiple business contexts
• Different data scales and complexities

Technical Documentation Examples

Key Elements:

• Clear structure (overview, details, examples)
• Precise technical terminology
• Practical examples
• Clear parameter descriptions
• Error conditions and handling

Diversity:

• API documentation
• User guides and tutorials
• Architecture and design docs
• Troubleshooting guides
• Reference material

Multi-Turn Conversation Generation

Structure Patterns

Support Conversation Pattern:

1. User presents problem
2. Assistant asks clarifying questions OR provides initial solution
3. User provides more detail OR accepts solution
4. Assistant refines response
5. Conversation reaches resolution

Debugging Conversation Pattern:

1. User describes error
2. Assistant asks for context (error message, code)
3. User provides information
4. Assistant proposes solution
5. User tests, reports result
6. Assistant refines or confirms success

Learning Conversation Pattern:

1. User asks conceptual question
2. Assistant explains with analogies
3. User asks follow-up about specific aspect
4. Assistant provides detailed example
5. User asks how to apply knowledge
6. Assistant gives practical application example

Conversation Quality Checklist

• [ ] Each turn feels natural and conversational
• [ ] Questions are specific and show understanding
• [ ] Responses build on previous context
• [ ] Complexity escalates appropriately
• [ ] Dialogue feels like real interaction, not templated
• [ ] Both parties demonstrate active listening
• [ ] Resolution or conclusion is clear

Batch Generation Workflow

Batch 1: Foundation Examples (5-10 examples)

• Core scenarios without complexity
• Establish baseline for quality
• Define system prompts and tone
• Get user feedback

Batch 2: Complexity Expansion (20-30 examples)

• Add edge cases and error scenarios
• Vary query formulations
• Include advanced use cases
• Target specific gaps identified in Batch 1

Batch 3: Coverage Filling (30-50 examples)

• Fill identified gaps in coverage
• Ensure good distribution across categories
• Add contextual variations
• Validate diversity metrics

Batch 4: Polish & Validation (10-20 examples)

• Fill any remaining gaps
• Ensure quality consistency
• Validate edge case coverage
• Run full validation suite

Avoiding Common Generation Pitfalls

Problem: Over-Templating

Bad Approach:

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, 
  {"role": "user", "content": "[TEMPLATE: I want to {VERB} {NOUN}"}, 
  {"role": "assistant", "content": "Here's how to {VERB} {NOUN}: ..."}]}

Better Approach:

• Write examples naturally without templates
• Vary phrasing and structure genuinely
• Let different user intents drive different response structures

Problem: Narrow Scenario Coverage

Bad Approach:

• All examples are basic use cases
• No error scenarios or edge cases
• Limited domain diversity

Better Approach:

• Include 20-30% edge cases
• Cover error conditions
• Vary domains and contexts
• Test with real-world variations

Problem: Inconsistent Quality

Bad Approach:

• Early examples are high quality, later ones degrade
• Inconsistent accuracy across examples
• Varying response depths without good reason

Better Approach:

• Maintain consistent quality standards throughout
• Use a checklist for each batch
• Review for consistency before finalizing
• Keep quality bar visible

Problem: Too Much Repetition

Bad Approach:

• Many examples say slightly different versions of same thing
• Similar queries with nearly identical responses
• Redundant information

Better Approach:

• Ensure each example teaches something new
• Vary not just the query but the concept/approach
• Check for duplicates after generation
• Use analyze_dataset.py to check diversity metrics

Quality Validation & Analysis

Validation Workflow

Pre-Delivery Validation

Run these checks before delivering a dataset:

1. JSON Validation

python scripts/validate_chatml.py training_data.jsonl

Checks:

• Valid JSON on each line
• Required fields present (messages, role, content)
• Valid role values (system, user, assistant)
• Proper message structure
• No empty content fields

2. Dataset Analysis

python scripts/analyze_dataset.py training_data.jsonl

Provides:

• Total example count
• Message length statistics
• System prompt variations
• User query patterns
• Assistant response patterns
• Quality metrics and diversity scores

3. Manual Quality Review

Sample Review (recommended for all datasets): 1. Read first 5 examples completely 2. Check middle batch (examples 25-35) 3. Review last 5 examples 4. Verify diversity in each batch

Look for:

• Natural language (not robotic)
• Accurate and helpful responses
• Proper JSON formatting
• Realistic user queries
• Consistent quality level

4. Duplicate Detection

Simple duplicate check:

# Count unique user messages
grep -o '"role": "user", "content": "[^"]*"' training_data.jsonl | sort | uniq | wc -l

Near-duplicate check: Run validation script and review warnings about similar message lengths

Common Validation Issues

Issue: Invalid JSON

Error Pattern:

Line 42: Invalid JSON - Expecting property name enclosed in double quotes

Root Causes:

• Unescaped quotes in content: "She said "hello"" should be "She said \"hello\""
• Literal newlines in JSON value (should use \n)
• Missing commas between fields
• Trailing commas

Fix:

• Ensure all quotes inside content are escaped with \
• Use \n for newlines, not literal line breaks
• Validate JSON before outputting

Issue: Missing Required Fields

Error Pattern:

Line 15: Missing 'content' field in message

Root Causes:

• Incomplete message structure
• Null/undefined values in JSON
• Formatting error during generation

Fix:

• Always include: role, content
• Verify no null values
• Test JSON generation code

Issue: Invalid Role Values

Error Pattern:

Line 8, Message 2: Invalid role 'Assistant' (should be lowercase 'assistant')

Root Causes:

• Case sensitivity error (role must be lowercase)
• Typo in role name

Fix:

• Ensure roles are exactly: "system", "user", "assistant"
• Check case sensitivity in generation code

Issue: Empty Content

Warning Pattern:

Line 23: Empty content string

Root Causes:

• Failed to generate content for a message
• Content accidentally set to empty string

Fix:

• Add validation to prevent empty content
• Regenerate affected examples

Diversity Assessment

Checking Diversity Metrics

Use analysis output to assess diversity:

User Message Patterns:
  questions: 450 (90%)
  commands: 480 (96%)
  code_related: 320 (64%)
  short_queries: 150 (30%)
  medium_queries: 280 (56%)
  long_queries: 70 (14%)

Good Diversity Signs:

• Multiple pattern types represented
• Reasonable distribution (not all same type)
• Length variety across messages
• Different content types

Poor Diversity Signs:

• 95%+ of examples are one type
• All queries similar length
• Repetitive patterns
• Limited variation in response types

Quality Indicators from Analysis

Diversity Score:

• Higher % of unique message lengths = better diversity
• Target: 70%+ unique lengths across examples

Balance Score:

• Ratio of single-turn to multi-turn
• Well-balanced dataset should be 60-80% single-turn, 20-40% multi-turn
• Depending on task type

Response Pattern Diversity:

Assistant Response Patterns:
  with_0_code_blocks: 200 (40%)
  with_1_code_blocks: 200 (40%)
  with_2+_code_blocks: 100 (20%)
  with_lists: 280 (56%)
  with_headers: 320 (64%)
  brief: 50 (10%)
  medium: 250 (50%)
  detailed: 200 (40%)

Good Coverage:

• Multiple response types represented
• Balanced distribution of lengths
• Variety in formatting (code, lists, headers)

Export & Documentation

Export Dataset Statistics

Generate exportable statistics report:

python scripts/analyze_dataset.py training_data.jsonl --export stats.json

Contents:

• Total examples and message counts
• Conversation length statistics
• System prompt variations
• User pattern distribution
• Response type distribution
• Token estimates

Create Dataset Documentation

dataset_info.txt should include:

FINE-TUNING DATASET: Customer Support Training Data
===================================================

Dataset Purpose:
  Training customer support chatbot for technical support domain

Generated: 2024-01-15
Total Examples: 500
  - Training: 450 examples
  - Validation: 50 examples

Quality Assurance:
  ✓ JSON validation: PASSED
  ✓ No duplicates found
  ✓ Diversity check: PASSED
  ✓ All required fields present
  ✓ Format specification compliance: 100%

Dataset Composition:
  Single-turn conversations: 350 (70%)
  Multi-turn conversations: 100 (30%)
  
  Response types:
    With code examples: 0 (0%)
    With structured lists: 180 (36%)
    With headers/sections: 280 (56%)
  
  Response lengths:
    Brief (< 200 chars): 50 (10%)
    Medium (200-800 chars): 250 (50%)
    Detailed (> 800 chars): 200 (40%)

System Prompts:
  Unique variations: 2
  1. "You are a helpful customer support agent..."
  2. "You are a technical support specialist..."

Category Distribution:
  Account/Login Issues: 100 (20%)
  Technical Issues: 180 (36%)
  Billing/Subscription: 120 (24%)
  General Questions: 100 (20%)

Estimated Tokens:
  Total dataset: ~125,000 tokens
  Average per example: ~250 tokens
  Estimated training cost (at $3/1M tokens): $0.38

Instructions for Use:
1. Load with: datasets.load_dataset('json', data_files='training_data.jsonl')
2. Configure training framework for ChatML format
3. Consider token limits for your model
4. Monitor for overfitting with validation split
5. Test sample outputs during training

Generation Notes:
  - Examples crafted to reflect real customer support interactions
  - Includes common edge cases and error scenarios
  - Balanced between technical and non-technical customers
  - Multi-turn examples show conversation flow

Performance Metrics

Dataset Quality Checklist

Before declaring dataset complete:

• [ ] Completeness: All identified scenarios covered
• [ ] Accuracy: All factual claims verified as correct
• [ ] Consistency: Quality level uniform across examples
• [ ] Diversity: Pattern variety measured and adequate
• [ ] Format: All JSON valid, all required fields present
• [ ] No Duplicates: Verified unique examples
• [ ] Realistic: Examples sound like real user interactions
• [ ] Documented: Dataset_info.txt created with statistics
• [ ] Reproducible: Notes on generation process kept
• [ ] Size Appropriate: Meets original requirements

Success Criteria

A dataset is ready when:

1. Validation passes with no errors, minimal warnings 2. Analysis shows good diversity metrics 3. Manual review finds no quality issues in sample 4. Token estimate aligns with available budget 5. Category coverage matches requirements 6. All stakeholders approve quality level

Troubleshooting

Dataset Too Small

Problem: Only 250 examples generated, need 500

Solutions: 1. Continue generation for additional 250 examples 2. Use batch generation workflow to add more systematically 3. Vary generation parameters to create more examples

Dataset Quality Declining

Problem: First 100 examples excellent, later ones mediocre

Symptoms:

• Analysis shows degrading diversity
• Manual review finds repetition
• Later responses less helpful/accurate

Solutions: 1. Return to strategy phase - identify what changed 2. Review generation approach for issues 3. Regenerate declining batches with original quality standards 4. Consider fatigue - take breaks between batches

High Duplication Rate

Problem: Validation shows many very similar examples

Symptoms:

• Validation warnings about similar message lengths
• Manual review finds paraphrased variations of same content
• Diversity score low

Solutions: 1. Identify duplicate patterns 2. Remove near-duplicates 3. Regenerate with focus on true variation 4. Use different generation approach (more creative phrasing)

Poor Domain Coverage

Problem: Most examples about Topic A, few about Topic B

Symptoms:

• Category distribution uneven
• Many examples redundant in topic A
• Topic B under-represented

Solutions: 1. Analyze current distribution with scripts 2. Generate targeted batch focused on Topic B 3. Manually specify category quotas for additional examples 4. Rebalance final dataset

Fine-Tuning Data Generation Plan

Project Overview

• Task Type: [e.g., customer support, code generation, Q&A]
• Domain: [e.g., e-commerce, healthcare, finance]
• Total Examples: [number]
• Training/Validation Split: [e.g., 80/20]

Dataset Composition

Categories and Distribution

Complexity Breakdown

• Simple/Basic: [#] examples ([%])
• Description: [What qualifies as simple]
• Medium: [#] examples ([%])
• Description: [What qualifies as medium]
• Complex/Advanced: [#] examples ([%])
• Description: [What qualifies as complex]

Content Strategy

System Prompt Approach

• [ ] Single consistent system prompt across all examples
• [ ] Multiple system prompts (specify variations below)

System Prompt(s):

[Insert system prompt(s) here]

User Query Characteristics

• Phrasing Variations: [How queries will be varied]
• Formality Levels: [Formal, casual, technical, etc.]
• Query Types: [Questions, commands, statements, etc.]
• Length Range: [Short: X tokens, Long: Y tokens]

Assistant Response Characteristics

• Tone: [Professional, friendly, technical, etc.]
• Style: [Concise, detailed, step-by-step, etc.]
• Average Length: [X-Y tokens]
• Special Formatting: [Code blocks, lists, tables, JSON, etc.]

Diversity Requirements

Scenario Coverage

• [ ] Common/expected scenarios
• [ ] Edge cases
• [ ] Error handling situations
• [ ] Multi-step interactions
• [ ] Ambiguous queries
• [ ] Follow-up questions

Variation Strategies

1. Phrasing: [Strategy for varying how users ask things] 2. Context: [Different scenarios/settings for same topic] 3. Expertise Levels: [Beginner, intermediate, expert users] 4. Response Depth: [When to be brief vs. detailed]

Quality Standards

Required Qualities

• [ ] Factually accurate responses
• [ ] Natural, realistic user queries
• [ ] Appropriate response length for query
• [ ] Consistent formatting
• [ ] Proper grammar and spelling
• [ ] No repetitive patterns
• [ ] Diverse vocabulary

Validation Criteria

• [ ] JSON validity
• [ ] No duplicate examples
• [ ] All required fields present
• [ ] System/user/assistant roles correct
• [ ] Content appropriate for domain
• [ ] Responses actually address queries

Example Breakdown by Category

Category 1: [Name]

Sample Topics:

• [Topic 1]
• [Topic 2]
• [Topic 3]

Example Query Types:

• [Example query type 1]
• [Example query type 2]

Expected Response Pattern:

• [Description of how responses should look]

---

Category 2: [Name]

Sample Topics:

• [Topic 1]
• [Topic 2]
• [Topic 3]

Example Query Types:

• [Example query type 1]
• [Example query type 2]

Expected Response Pattern:

• [Description of how responses should look]

---

Edge Cases and Special Scenarios

Edge Cases to Include ([#] examples)

1. [Edge case 1]: [Description] 2. [Edge case 2]: [Description] 3. [Edge case 3]: [Description]

Error Scenarios ([#] examples)

1. [Error scenario 1]: [How to handle] 2. [Error scenario 2]: [How to handle]

Multi-Turn Conversations ([#] examples)

• [Description of conversation flow types to include]

Generation Approach

Batch Strategy

• Batch Size: [#] examples per batch
• Batches: [#] total batches
• Review Points: After each batch

Iteration Plan

1. Generate first batch ([#] examples) 2. Review for quality and diversity 3. Adjust approach based on findings 4. Generate subsequent batches 5. Final validation and deduplication

Output Files

• training_data.jsonl - [#] examples
• validation_data.jsonl - [#] examples (optional)
• dataset_info.txt - Metadata and statistics
• generation_notes.md - Process notes and decisions (optional)

Success Metrics

• [ ] Target number of examples reached
• [ ] No duplicate examples
• [ ] Distribution matches plan
• [ ] All quality criteria met
• [ ] JSON validation passes
• [ ] Diversity check passes

Timeline

• Planning: [Completed]
• First Batch: [Status]
• Subsequent Batches: [Status]
• Review & Refinement: [Status]
• Final Validation: [Status]
• Delivery: [Status]

Notes and Considerations

[Any additional notes, special requirements, or considerations for this dataset]

---

Plan Status: [ ] Draft [ ] Approved [ ] In Progress [ ] Complete Last Updated: [Date]