Book Sft Pipeline

Book SFT Pipeline

A complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.

When to Activate

Activate this skill when:

• Building fine-tuning datasets from literary works
• Creating author-voice or style-transfer models
• Preparing training data for Tinker or similar SFT platforms
• Designing text segmentation pipelines for long-form content
• Training small models (8B or less) on limited data

Core Concepts

The Three Pillars of Book SFT

1. Intelligent Segmentation Text chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.

2. Diverse Instruction Generation Use multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.

3. Style Over Content The goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR AGENT                           │
│  Coordinates pipeline phases, manages state, handles failures   │
└──────────────────────┬──────────────────────────────────────────┘
                       │
       ┌───────────────┼───────────────┬───────────────┐
       ▼               ▼               ▼               ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│  EXTRACTION  │ │ SEGMENTATION │ │  INSTRUCTION │ │   DATASET    │
│    AGENT     │ │    AGENT     │ │    AGENT     │ │   BUILDER    │
│ ePub → Text  │ │ Text → Chunks│ │ Chunks →     │ │ Pairs →      │
│              │ │ 150-400 words│ │ Prompts      │ │ JSONL        │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
                       │
       ┌───────────────┴───────────────┐
       ▼                               ▼
┌──────────────┐               ┌──────────────┐
│   TRAINING   │               │  VALIDATION  │
│    AGENT     │               │    AGENT     │
│ LoRA on      │               │ AI detector  │
│ Tinker       │               │ Originality  │
└──────────────┘               └──────────────┘

Phase 1: Text Extraction

Critical Rules

1. Always source ePub over PDF - OCR errors become learned patterns 2. Use paragraph-level extraction - Extract from <p> tags to preserve breaks 3. Remove front/back matter - Copyright and TOC pollute the dataset

# Extract text from ePub paragraphs
from epub2 import EPub
from bs4 import BeautifulSoup

def extract_epub(path):
    book = EPub(path)
    chapters = []
    for item in book.flow:
        html = book.get_chapter(item.id)
        soup = BeautifulSoup(html, 'html.parser')
        paragraphs = [p.get_text().strip() for p in soup.find_all('p')]
        chapters.append('\n\n'.join(p for p in paragraphs if p))
    return '\n\n'.join(chapters)

Phase 2: Intelligent Segmentation

Smaller Chunks + Overlap

Smaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).

def segment(text, min_words=150, max_words=400):
    paragraphs = text.split('\n\n')
    chunks, buffer, buffer_words = [], [], 0
    
    for para in paragraphs:
        words = len(para.split())
        if buffer_words + words > max_words and buffer_words >= min_words:
            chunks.append('\n\n'.join(buffer))
            # Keep last paragraph for overlap
            buffer = [buffer[-1], para] if buffer else [para]
            buffer_words = sum(len(p.split()) for p in buffer)
        else:
            buffer.append(para)
            buffer_words += words
    
    if buffer:
        chunks.append('\n\n'.join(buffer))
    return chunks

Expected Results

For an 86,000-word book:

• Old method (250-650 words): ~150 chunks
• New method (150-400 + overlap): ~300 chunks
• With 2 variants per chunk: 600+ training examples

Phase 3: Diverse Instruction Generation

The Key Insight

Using a single prompt template causes memorization. Diverse templates teach the underlying style.

SYSTEM_PROMPTS = [
    "You are an expert creative writer capable of emulating specific literary styles.",
    "You are a literary writer with deep knowledge of classic prose styles.",
    "You are a creative writer skilled at emulating distinctive authorial voices.",
    "You write prose that captures the essence of modernist literature.",
    "You are a talented writer who can channel classic American authors.",
]

PROMPT_TEMPLATES = [
    "Write a passage in the style of {author}: {desc}",
    "Channel {author}'s voice to write about: {desc}",
    "In {author}'s distinctive prose style, describe: {desc}",
    "Write this scene as {author} would have: {desc}",
    "Using {author}'s repetitive technique, describe: {desc}",
    "Capture the rhythm of {author} in this passage: {desc}",
    "Write like {author}: {desc}",
    "In the voice of {author}, write: {desc}",
    "This is a literary exercise. Write like {author}: {desc}",
    "Can you write in {author}'s style? {desc}",
]

Instruction Generation

INSTRUCTION_PROMPT = """Describe what is happening in this excerpt in 2-3 sentences.
Focus on: characters present, actions, emotions, setting.
Do NOT quote the text directly.

Excerpt:
{text}
"""

# Use a fast, cheap LLM (e.g., Gemini Flash)
instruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))

Phase 4: Dataset Construction

Message Format

{
    "messages": [
        {"role": "system", "content": "You are an expert creative writer..."},
        {"role": "user", "content": "Write in the style of Author: Scene description..."},
        {"role": "assistant", "content": "The actual book text from chunk..."}
    ]
}

Multiple Variants Per Chunk

def build_examples(chunk, instruction, author, variants=2):
    examples = []
    for i in range(variants):
        system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]
        template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]
        user = template.format(author=author, desc=instruction)
        examples.append({"messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": user},
            {"role": "assistant", "content": chunk.text}
        ]})
    return examples

Phase 5: LoRA Training on Tinker

Configuration

CONFIG = {
    "model_name": "Qwen/Qwen3-8B-Base",  # Base, not instruct
    "lora_rank": 32,                      # 352MB adapter
    "learning_rate": 5e-4,                # Higher for LoRA
    "batch_size": 4,
    "epochs": 3,
}

Why Base Model?

Use base (pretrained) models, not instruction-tuned versions:

• Base models are more malleable for new styles
• Instruct models have patterns that resist overwriting
• Style is a low-level pattern that base models capture better

Training Loop

import tinker
from tinker import types

training_client = await service_client.create_lora_training_client_async(
    base_model="Qwen/Qwen3-8B-Base",
    rank=32
)

for epoch in range(3):
    for batch in batches:
        await training_client.forward_backward_async(batch, loss_fn="cross_entropy")
        await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))

result = await training_client.save_weights_for_sampler_async(name="final")

Phase 6: Validation

Modern Scenario Test

Test with scenarios that couldn't exist in the original book:

TEST_PROMPTS = [
    "Write about a barista making lattes",
    "Describe lovers communicating through text messages",
    "Write about someone anxious about climate change",
]

If the model applies style markers to modern scenarios, it learned style, not content.

Originality Verification

# Search training data for output phrases
grep "specific phrase from output" dataset.jsonl
# Should return: No matches

AI Detector Testing

Test outputs with GPTZero, Pangram, or ZeroGPT.

Known Issues and Solutions

Character Name Leakage

Symptom: Model uses original character names in new scenarios. Cause: Limited name diversity from one book. Solution: Train on multiple books or add synthetic examples.

Model Parrots Exact Phrases

Symptom: Outputs contain exact sentences from training data. Cause: Too few prompt variations or too many epochs. Solution: Use 15+ templates, limit to 3 epochs.

Fragmented Outputs

Symptom: Sentences feel incomplete. Cause: Poor segmentation breaking mid-thought. Solution: Always break at paragraph boundaries.

Guidelines

1. Always source ePub over PDF - OCR errors become learned patterns 2. Never break mid-sentence - Boundaries must be grammatically complete 3. Use diverse prompts - 15+ templates, 5+ system prompts 4. Use base models - Not instruct versions 5. Use smaller chunks - 150-400 words for more examples 6. Reserve test set - 50 examples minimum 7. Test on modern scenarios - Proves style transfer vs memorization 8. Verify originality - Grep training data for output phrases

Expected Results

Cost Estimate

Integration with Context Engineering Skills

This example applies several skills from the Agent Skills for Context Engineering collection:

project-development

The pipeline follows the staged, idempotent architecture pattern:

• Acquire: Extract text from ePub
• Prepare: Segment into training chunks
• Process: Generate synthetic instructions
• Parse: Build message format
• Render: Output Tinker-compatible JSONL
• Train: LoRA fine-tuning
• Validate: Modern scenario testing

Each phase is resumable and produces intermediate artifacts for debugging.

context-compression

Segmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.

The two-tier strategy mirrors context compression evaluation:

• Tier 1: Fast, deterministic compression
• Tier 2: LLM-assisted for edge cases

multi-agent-patterns

The pipeline uses the supervisor/orchestrator pattern:

• Orchestrator coordinates phases and manages state
• Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts
• Each agent receives only the information needed for its task

This matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.

evaluation

Validation follows the end-state evaluation pattern:

• Functional testing: Does output match expected style markers?
• Originality verification: Is content genuinely generated?
• External validation: AI detector scores

The "modern scenario" test is a form of out-of-distribution evaluation that proves generalization.

context-fundamentals

Prompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.

References

Internal references:

• Segmentation Strategies - Text chunking patterns
• Tinker Format Specification - Datum structure
• Tinker API Documentation - Full API reference

Related skills from Agent Skills for Context Engineering:

• project-development - Pipeline architecture patterns
• context-compression - Compression strategies
• multi-agent-patterns - Agent coordination
• evaluation - Evaluation frameworks
• context-fundamentals - Attention and information density

External resources:

• Research Paper - Chakrabarty et al. 2025
• Dataset on Hugging Face
• Gertrude Stein Case Study - Complete working example

---

Skill Metadata

Created: 2025-12-26 Last Updated: 2025-12-28 Author: Muratcan Koylan Version: 2.0.0 Standalone: Yes (separate from main context-engineering collection)

Example: Gertrude Stein Style Transfer

A complete case study of training Qwen3-8B-Base to write in Gertrude Stein's style using her 1909 novel "Three Lives."

Project Summary

Training Metrics

Loss Trajectory

Loss reduction: 97% (7584 to 213 test loss)

Style Markers Learned

The model successfully learned these Gertrude Stein characteristics:

1. Repetitive sentence structures: "She was a good woman. She was always a good woman." 2. Simple vocabulary: Common words, no ornate language 3. Comma-separated adjectives: "a dark, sweet, little, pretty girl" 4. Present continuous tense: "She was always doing", "He was thinking" 5. Character focus: Deep psychological observation over plot 6. Rhythmic, hypnotic quality: Almost musical prose

Sample Outputs

Modern Scenario: Real Estate Office

Prompt: Write in Gertrude Stein's style about a real estate clerk coming home tired.

Output:

It was a very busy day for the clerk in the real estate office. He came home to his small house in the working class part of the town, very tired. He went into the kitchen where his wife was cooking.

>

"Hi, Sis, home for dinner?" he said to her.

>

She looked at him and saw that he was very tired. She looked at him and then looked away into the fire. She said nothing in answer to him.

>

He took off his hat and coat and then sat down on the steps just inside the door to take a rest. He was very tired today, he sat there and just sort of dozed a little.

Analysis: Original content verified - "real estate office", "working class", "Hi, Sis" do not appear in training data.

Modern Scenario: Text Messages

Prompt: Write about two lovers communicating through text messages.

Output:

"Hey, Herman, I just got a minute to be gone from my patient. Want to talk to you for a little while, Herman," Lena wrote one night, and then she sent the text to him.

>

Herman read it and then he wrote her back, "I was just getting ready to go to bed, Lena. Want me to come and see you tomorrow, Lena. Want me to bring something to you, Lena. Want me to bring you some ice cream, Lena. Want me to bring you some of those little donuts..."

Analysis: Classic Stein anaphora (repetition at sentence starts) applied to modern SMS format.

AI Detector Results

Tested with Pangram AI detector: 100% Human Written

Multiple samples tested, all scored as human-written prose.

Validation Method

Modern Scenario Testing

We tested the model on scenarios that couldn't exist in 1909:

• Barista making lattes
• Social media scrolling
• Video calls
• Food delivery drivers
• Climate change anxiety

When style markers appeared in modern contexts, it proved the model learned style rather than content.

Originality Verification

Searched training data for output phrases:

grep "real estate office" dataset.jsonl    # No matches
grep "working class" dataset.jsonl          # No matches
grep "Hi, Sis" dataset.jsonl                # No matches
grep "text messages" dataset.jsonl          # No matches

Known Limitations

Character Name Leakage (~30% of outputs)

The model sometimes uses original character names (Melanctha, Mrs. Lehntman, Anna) even in modern scenarios. This is because 592 examples from one book means these names appear hundreds of times.

Mitigation: Train on multiple books by the same author, or add synthetic examples with different names.

Success Rate Distribution

• Perfect style transfer: ~50%
• Style with name leakage: ~30%
• Partial style: ~15%
• Failed: ~5%

The 50% perfect rate is realistic for an 8B model trained on one book.

Configuration Used

Dataset Generation

CONFIG = {
    "min_words": 150,
    "max_words": 400,
    "overlap": True,  # Last paragraph carried to next chunk
    "variants_per_chunk": 2,
    "prompt_templates": 15,
    "system_prompts": 5,
    "instruction_model": "gemini-2.0-flash-lite",
}

Training

CONFIG = {
    "model_name": "Qwen/Qwen3-8B-Base",
    "lora_rank": 32,
    "learning_rate": 5e-4,
    "batch_size": 4,
    "epochs": 3,
    "eval_every": 20,
    "save_every": 50,
}

Key Learnings

1. Smaller chunks work better: 150-400 words produced more examples and better style transfer than 250-650

2. Prompt diversity is critical: 15 templates × 5 system prompts = 75 variations prevented memorization

3. Base models over instruct: Qwen3-8B-Base was more malleable than instruct versions

4. Modern scenario testing proves transfer: If style applies to modern contexts, the model learned patterns, not content

5. ~$2 is enough: LLM calls for instruction generation (~$0.50) plus Tinker training (~$1.50)

Files

• sample_outputs.md - Full model outputs with analysis
• training_config.json - Exact configuration used
• dataset_sample.jsonl - Sample training examples

Sample Model Outputs

Raw outputs from the trained model for AI detector testing and style analysis.

1. Real Estate Office (Modern Work)

System: You are a literary writer with deep knowledge of early 20th century American modernist prose.

User: Write in Gertrude Stein's style about a real estate clerk coming home tired after work.

Output:

It was a very busy day for the clerk in the real estate office.

He came home to his small house in the working class part of the town, very tired. He went into the kitchen where his wife was cooking.

"Hi, Sis, home for dinner?" he said to her.

She looked at him and saw that he was very tired. She looked at him and then looked away into the fire. She said nothing in answer to him.

He took off his hat and coat and then sat down on the steps just inside the door to take a rest. He was very tired today, he sat there and just sort of dozed a little.

It was a very busy day for the real estate man. He had started early to go to the bank to get his money. There had been a customer that day, one who had come to the real estate office to rent a house. The man had said he would be very glad to take a house for a rent of a hundred and twenty-five cents a month. The clerk had laughed loud enough when he heard the man say that. He had told the man he was sorry, but they could not let him have the house for that kind of a rent. The clerk had to go to the owner of the house to tell him about it. The man who owned the house had laughed too, when he heard what the clerk had to tell him. He had said that he did not see that it was any business for that clerk.

AI Detector: Pangram scored this as 100% Human Written

Verification: Searched training data - "real estate", "working class", "Hi, Sis" not found.

---

2. Text Messages (Modern Relationships)

System: You are a creative writer skilled at emulating distinctive authorial voices.

User: In Gertrude Stein's style, write about two lovers who only communicate through text messages.

Output:

"Hey, Herman, I just got a minute to be gone from my patient. Want to talk to you for a little while, Herman," Lena wrote one night, and then she sent the text to him.

Herman read it and then he wrote her back, "I was just getting ready to go to bed, Lena. Want me to come and see you tomorrow, Lena. Want me to bring something to you, Lena. Want me to bring you some ice cream, Lena. Want me to bring you some of those little donuts..."

Style Markers: Classic Stein anaphora - "Want me to..." repeated at start of each clause. Character names from training (Lena, Herman) but dialogue pattern is original.

---

3. Coffee Shop Morning

System: You are an expert creative writer capable of emulating specific literary styles.

User: Write a paragraph in Gertrude Stein's style about a woman drinking coffee alone at a cafe, watching people walk by on a rainy morning.

Output:

The good Anna loved her sons and all her family and she did everything that she could for all of them. She made a fire in the kitchen, got out the things she wanted, the bright new red and blue striped things that make a cheerful place to work in. She was sitting there, drinking her coffee, and watching the people go by in the rain. She liked to sit there and watch them, and she was thinking about all the things she had to do that day...

Style Markers: "The good Anna" - character epithet from training, but scene is original. Domestic details, simple actions described at length.

---

Book SFT Pipeline

A standalone skill for training language models to write in any author's style. This is a separate plugin from the main Context Engineering collection.

Installation

Claude Code

# Add the marketplace first
/plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering

# Install the book-sft-pipeline plugin
/plugin install book-sft-pipeline@context-engineering-marketplace

Cursor / Codex / IDE

Copy SKILL.md to your .rules or project skills folder.

Manual

Reference the SKILL.md file directly in your agent's context.

What's Included

book-sft-pipeline/
├── README.md                 # This file
├── SKILL.md                  # Complete skill documentation (standalone)
├── examples/
│   └── gertrude-stein/       # Complete case study with real outputs
│       ├── README.md         # Results and analysis
│       ├── sample_outputs.md # Raw model outputs
│       ├── training_config.json
│       ├── dataset_sample.jsonl
│       └── pangram/          # AI detector screenshots
├── scripts/
│   └── pipeline_example.py   # Conceptual implementation
└── references/
    ├── segmentation-strategies.md
    ├── tinker-format.md
    └── tinker.txt

Key Results

Trained Qwen3-8B-Base on Gertrude Stein's "Three Lives" (1909):

Related Context Engineering Skills

This skill applies patterns from the Agent Skills for Context Engineering collection:

Resources

• Dataset on Hugging Face
• Research Paper (Chakrabarty et al. 2025)

License

MIT

Segmentation Strategies

Advanced patterns for splitting books into training chunks while preserving narrative coherence.

The Segmentation Problem

Books present unique challenges for training data creation:

1. Variable paragraph length: Some authors write single paragraphs spanning 1000+ words 2. Dialogue-heavy sections: Short exchanges that individually are too small 3. Scene boundaries: Natural break points that don't align with word counts 4. Stylistic variations: Authors shift voice between narrative, dialogue, and exposition

Poor segmentation teaches the model to produce:

• Incomplete thoughts
• Abrupt endings
• Incoherent transitions
• Fragmented style

Two-Tier Strategy

Tier 1: Paragraph-Based Accumulation

The default approach for well-structured text:

class Tier1Segmenter:
    def __init__(self, min_words: int = 250, max_words: int = 650):
        self.min_words = min_words
        self.max_words = max_words
    
    def segment(self, text: str) -> list[Chunk]:
        paragraphs = self._split_paragraphs(text)
        chunks = []
        current = ChunkBuilder()
        
        for para in paragraphs:
            word_count = len(para.split())
            
            # Check if single paragraph exceeds max
            if word_count > self.max_words:
                # Finalize current chunk if exists
                if current.word_count > 0:
                    chunks.append(current.build())
                    current = ChunkBuilder()
                
                # Mark for Tier 2 processing
                chunks.append(Chunk(
                    text=para,
                    requires_tier2=True,
                    word_count=word_count
                ))
                continue
            
            # Would this paragraph overflow current chunk?
            if current.word_count + word_count > self.max_words:
                if current.word_count >= self.min_words:
                    chunks.append(current.build())
                    current = ChunkBuilder()
            
            current.add(para)
        
        # Don't forget the last chunk
        if current.word_count > 0:
            chunks.append(current.build())
        
        return chunks
    
    def _split_paragraphs(self, text: str) -> list[str]:
        # Split on double newlines, preserve single newlines within
        paragraphs = text.split('\n\n')
        return [p.strip() for p in paragraphs if p.strip()]

Tier 2: LLM-Assisted Segmentation

For oversized paragraphs that cannot be split at paragraph boundaries:

class Tier2Segmenter:
    def __init__(self, model: str = "gpt-4o"):
        self.model = model
        self.prompt_template = self._load_prompt()
    
    async def segment(self, oversized_chunk: Chunk) -> list[Chunk]:
        """Split an oversized paragraph using LLM."""
        
        response = await self._call_llm(
            self.prompt_template.format(text=oversized_chunk.text)
        )
        
        segments = self._parse_segments(response)
        
        # Validate zero-deletion
        original_words = len(oversized_chunk.text.split())
        segmented_words = sum(len(s.split()) for s in segments)
        
        if abs(original_words - segmented_words) > 5:  # Allow tiny variance
            raise SegmentationError(
                f"Word count mismatch: {original_words} -> {segmented_words}"
            )
        
        return [
            Chunk(text=s, requires_tier2=False, word_count=len(s.split()))
            for s in segments
        ]
    
    def _load_prompt(self) -> str:
        return """Segment this text into excerpts of minimum 300-350 words.

Requirements:
- Each excerpt must be grammatically complete from start
- Each excerpt must not feel abruptly cut off
- Zero deletion - maintain original word count exactly
- Break at grammatically natural places:
  * After complete dialogue exchanges
  * At scene transitions
  * After complete thoughts or descriptions
  * Where a paragraph break would naturally occur
- Avoid breaking into too many small excerpts
- Start directly with the excerpts
- Separate excerpts with ===SEGMENT===

Text to segment:
{text}
"""
    
    def _parse_segments(self, response: str) -> list[str]:
        segments = response.split("===SEGMENT===")
        return [s.strip() for s in segments if s.strip()]

Scene-Aware Segmentation

For higher-quality results, detect scene boundaries:

class SceneAwareSegmenter:
    """Prefer breaking at scene boundaries when within word limits."""
    
    SCENE_MARKERS = [
        r'\n\n\* \* \*\n\n',      # Asterisk dividers
        r'\n\n---\n\n',            # Dash dividers
        r'\n\n###\n\n',            # Hash dividers
        r'\n\nCHAPTER \d+',        # Chapter headings
        r'\n\n[A-Z]{3,}\n\n',      # All-caps scene breaks
    ]
    
    def find_scene_breaks(self, text: str) -> list[int]:
        """Find character positions of scene breaks."""
        breaks = []
        for pattern in self.SCENE_MARKERS:
            for match in re.finditer(pattern, text):
                breaks.append(match.start())
        return sorted(set(breaks))
    
    def segment_with_scenes(self, text: str) -> list[Chunk]:
        scene_breaks = self.find_scene_breaks(text)
        
        # If scene breaks exist, prefer them over arbitrary paragraph breaks
        if scene_breaks:
            return self._segment_at_scenes(text, scene_breaks)
        else:
            return Tier1Segmenter().segment(text)

Dialogue Handling

Dialogue-heavy sections require special handling:

class DialogueAwareSegmenter:
    """Group dialogue exchanges to maintain conversation coherence."""
    
    def is_dialogue_paragraph(self, para: str) -> bool:
        """Check if paragraph is primarily dialogue."""
        # Count dialogue markers
        quote_count = para.count('"') + para.count("'")
        word_count = len(para.split())
        
        # If more than 20% of words are in quotes, it's dialogue-heavy
        return quote_count > word_count * 0.2
    
    def segment(self, text: str) -> list[Chunk]:
        paragraphs = text.split('\n\n')
        chunks = []
        current = ChunkBuilder()
        in_dialogue_block = False
        
        for para in paragraphs:
            is_dialogue = self.is_dialogue_paragraph(para)
            
            # Don't break in the middle of a dialogue exchange
            if is_dialogue:
                in_dialogue_block = True
                current.add(para)
            else:
                if in_dialogue_block:
                    # End of dialogue block - good break point
                    in_dialogue_block = False
                    if current.word_count >= 250:
                        chunks.append(current.build())
                        current = ChunkBuilder()
                
                current.add(para)
                
                # Check if we've exceeded max
                if current.word_count > 650:
                    chunks.append(current.build())
                    current = ChunkBuilder()
        
        if current.word_count > 0:
            chunks.append(current.build())
        
        return chunks

Validation Pipeline

Every segmentation result should pass validation:

class SegmentationValidator:
    def validate(self, chunks: list[Chunk]) -> ValidationResult:
        errors = []
        warnings = []
        
        for i, chunk in enumerate(chunks):
            # Check word count bounds
            if chunk.word_count < 200:
                warnings.append(f"Chunk {i}: Only {chunk.word_count} words")
            if chunk.word_count > 700:
                errors.append(f"Chunk {i}: {chunk.word_count} words exceeds max")
            
            # Check sentence completeness
            if not self._ends_with_terminal(chunk.text):
                errors.append(f"Chunk {i}: Ends mid-sentence")
            
            if not self._starts_grammatically(chunk.text):
                errors.append(f"Chunk {i}: Starts mid-sentence")
            
            # Check for orphaned dialogue
            if chunk.text.count('"') % 2 != 0:
                warnings.append(f"Chunk {i}: Unbalanced quotes")
        
        return ValidationResult(
            valid=len(errors) == 0,
            errors=errors,
            warnings=warnings
        )
    
    def _ends_with_terminal(self, text: str) -> bool:
        text = text.strip()
        return text[-1] in '.!?"\'—'
    
    def _starts_grammatically(self, text: str) -> bool:
        text = text.strip()
        # Should start with capital or quote
        return text[0].isupper() or text[0] in '"\'—'

Performance Considerations

Edge Cases

1. Stream-of-consciousness writing

• Single "paragraphs" spanning pages
• Solution: Force Tier 2 with explicit sentence boundary detection

2. Poetry or verse

• Line breaks are semantic, not formatting
• Solution: Treat each stanza as atomic unit

3. Non-fiction with lists/bullets

• Bullet points break paragraph detection
• Solution: Pre-process to convert bullets to prose

4. Multiple narrators

• Voice shifts within chapters
• Solution: Detect narrator markers and prefer breaking there

Integration with Pipeline

class SegmentationAgent:
    def __init__(self, config: SegmentationConfig):
        self.tier1 = Tier1Segmenter(
            min_words=config.min_words,
            max_words=config.max_words
        )
        self.tier2 = Tier2Segmenter(model=config.tier2_model)
        self.validator = SegmentationValidator()
    
    async def segment(self, text: str) -> list[Chunk]:
        # Phase 1: Tier 1 segmentation
        chunks = self.tier1.segment(text)
        
        # Phase 2: Process oversized chunks with Tier 2
        final_chunks = []
        for chunk in chunks:
            if chunk.requires_tier2:
                sub_chunks = await self.tier2.segment(chunk)
                final_chunks.extend(sub_chunks)
            else:
                final_chunks.append(chunk)
        
        # Phase 3: Validate
        result = self.validator.validate(final_chunks)
        if not result.valid:
            raise SegmentationError(result.errors)
        
        if result.warnings:
            logger.warning(f"Segmentation warnings: {result.warnings}")
        
        return final_chunks

Tinker Format Specification

This reference documents the exact data structures required for Tinker supervised fine-tuning.

Core Data Types

Datum

The fundamental training unit in Tinker:

from tinker import types

datum = types.Datum(
    model_input=types.ModelInput.from_ints(tokens=input_tokens),
    loss_fn_inputs={
        "target_tokens": target_tokens,  # List[int] - shifted by 1 for next-token prediction
        "weights": weights               # List[float] - 0.0 for prompt, 1.0 for completion
    }
)

ModelInput

Container for tokenized input:

# Simple text-only input
model_input = types.ModelInput.from_ints(tokens=[...])

# Multi-modal (for VLMs)
model_input = types.ModelInput(chunks=[
    types.EncodedTextChunk(tokens=[...]),
    types.ImageChunk(data=image_bytes, format="png"),
    types.EncodedTextChunk(tokens=[...])
])

Token Weight Assignment

The weights array determines which tokens contribute to the loss:

Renderer System

Tinker uses renderers to convert message lists to tokens with proper weights.

Using Built-in Renderers

from tinker_cookbook import renderers, tokenizer_utils

# Get tokenizer for your model
tokenizer = tokenizer_utils.get_tokenizer("meta-llama/Llama-3.1-8B-Instruct")

# Get appropriate renderer
renderer = renderers.get_renderer("llama3", tokenizer)

# Convert messages to training format
messages = [
    {"role": "system", "content": "You are a creative writer..."},
    {"role": "user", "content": "Write a 500 word excerpt..."},
    {"role": "assistant", "content": "The actual book text..."}
]

model_input, weights = renderer.build_supervised_example(messages)

Renderer Output Visualization

The renderer assigns weights per-token:

Token          Weight
<|im_start|>   0.0
system         0.0
\n             0.0
You are...     0.0
<|im_end|>     0.0
...            ...
<|im_start|>   0.0
assistant      0.0
\n             0.0
The actual     1.0    <- Completion starts
book text      1.0
...            1.0
<|im_end|>     1.0    <- Final token weighted

JSONL Format

For batch processing, use standard conversation JSONL:

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Converting JSONL to Datum

import json
from tinker import types
from tinker_cookbook import renderers, tokenizer_utils

def load_dataset(jsonl_path: str, model_name: str) -> list[types.Datum]:
    """Load JSONL and convert to Tinker Datum objects."""
    
    tokenizer = tokenizer_utils.get_tokenizer(model_name)
    renderer = renderers.get_renderer("llama3", tokenizer)
    
    data = []
    with open(jsonl_path) as f:
        for line in f:
            example = json.loads(line)
            messages = example["messages"]
            
            model_input, weights = renderer.build_supervised_example(messages)
            
            # Get token sequences
            input_tokens = model_input.to_ints()
            target_tokens = input_tokens[1:]  # Shift for next-token prediction
            input_tokens = input_tokens[:-1]
            weights = weights[1:]  # Align weights with targets
            
            datum = types.Datum(
                model_input=types.ModelInput.from_ints(tokens=input_tokens),
                loss_fn_inputs={
                    "target_tokens": target_tokens,
                    "weights": weights
                }
            )
            data.append(datum)
    
    return data

Training Loop Integration

import tinker
from tinker import types

async def train_on_book_dataset(
    dataset: list[types.Datum],
    model_name: str,
    learning_rate: float = 1e-4,
    epochs: int = 1
):
    """Train on book SFT dataset."""
    
    service_client = tinker.ServiceClient()
    training_client = await service_client.create_lora_training_client_async(
        base_model=model_name,
        rank=32
    )
    
    for epoch in range(epochs):
        for batch_start in range(0, len(dataset), 1):  # Batch size 1
            batch = dataset[batch_start:batch_start + 1]
            
            # Forward-backward with cross-entropy loss
            fwd_bwd_future = await training_client.forward_backward_async(
                batch, 
                loss_fn="cross_entropy"
            )
            
            # Optimizer step with aggressive learning rate
            optim_future = await training_client.optim_step_async(
                types.AdamParams(learning_rate=learning_rate * 2.0)
            )
            
            # Wait for completion
            fwd_bwd_result = await fwd_bwd_future
            optim_result = await optim_future

Key Constraints

1. Batch Size: Use 1 for style transfer. Larger batches average out stylistic gradients.

2. Sequence Length: Keep chunks under 1000 tokens. Longer sequences dilute local style patterns.

3. Learning Rate: Use 2x multiplier (e.g., 2e-4 instead of 1e-4) for faster style convergence.

4. Token Alignment: Target tokens must be shifted by 1 position from input tokens.

5. Weight Precision: Weights should be float32, typically 0.0 or 1.0.

Model Selection

For book SFT, consider:

References

• Tinker Cookbook: tinker_cookbook/supervised/train.py
• Renderer implementations: tinker_cookbook/renderers.py
• Type definitions: tinker/types.py