Fine Tuning With Trl

TRL - Transformer Reinforcement Learning

Quick start

TRL provides post-training methods for aligning language models with human preferences.

Installation:

pip install trl transformers datasets peft accelerate

Supervised Fine-Tuning (instruction tuning):

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()

DPO (align with preferences):

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()

Common workflows

Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer
)
trainer.train()
trainer.save_model()

Step 2: Train reward model

Train model to predict human preferences:

from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig

# Load SFT model as base
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-SFT",
    num_labels=1  # Single reward score
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

# Load preference data (chosen/rejected pairs)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# Configure training
training_args = RewardConfig(
    output_dir="Qwen2.5-0.5B-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

# Train reward model
trainer = RewardTrainer(
    model=model,
    args=training_args,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()
trainer.save_model()

Step 3: PPO reinforcement learning

Optimize policy using reward model:

python -m trl.scripts.ppo \
    --model_name_or_path Qwen2.5-0.5B-SFT \
    --reward_model_path Qwen2.5-0.5B-Reward \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir Qwen2.5-0.5B-PPO \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --total_episodes 10000

Step 4: Evaluate

from transformers import pipeline

# Load aligned model
generator = pipeline("text-generation", model="Qwen2.5-0.5B-PPO")

# Test
prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200)[0]["generated_text"]
print(output)

Workflow 2: Simple preference alignment with DPO

Align model with preferences without reward model.

Copy this checklist:

DPO Training:
- [ ] Step 1: Prepare preference dataset
- [ ] Step 2: Configure DPO
- [ ] Step 3: Train with DPOTrainer
- [ ] Step 4: Evaluate alignment

Step 1: Prepare preference dataset

Dataset format:

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

Load dataset:

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Or load your own
# dataset = load_dataset("json", data_files="preferences.json")

Step 2: Configure DPO

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,  # KL penalty strength
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

Step 3: Train with DPOTrainer

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)

trainer.train()
trainer.save_model()

CLI alternative:

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

Workflow 3: Memory-efficient online RL with GRPO

Train with reinforcement learning using minimal memory.

For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see [references/grpo-training.md](references/grpo-training.md). A production-ready training script is in [templates/basic_grpo_training.py](templates/basic_grpo_training.py).

Copy this checklist:

GRPO Training:
- [ ] Step 1: Define reward function
- [ ] Step 2: Configure GRPO
- [ ] Step 3: Train with GRPOTrainer

Step 1: Define reward function

def reward_function(completions, **kwargs):
    """
    Compute rewards for completions.

    Args:
        completions: List of generated texts

    Returns:
        List of reward scores (floats)
    """
    rewards = []
    for completion in completions:
        # Example: reward based on length and unique words
        score = len(completion.split())  # Favor longer responses
        score += len(set(completion.lower().split()))  # Reward unique words
        rewards.append(score)
    return rewards

Or use a reward model:

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    # Combine prompt + completion
    full_texts = [p + c for p, c in zip(prompts, completions)]
    # Get reward scores
    results = reward_model(full_texts)
    return [r["score"] for r in results]

Step 2: Configure GRPO

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,  # Generate 4 completions per prompt
    max_new_tokens=128
)

Step 3: Train with GRPOTrainer

from datasets import load_dataset
from trl import GRPOTrainer

# Load prompt-only dataset
dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,  # Your reward function
    args=config,
    train_dataset=dataset
)

trainer.train()

CLI:

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

When to use vs alternatives

Use TRL when:

• Need to align model with human preferences
• Have preference data (chosen/rejected pairs)
• Want to use reinforcement learning (PPO, GRPO)
• Need reward model training
• Doing RLHF (full pipeline)

Method selection:

• SFT: Have prompt-completion pairs, want basic instruction following
• DPO: Have preferences, want simple alignment (no reward model needed)
• PPO: Have reward model, need maximum control over RL
• GRPO: Memory-constrained, want online RL
• Reward Model: Building RLHF pipeline, need to score generations

Use alternatives instead:

• HuggingFace Trainer: Basic fine-tuning without RL
• Axolotl: YAML-based training configuration
• LitGPT: Educational, minimal fine-tuning
• Unsloth: Fast LoRA training

Common issues

Issue: OOM during DPO training

Reduce batch size and sequence length:

config = DPOConfig(
    per_device_train_batch_size=1,  # Reduce from 4
    max_length=512,  # Reduce from 1024
    gradient_accumulation_steps=8  # Maintain effective batch
)

Or use gradient checkpointing:

model.gradient_checkpointing_enable()

Issue: Poor alignment quality

Tune beta parameter:

# Higher beta = more conservative (stays closer to reference)
config = DPOConfig(beta=0.5)  # Default 0.1

# Lower beta = more aggressive alignment
config = DPOConfig(beta=0.01)

Issue: Reward model not learning

Check loss type and learning rate:

config = RewardConfig(
    learning_rate=1e-5,  # Try different LR
    num_train_epochs=3  # Train longer
)

Ensure preference dataset has clear winners:

# Verify dataset
print(dataset[0])
# Should have clear chosen > rejected

Issue: PPO training unstable

Adjust KL coefficient:

config = PPOConfig(
    kl_coef=0.1,  # Increase from 0.05
    cliprange=0.1  # Reduce from 0.2
)

Advanced topics

SFT training guide: See references/sft-training.md for dataset formats, chat templates, packing strategies, and multi-GPU training.

DPO variants: See references/dpo-variants.md for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.

Reward modeling: See references/reward-modeling.md for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.

Online RL methods: See references/online-rl.md for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.

GRPO deep dive: See references/grpo-training.md for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in templates/basic_grpo_training.py.

Hardware requirements

• GPU: NVIDIA (CUDA required)
• VRAM: Depends on model and method
• SFT 7B: 16GB (with LoRA)
• DPO 7B: 24GB (stores reference model)
• PPO 7B: 40GB (policy + reward model)
• GRPO 7B: 24GB (more memory efficient)
• Multi-GPU: Supported via accelerate
• Mixed precision: BF16 recommended (A100/H100)

Memory optimization:

• Use LoRA/QLoRA for all methods
• Enable gradient checkpointing
• Use smaller batch sizes with gradient accumulation

Resources

• Docs: https://huggingface.co/docs/trl/
• GitHub: https://github.com/huggingface/trl
• Papers:
• "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
• "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
• "Group Relative Policy Optimization" (GRPO, 2024)
• Examples: https://github.com/huggingface/trl/tree/main/examples/scripts

DPO Variants

Complete guide to Direct Preference Optimization loss variants in TRL.

Overview

DPO optimizes models using preference data (chosen/rejected pairs). TRL supports 10+ loss variants for different scenarios.

Loss Types

1. Sigmoid (Standard DPO)

Formula: -log(sigmoid(β * logits))

When to use: Default choice, general preference alignment

Config:

DPOConfig(
    loss_type="sigmoid",
    beta=0.1,  # KL penalty
    per_device_train_batch_size=64,
    learning_rate=1e-6
)

2. IPO (Identity Policy Optimization)

Formula: (logits - 1/(2β))²

When to use: Better theoretical foundation, reduce overfitting

Config:

DPOConfig(
    loss_type="ipo",
    beta=0.1,
    per_device_train_batch_size=90,
    learning_rate=1e-2
)

3. Hinge (SLiC)

Formula: ReLU(1 - β * logits)

When to use: Margin-based objective

Config:

DPOConfig(
    loss_type="hinge",
    beta=0.1,
    per_device_train_batch_size=512,
    learning_rate=1e-4
)

4. Robust DPO

Formula: Sigmoid with label smoothing for noise robustness

When to use: Noisy preference labels

Config:

DPOConfig(
    loss_type="robust",
    beta=0.01,
    label_smoothing=0.1,  # Noise probability
    per_device_train_batch_size=16,
    learning_rate=1e-3,
    max_prompt_length=128,
    max_length=512
)

5. BCO Pair (Binary Classification)

Formula: Train binary classifier (chosen=1, rejected=0)

When to use: Pairwise preference data

Config:

DPOConfig(
    loss_type="bco_pair",
    beta=0.01,
    per_device_train_batch_size=128,
    learning_rate=5e-7,
    max_prompt_length=1536,
    max_completion_length=512
)

6. SPPO Hard

Formula: Push chosen→0.5, rejected→-0.5

When to use: Nash equilibrium, sparse data

Config:

DPOConfig(
    loss_type="sppo_hard",
    beta=0.1
)

7. DiscoPOP

Formula: Log-Ratio Modulated Loss

When to use: Automated loss discovery

Config:

DPOConfig(
    loss_type="discopop",
    beta=0.05,
    discopop_tau=0.05,
    per_device_train_batch_size=64,
    learning_rate=5e-7
)

8. APO Zero

Formula: Increase chosen, decrease rejected likelihood

When to use: Model worse than winning outputs

Config:

DPOConfig(
    loss_type="apo_zero",
    beta=0.1,
    per_device_train_batch_size=64,
    learning_rate=2e-7,
    max_prompt_length=512,
    max_completion_length=512
)

9. APO Down

Formula: Decrease both, emphasize rejected reduction

When to use: Model better than winning outputs

Config:

DPOConfig(
    loss_type="apo_down",
    beta=0.1,
    # Same hyperparameters as apo_zero
)

10. AOT & AOT Pair

Formula: Distributional alignment via stochastic dominance

When to use:

• aot_pair: Paired preference data
• aot: Unpaired data

Config:

DPOConfig(
    loss_type="aot_pair",  # or "aot"
    beta=0.1,
    label_smoothing=0.0
)

Multi-Loss Training

Combine multiple losses:

DPOConfig(
    loss_type=["sigmoid", "ipo"],
    loss_weights=[0.7, 0.3],  # Weighted combination
    beta=0.1
)

Key Parameters

Beta (β)

Controls deviation from reference model:

• Higher (0.5): More conservative, stays close to reference
• Lower (0.01): More aggressive alignment
• Default: 0.1

Label Smoothing

For robust DPO:

• 0.0: No smoothing (default)
• 0.1-0.3: Moderate noise robustness
• 0.5: Maximum noise tolerance

Max Lengths

• max_prompt_length: 128-1536
• max_completion_length: 128-512
• max_length: Total sequence (1024-2048)

Comparison Table

References

• DPO paper: https://arxiv.org/abs/2305.18290
• IPO paper: https://arxiv.org/abs/2310.12036
• TRL docs: https://huggingface.co/docs/trl/dpo_trainer

GRPO (Group Relative Policy Optimization) — Deep Guide

Expert-level patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions using TRL's GRPOTrainer. This is the deep reference for the GRPO workflow summarized in the main skill.

When to use GRPO

Use GRPO when you need to:

• Enforce specific output formats (XML tags, JSON, structured reasoning)
• Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking)
• Improve reasoning capabilities by rewarding chain-of-thought patterns
• Align models to domain-specific behaviors without labeled preference data
• Optimize for multiple objectives simultaneously (format + correctness + style)

Do NOT use GRPO for:

• Simple supervised fine-tuning tasks → use SFT
• Tasks without clear reward signals
• When you already have high-quality preference pairs → use DPO/PPO

Core concepts

1. GRPO algorithm fundamentals

Key mechanism:

• Generates multiple completions per prompt (group size: 4–16)
• Compares completions within each group using reward functions
• Updates policy to favor higher-rewarded responses relative to the group

Critical differences from PPO:

• No separate reward model needed
• More sample-efficient (learns from within-group comparisons)
• Simpler to implement and debug

Mathematical intuition:

For each prompt p:
  1. Generate N completions: {c₁, c₂, ..., cₙ}
  2. Compute rewards: {r₁, r₂, ..., rₙ}
  3. Learn to increase probability of high-reward completions
     relative to low-reward ones in the same group

2. Reward function design philosophy

Golden rules: 1. Compose multiple reward functions — each handles one aspect (format, correctness, style) 2. Scale rewards appropriately — higher weight = stronger signal 3. Use incremental rewards — partial credit for partial compliance 4. Test rewards independently — debug each reward function in isolation

Reward function types:

Implementation workflow

Step 1: Dataset preparation

Critical requirements:

• Prompts in chat format (list of dicts with role and content)
• Include system prompts to set expectations
• For verifiable tasks, include ground truth answers as additional columns

from datasets import load_dataset, Dataset

SYSTEM_PROMPT = """
Respond in the following format:
<reasoning>
[Your step-by-step thinking]
</reasoning>
<answer>
[Final answer]
</answer>
"""

def prepare_dataset(raw_data):
    """Transform raw data into GRPO-compatible format.

    Returns: Dataset with columns:
    - 'prompt': List[Dict] with role/content (system + user messages)
    - 'answer': str (ground truth, optional but recommended)
    """
    return raw_data.map(lambda x: {
        'prompt': [
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': x['question']}
        ],
        'answer': extract_answer(x['raw_answer'])
    })

Pro tips:

• Use one-shot or few-shot examples in the system prompt for complex formats
• Keep prompts concise (max_prompt_length: 256–512 tokens)
• Validate data quality before training (garbage in = garbage out)

Step 2: Reward function implementation

Template structure:

def reward_function_name(
    prompts,        # List[List[Dict]]: Original prompts
    completions,    # List[List[Dict]]: Model generations
    answer=None,    # Optional: Ground truth from dataset
    **kwargs        # Additional dataset columns
) -> list[float]:
    """Evaluate completions and return rewards (one per completion)."""
    responses = [comp[0]['content'] for comp in completions]
    rewards = []
    for response in responses:
        score = compute_score(response)
        rewards.append(score)
    return rewards

Example 1: correctness reward (math/coding)

def correctness_reward(prompts, completions, answer, **kwargs):
    """Reward correct answers with high score."""
    responses = [comp[0]['content'] for comp in completions]
    extracted = [extract_final_answer(r) for r in responses]
    return [2.0 if ans == gt else 0.0
            for ans, gt in zip(extracted, answer)]

Example 2: format reward (structured output)

import re

def format_reward(completions, **kwargs):
    """Reward XML-like structured format."""
    pattern = r'<reasoning>.*?</reasoning>\s*<answer>.*?</answer>'
    responses = [comp[0]['content'] for comp in completions]
    return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0
            for r in responses]

Example 3: incremental format reward (partial credit)

def incremental_format_reward(completions, **kwargs):
    """Award partial credit for format compliance."""
    responses = [comp[0]['content'] for comp in completions]
    rewards = []

    for r in responses:
        score = 0.0
        if '<reasoning>' in r:  score += 0.25
        if '</reasoning>' in r: score += 0.25
        if '<answer>' in r:     score += 0.25
        if '</answer>' in r:    score += 0.25
        # Penalize extra text after closing tag
        if r.count('</answer>') == 1:
            extra_text = r.split('</answer>')[-1].strip()
            score -= len(extra_text) * 0.001
        rewards.append(score)

    return rewards

Critical insight: Combine 3–5 reward functions for robust training. Order matters less than diversity of signals.

Step 3: Training configuration

Memory-optimized config (small GPU)

from trl import GRPOConfig

training_args = GRPOConfig(
    output_dir="outputs/grpo-model",

    # Learning rate
    learning_rate=5e-6,          # Lower = more stable
    adam_beta1=0.9,
    adam_beta2=0.99,
    weight_decay=0.1,
    warmup_ratio=0.1,
    lr_scheduler_type='cosine',

    # Batch settings
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,  # Effective batch = 4

    # GRPO-specific
    num_generations=8,            # Group size: 8–16 recommended
    max_prompt_length=256,
    max_completion_length=512,

    # Training duration
    num_train_epochs=1,
    max_steps=None,

    # Optimization
    bf16=True,                    # Faster on A100/H100
    optim="adamw_8bit",          # Memory-efficient optimizer
    max_grad_norm=0.1,

    # Logging
    logging_steps=1,
    save_steps=100,
    report_to="wandb",
)

High-performance config (large GPU)

training_args = GRPOConfig(
    output_dir="outputs/grpo-model",
    learning_rate=1e-5,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=2,
    num_generations=16,           # Larger groups = better signal
    max_prompt_length=512,
    max_completion_length=1024,
    num_train_epochs=1,
    bf16=True,
    use_vllm=True,                # Fast generation with vLLM
    logging_steps=10,
)

Critical hyperparameters:

Step 4: Model setup and training

Standard setup (Transformers + TRL)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import GRPOTrainer

model_name = "Qwen/Qwen2.5-1.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # 2–3× faster
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Optional: LoRA for parameter-efficient training
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    task_type="CAUSAL_LM",
    lora_dropout=0.05,
)

trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[
        incremental_format_reward,
        format_reward,
        correctness_reward,
    ],
    args=training_args,
    train_dataset=dataset,
    peft_config=peft_config,   # Remove for full fine-tuning
)

trainer.train()
trainer.save_model("final_model")

Unsloth setup (2–3× faster)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-3-1b-it",
    max_seq_length=1024,
    load_in_4bit=True,
    fast_inference=True,
    max_lora_rank=32,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    use_gradient_checkpointing="unsloth",
)

# Rest is identical to the standard setup
trainer = GRPOTrainer(model=model, ...)
trainer.train()

Critical training insights

1. Loss behavior (EXPECTED pattern)

• Loss starts near 0 and INCREASES during training — this is CORRECT
• Loss measures KL divergence from initial policy; the model is learning (diverging from original behavior to optimize rewards)
• Monitor reward metrics, not loss, for progress

2. Reward tracking

Key metrics to watch:

• reward — average across all completions
• reward_std — diversity within groups (should remain > 0)
• kl — KL divergence from reference (should grow moderately)

Healthy pattern:

Step   Reward    Reward_Std   KL
100    0.5       0.3          0.02
200    0.8       0.25         0.05
300    1.2       0.2          0.08  ← Good progression
400    1.5       0.15         0.12

Warning signs:

• reward_std → 0 (model collapsing to a single response)
• kl exploding (> 0.5) — diverging too much, reduce LR
• Reward stuck — reward functions too harsh or model capacity issue

3. Common pitfalls and solutions

Advanced patterns

1. Multi-stage training

For complex tasks, train in stages:

# Stage 1: Format compliance
trainer_stage1 = GRPOTrainer(
    model=model,
    reward_funcs=[incremental_format_reward, format_reward],
    ...
)
trainer_stage1.train()

# Stage 2: Correctness
trainer_stage2 = GRPOTrainer(
    model=model,
    reward_funcs=[format_reward, correctness_reward],
    ...
)
trainer_stage2.train()

2. Adaptive reward scaling

class AdaptiveReward:
    def __init__(self, base_reward_func, initial_weight=1.0):
        self.func = base_reward_func
        self.weight = initial_weight

    def __call__(self, *args, **kwargs):
        rewards = self.func(*args, **kwargs)
        return [r * self.weight for r in rewards]

    def adjust_weight(self, success_rate):
        """Increase weight if model struggling, decrease if succeeding."""
        if success_rate < 0.3:
            self.weight *= 1.2
        elif success_rate > 0.8:
            self.weight *= 0.9

3. Custom dataset integration

def load_custom_knowledge_base(csv_path):
    import pandas as pd
    df = pd.read_csv(csv_path)
    return Dataset.from_pandas(df).map(lambda x: {
        'prompt': [
            {'role': 'system', 'content': CUSTOM_SYSTEM_PROMPT},
            {'role': 'user', 'content': x['question']}
        ],
        'answer': x['expert_answer']
    })

Deployment and inference

Save and merge LoRA

if hasattr(trainer.model, 'merge_and_unload'):
    merged_model = trainer.model.merge_and_unload()
    merged_model.save_pretrained("production_model")
    tokenizer.save_pretrained("production_model")

Inference

from transformers import pipeline

generator = pipeline("text-generation", model="production_model", tokenizer=tokenizer)

result = generator(
    [
        {'role': 'system', 'content': SYSTEM_PROMPT},
        {'role': 'user', 'content': "What is 15 + 27?"},
    ],
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(result[0]['generated_text'])

Best practices checklist

Before training:

• [ ] Validate dataset format (prompts as List[Dict])
• [ ] Test reward functions on sample data
• [ ] Calculate expected max_prompt_length from data
• [ ] Choose num_generations based on GPU memory
• [ ] Set up logging (wandb recommended)

During training:

• [ ] Monitor reward progression (should increase)
• [ ] Check reward_std (should stay > 0.1)
• [ ] Watch for OOM errors (reduce batch size if needed)
• [ ] Sample generations every 50–100 steps
• [ ] Validate format compliance on holdout set

After training:

• [ ] Merge LoRA weights if using PEFT
• [ ] Test on diverse prompts
• [ ] Compare to baseline model
• [ ] Document reward weights and hyperparameters
• [ ] Save reproducibility config

Troubleshooting

Debugging workflow

1. Isolate reward functions — test each independently 2. Check data distribution — ensure diversity in prompts 3. Reduce complexity — start with single reward, add gradually 4. Monitor generations — print samples every N steps 5. Validate extraction logic — ensure answer parsing works

Quick debug reward

def debug_reward(completions, **kwargs):
    responses = [comp[0]['content'] for comp in completions]
    for i, r in enumerate(responses[:2]):
        print(f"Response {i}: {r[:200]}...")
    return [1.0] * len(responses)

# Test without training
trainer = GRPOTrainer(..., reward_funcs=[debug_reward])
trainer.generate_completions(dataset[:1])

Template

A production-ready training script lives at `../templates/basic_grpo_training.py`. It uses Qwen 2.5-1.5B-Instruct with LoRA and three reward functions (incremental format, strict format, correctness) on GSM8K. Copy and adapt: 1. get_dataset() — swap in your data loader 2. Reward functions — tune to your task 3. SYSTEM_PROMPT — match your output format 4. GRPOConfig — adjust hyperparameters for your GPU

References and resources

• TRL GRPO Trainer: https://huggingface.co/docs/trl/grpo_trainer
• GRPO paper (DeepSeek): https://arxiv.org/abs/2402.03300
• DeepSeek R1 paper: https://arxiv.org/abs/2501.12948
• Open R1 implementation: https://github.com/huggingface/open-r1
• TRL examples: https://github.com/huggingface/trl/tree/main/examples
• Unsloth (faster training): https://docs.unsloth.ai/

Critical reminders

• Loss goes UP during training — this is normal (it's KL divergence)
• Use 3–5 reward functions — single rewards often fail
• Test rewards before training — debug each function independently
• Monitor `reward_std` — should stay > 0.1 (avoid mode collapse)
• Start with `num_generations=4–8` — scale up if GPU allows

Online RL Methods

Guide to online reinforcement learning with PPO, GRPO, RLOO, and OnlineDPO.

Overview

Online RL generates completions during training and optimizes based on rewards.

PPO (Proximal Policy Optimization)

Classic RL algorithm for LLM alignment.

Basic Usage

python -m trl.scripts.ppo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --reward_model_path reward-model \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir model-ppo \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --total_episodes 10000 \
    --num_ppo_epochs 4 \
    --kl_coef 0.05

Key Parameters

• kl_coef: KL penalty (0.05-0.2)
• num_ppo_epochs: Epochs per batch (2-4)
• cliprange: PPO clip (0.1-0.3)
• vf_coef: Value function coef (0.1)

GRPO (Group Relative Policy Optimization)

Memory-efficient online RL.

Basic Usage

from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset

# Define reward function
def reward_func(completions, **kwargs):
    return [len(set(c.split())) for c in completions]

config = GRPOConfig(
    output_dir="model-grpo",
    num_generations=4,  # Completions per prompt
    max_new_tokens=128
)

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_func,
    args=config,
    train_dataset=load_dataset("trl-lib/tldr", split="train")
)
trainer.train()

Key Parameters

• num_generations: 2-8 completions
• max_new_tokens: 64-256
• Learning rate: 1e-5 to 1e-4

Memory Comparison

References

• PPO paper: https://arxiv.org/abs/1707.06347
• GRPO paper: https://arxiv.org/abs/2402.03300
• TRL docs: https://huggingface.co/docs/trl/

Reward Modeling

Guide to training reward models with TRL for RLHF pipelines.

Overview

Reward models score completions based on human preferences. Used in:

• PPO training (RL feedback)
• GRPO online RL
• Completion ranking

Basic Training

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

# Load model (num_labels=1 for single reward score)
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    num_labels=1
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

# Load preference dataset (chosen/rejected pairs)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# Configure
config = RewardConfig(
    output_dir="Qwen2.5-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

# Train
trainer = RewardTrainer(
    model=model,
    args=config,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()

Dataset Format

Required fields:

{
  "prompt": "Question or instruction",
  "chosen": "Better response",
  "rejected": "Worse response"
}

Bradley-Terry Loss

Default loss function:

loss = -log(sigmoid(reward_chosen - reward_rejected))

Learns to score chosen > rejected.

Using Reward Models

Inference

from transformers import pipeline

# Load trained reward model
reward_pipe = pipeline("text-classification", model="Qwen2.5-Reward")

# Score completions
texts = ["Good answer", "Bad answer"]
scores = reward_pipe(texts)
print(scores)  # Higher score = better

In PPO

from trl import PPOTrainer, PPOConfig

config = PPOConfig(
    reward_model_path="Qwen2.5-Reward"  # Use trained reward model
)

trainer = PPOTrainer(
    model=policy_model,
    config=config,
    # Reward model loaded automatically
)

Hyperparameters

Evaluation

Check reward separation:

# Chosen should score higher than rejected
chosen_rewards = model(**chosen_inputs).logits
rejected_rewards = model(**rejected_inputs).logits

accuracy = (chosen_rewards > rejected_rewards).float().mean()
print(f"Accuracy: {accuracy:.2%}")  # Target: >80%

References

• InstructGPT paper: https://arxiv.org/abs/2203.02155
• TRL docs: https://huggingface.co/docs/trl/reward_trainer

SFT Training Guide

Complete guide to Supervised Fine-Tuning (SFT) with TRL for instruction tuning and task-specific fine-tuning.

Overview

SFT trains models on input-output pairs to minimize cross-entropy loss. Use for:

• Instruction following
• Task-specific fine-tuning
• Chatbot training
• Domain adaptation

Dataset Formats

Format 1: Prompt-Completion

[
  {
    "prompt": "What is the capital of France?",
    "completion": "The capital of France is Paris."
  }
]

Format 2: Conversational (ChatML)

[
  {
    "messages": [
      {"role": "user", "content": "What is Python?"},
      {"role": "assistant", "content": "Python is a programming language."}
    ]
  }
]

Format 3: Text-only

[
  {"text": "User: Hello\nAssistant: Hi! How can I help?"}
]

Basic Training

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure
config = SFTConfig(
    output_dir="Qwen2.5-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer
)
trainer.train()

Chat Templates

Apply chat templates automatically:

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,  # Messages format
    tokenizer=tokenizer
    # Chat template applied automatically
)

Or manually:

def format_chat(example):
    messages = example["messages"]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    return {"text": text}

dataset = dataset.map(format_chat)

Packing for Efficiency

Pack multiple sequences into one to maximize GPU utilization:

config = SFTConfig(
    packing=True,  # Enable packing
    max_seq_length=2048,
    dataset_text_field="text"
)

Benefits: 2-3× faster training Trade-off: Slightly more complex batching

Multi-GPU Training

accelerate launch --num_processes 4 train_sft.py

Or with config:

config = SFTConfig(
    output_dir="model-sft",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1
)

LoRA Fine-Tuning

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    peft_config=lora_config  # Add LoRA
)

Hyperparameters

References

• TRL docs: https://huggingface.co/docs/trl/sft_trainer
• Examples: https://github.com/huggingface/trl/tree/main/examples/scripts