Reinforcement Learning

Reinforcement Learning

Identity

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Reinforcement Learning

Patterns

Golden Rules

---

Rule

Reward shaping is critical

Reason

Sparse rewards make learning nearly impossible

---

Rule

Start simple, scale up

Reason

Debug on toy environments before complex ones

---

Rule

Monitor training metrics obsessively

Reason

RL training is notoriously unstable

---

Rule

Use appropriate baselines

Reason

Reduces variance in policy gradients

---

Rule

Clip/constrain policy updates

Reason

Prevents catastrophic policy collapse

---

Rule

Separate exploration from exploitation

Reason

Ensures sufficient state-space coverage

Algorithm Taxonomy

Value Based

Algorithms

• Q-Learning
• DQN
• Double DQN
• Dueling DQN

Learns

Q(s,a) - Value of state-action pairs

Best For

• Discrete actions
• Atari games

Policy Based

Algorithms

• REINFORCE
• Policy Gradient

Learns

pi(a|s) - Policy directly

Best For

• Continuous actions
• Robotics

Actor Critic

Algorithms

• A2C/A3C
• PPO
• SAC
• TRPO

Learns

Both V and pi

Best For

• Most tasks
• LLM alignment

On Vs Off Policy

On Policy

Algorithms

• PPO
• A2C

Property

Learn from current policy samples

Pros

More stable

Cons

Fresh data required

Off Policy

Algorithms

• DQN
• SAC

Property

Learn from any policy samples

Pros

More sample efficient

Cons

Requires replay buffer

Discount Factor

Short Horizon

Medium Horizon

Long Horizon

Infinite Horizon

Ppo Config

Clip Epsilon

0.1-0.3 (typically 0.2)

Entropy Coef

0.01 (encourages exploration)

Value Coef

0.5

Max Grad Norm

0.5

N Epochs

3-10 per batch

Rlhf Pipeline

Step1 Sft

Description

Supervised Fine-Tuning

Purpose

Establish baseline helpful behavior

Step2 Reward Model

Description

Train on human preference comparisons

Output

Reward(prompt, response) = scalar

Loss

Bradley-Terry: -log(sigmoid(r_chosen - r_rejected))

Step3 Ppo

Description

Optimize policy with KL penalty

Formula

reward = r(x,y) - beta * KL(pi || pi_ref)

Anti-Patterns

---

Pattern

Sparse rewards

Problem

Agent learns nothing

Solution

Reward shaping, dense rewards

---

Pattern

No baseline/advantage

Problem

High variance gradients

Solution

Use GAE, value baseline

---

Pattern

Large policy updates

Problem

Training collapse

Solution

PPO clipping, KL penalty

---

Pattern

No replay buffer (off-policy)

Problem

Sample inefficiency

Solution

Experience replay

---

Pattern

Same network for Q and target

Problem

Unstable learning

Solution

Separate target network

---

Pattern

Ignoring KL in RLHF

Problem

Model drift, reward hacking

Solution

KL penalty to reference model

Reinforcement Learning - Sharp Edges

Reward Hacking in RLHF

Id

reward-hacking

Severity

critical

Summary

Model finds exploits in reward model instead of being helpful

Symptoms

• Reward score increases but quality decreases
• Model produces verbose but unhelpful responses
• Responses game the reward model's biases
• Human evaluators disagree with high reward scores

Why

The reward model is an imperfect proxy for human preferences. Given enough optimization pressure, the policy finds reward model exploits. Common exploits: verbosity, sycophancy, specific phrases reward model likes.

Gotcha

Optimizing reward too aggressively

for step in range(1000000): reward = reward_model(response) loss = -reward # Pure reward maximization loss.backward()

Model learns to game reward model

Solution

1. KL penalty to stay close to reference

reward = reward_model(response) - kl_coef * kl_divergence(policy, reference)

2. Periodically refresh reward model on new data

3. Ensemble multiple reward models

4. Human evaluation checkpoints

5. Early stopping based on held-out evaluation

if eval_score < best_score - tolerance: break # Stop before overfitting to reward model

Catastrophic Policy Collapse

Id

policy-collapse

Severity

critical

Summary

Policy suddenly degenerates after seeming stable

Symptoms

• Entropy drops to near zero
• Policy outputs become deterministic/repetitive
• Reward suddenly crashes
• All samples look identical

Why

Without proper constraints, policy gradient updates can be too large. A large bad update can push the policy into a degenerate state. From there, all samples reinforce the bad behavior.

Gotcha

REINFORCE without clipping

ratio = new_prob / old_prob loss = -ratio * advantage # No limit on ratio!

If ratio >> 1, can destroy the policy

Solution

PPO clipping prevents catastrophic updates

ratio = torch.exp(new_log_prob - old_log_prob)

surr1 = ratio advantage surr2 = torch.clamp(ratio, 1 - clip_epsilon, 1 + clip_epsilon) advantage

loss = -torch.min(surr1, surr2).mean()

Also: monitor entropy, add entropy bonus

entropy_bonus = -entropy_coef * entropy.mean() total_loss = loss + entropy_bonus

Agent Never Learns Due to Sparse Rewards

Id

sparse-reward-failure

Severity

high

Summary

Reward signal too rare for learning to occur

Symptoms

• Agent takes random actions indefinitely
• No improvement over random baseline
• Policy gradient has near-zero signal

Why

If reward only comes at episode end (or rarely), the agent gets no feedback about which intermediate actions were good. Credit assignment becomes impossible.

Gotcha

Sparse reward environment

def step(action):

Only reward at the very end

if is_goal_reached(): return observation, 1.0, True, {} # Reward only here return observation, 0.0, False, {} # No intermediate signal

Solution

1. Reward shaping - add intermediate rewards

def shaped_reward(state, action, next_state): sparse = 1.0 if is_goal_reached(next_state) else 0.0

Potential-based shaping (preserves optimal policy)

potential_diff = gamma * potential(next_state) - potential(state)

return sparse + shaping_coef * potential_diff

2. Curiosity-driven exploration

3. Hierarchical RL with subgoals

4. Curriculum learning - start with easier tasks

Q-Value Overestimation in DQN

Id

value-function-overestimation

Severity

high

Summary

Q-learning systematically overestimates values

Symptoms

• Q-values grow unrealistically large
• Agent is overconfident about bad actions
• Performance is worse than expected from Q-values

Why

max_a Q(s,a) takes the maximum over noisy estimates. This systematically picks the action with the highest positive noise. Over many updates, this bias compounds.

Gotcha

Standard DQN - has overestimation bias

target_q = reward + gamma * target_net(next_state).max()

max() selects the noisiest high estimate

Solution

Double DQN - use online net to select, target net to evaluate

next_actions = online_net(next_state).argmax(dim=1) target_q = reward + gamma * target_net(next_state).gather(1, next_actions)

The action selection and value estimation use different networks

This breaks the overestimation cycle

KL Divergence Explodes During RLHF

Id

kl-divergence-explosion

Severity

high

Summary

Policy drifts too far from reference model

Symptoms

• KL penalty term dominates the loss
• Model forgets base capabilities
• Responses become incoherent
• Generation quality degrades

Why

Without proper KL constraint, the policy can drift arbitrarily far. The reference model represents the base capabilities we want to preserve. Drifting too far means catastrophic forgetting.

Gotcha

KL coefficient too low

kl_coef = 0.001 # Too weak! reward = reward_score - kl_coef * kl # Barely constrains

Solution

1. Appropriate KL coefficient (0.1 - 0.5 typical)

kl_coef = 0.1

2. Adaptive KL penalty

if kl > target_kl 1.5: kl_coef = 1.5 elif kl < target_kl / 1.5: kl_coef /= 1.5

3. Hard KL constraint (TRPO-style)

if kl > max_kl: reject_update()

Reinforcement Learning - Validations

PPO Without Clipping

Id

ppo-no-clipping

Severity

error

Type

regex

Pattern

• ratio.advantage(?!.clamp|clip)
• policy_loss.=.-.ratio.advantage(?!.*min)

Message

PPO requires clipping to prevent catastrophic policy updates.

Fix Action

Add: torch.clamp(ratio, 1-eps, 1+eps) and use min of clipped/unclipped

Applies To

• */.py

Advantages Not Normalized

Id

no-advantage-normalization

Severity

warning

Type

regex

Pattern

• advantage.=(?!.(mean|std|normalize))

Message

Normalizing advantages reduces variance and improves training stability.

Fix Action

Add: advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

Applies To

• */ppo*.py
• */rl*.py

Missing Entropy Bonus

Id

no-entropy-bonus

Severity

warning

Type

regex

Pattern

• policy_loss(?!.*entropy)
• actor_loss(?!.*entropy)

Message

Entropy bonus encourages exploration and prevents premature convergence.

Fix Action

Add: total_loss = policy_loss - entropy_coef * entropy.mean()

Applies To

• */ppo*.py
• */a2c*.py

RLHF Without KL Penalty

Id

rlhf-no-kl-penalty

Severity

error

Type

regex

Pattern

• reward_model.response(?!.kl|.*reference)

Message

RLHF requires KL penalty to prevent model drift and reward hacking.

Fix Action

Add: reward = reward_score - kl_coef * kl_divergence(policy, reference)

Applies To

• */rlhf*.py
• */alignment*.py

DQN Without Target Network

Id

dqn-no-target-network

Severity

error

Type

regex

Pattern

• q_network.max(?!.target)
• q_net.next_state(?!.target)

Message

DQN requires separate target network for stable learning.

Fix Action

Add target network and periodically update: target_net.load_state_dict(q_net.state_dict())

Applies To

• */dqn*.py
• */q_learning*.py

RL Training Without Gradient Clipping

Id

no-gradient-clipping-rl

Severity

warning

Type

regex

Pattern

• loss\.backward\(\)\s\n\soptimizer\.step(?!.*clip_grad)

Message

RL training benefits from gradient clipping for stability.

Fix Action

Add: nn.utils.clip_grad_norm_(parameters, max_grad_norm)

Applies To

• */rl*.py
• */ppo*.py

Training Without Reward Logging

Id

no-reward-logging

Severity

info

Type

regex

Pattern

• for.episode(?!.log|.print|.wandb|.*writer)

Message

RL training requires careful monitoring of reward and metrics.

Fix Action

Log: episode_reward, policy_loss, value_loss, entropy, KL divergence

Applies To

• */train*.py
• */rl*.py