Peft Fine Tuning

# PEFT Advanced Usage Guide

## Advanced LoRA Variants

### DoRA (Weight-Decomposed Low-Rank Adaptation)

DoRA decomposes weights into magnitude and direction components, often achieving better results than standard LoRA:

```python
from peft import LoraConfig

dora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    use_dora=True,  # Enable DoRA
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, dora_config)
```

**When to use DoRA**:
- Consistently outperforms LoRA on instruction-following tasks
- Slightly higher memory (~10%) due to magnitude vectors
- Best for quality-critical fine-tuning

### AdaLoRA (Adaptive Rank)

Automatically adjusts rank per layer based on importance:

```python
from peft import AdaLoraConfig

adalora_config = AdaLoraConfig(
    init_r=64,              # Initial rank
    target_r=16,            # Target average rank
    tinit=200,              # Warmup steps
    tfinal=1000,            # Final pruning step
    deltaT=10,              # Rank update frequency
    beta1=0.85,
    beta2=0.85,
    orth_reg_weight=0.5,    # Orthogonality regularization
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM"
)
```

**Benefits**:
- Allocates more rank to important layers
- Can reduce total parameters while maintaining quality
- Good for exploring optimal rank distribution

### LoRA+ (Asymmetric Learning Rates)

Different learning rates for A and B matrices:

```python
from peft import LoraConfig

# LoRA+ uses higher LR for B matrix
lora_plus_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    use_rslora=True,  # Rank-stabilized LoRA (related technique)
)

# Manual implementation of LoRA+
from torch.optim import AdamW

# Group parameters
lora_A_params = [p for n, p in model.named_parameters() if "lora_A" in n]
lora_B_params = [p for n, p in model.named_parameters() if "lora_B" in n]

optimizer = AdamW([
    {"params": lora_A_params, "lr": 1e-4},
    {"params": lora_B_params, "lr": 1e-3},  # 10x higher for B
])
```

### rsLoRA (Rank-Stabilized LoRA)

Scales LoRA outputs to stabilize training with different ranks:

```python
lora_config = LoraConfig(
    r=64,
    lora_alpha=64,
    use_rslora=True,  # Enables rank-stabilized scaling
    target_modules="all-linear"
)
```

**When to use**:
- When experimenting with different ranks
- Helps maintain consistent behavior across rank values
- Recommended for r > 32

## LoftQ (LoRA-Fine-Tuning-aware Quantization)

Initializes LoRA weights to compensate for quantization error:

```python
from peft import LoftQConfig, LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# LoftQ configuration
loftq_config = LoftQConfig(
    loftq_bits=4,              # Quantization bits
    loftq_iter=5,              # Alternating optimization iterations
)

# LoRA config with LoftQ initialization
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    init_lora_weights="loftq",
    loftq_config=loftq_config,
    task_type="CAUSAL_LM"
)

# Load quantized model
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config
)

model = get_peft_model(model, lora_config)
```

**Benefits over standard QLoRA**:
- Better initial quality after quantization
- Faster convergence
- ~1-2% better final accuracy on benchmarks

## Custom Module Targeting

### Target specific layers

```python
# Target only first and last transformer layers
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["model.layers.0.self_attn.q_proj",
                    "model.layers.0.self_attn.v_proj",
                    "model.layers.31.self_attn.q_proj",
                    "model.layers.31.self_attn.v_proj"],
    layers_to_transform=[0, 31]  # Alternative approach
)
```

### Layer pat