Quantizing Models Bitsandbytes
Fit and train larger Hugging Face models on limited GPU RAM using bitsandbytes quantization, CPU offloading, and related memory tricks.
Overview
Quantizing Models Bitsandbytes is an agent skill for the Build phase that configures bitsandbytes quantization and memory offloading so larger Hugging Face models fit on constrained GPUs.
Install
npx skills add https://github.com/orchestra-research/ai-research-skills --skill quantizing-models-bitsandbytesWhat is this skill?
- BitsAndBytesConfig patterns for 4-bit load with bfloat16 compute dtype
- CPU and multi-GPU offloading via device_map auto and max_memory caps
- Documents quantization as roughly 50–75% memory reduction alongside other techniques
- Covers gradient checkpointing, 8-bit optimizers, and mixed FP16/BF16 training
- Explains ~5–10× slowdown trade-off when weights live on CPU RAM
- Quantization cited as 50–75% memory reduction in the skill overview
- CPU offloading trade-off described as roughly 5–10× slower versus all-GPU residency
Adoption & trust: 1 installs on skills.sh; 9.4k GitHub stars; 2/3 security scanners passed (skills.sh audits).
What problem does it solve?
Your fine-tune or local inference setup crashes with OOM because full-precision weights do not fit on available GPU memory.
Who is it for?
Indie ML builders fine-tuning or running open-weight LLMs locally or on a single cloud GPU with tight VRAM.
Skip if: Teams with ample multi-GPU clusters who train full precision by default, or builders who only need hosted API inference with no local model code.
When should I use this skill?
OOM or VRAM limits block loading, fine-tuning, or running a transformers model and you need bitsandbytes quantization or offload recipes.
What do I get? / Deliverables
You leave with working quantization_config and device_map patterns that load multi-billion-parameter models with documented memory trade-offs and training options.
- BitsAndBytesConfig and from_pretrained loading snippet
- max_memory and device_map plan for GPU and CPU
- Documented memory strategy combining quantization and checkpointing
Recommended Skills
Journey fit
Model loading and training configuration is core product engineering during Build, especially when the backend or agent stack depends on an open-weight LLM you host or fine-tune. Backend is the right shelf because memory layout, device maps, and training configs are infrastructure decisions—not UI, docs, or launch SEO.
How it compares
Skill package for Hugging Face + bitsandbytes memory tuning—not a managed inference platform or generic prompt-engineering guide.
Common Questions / FAQ
Who is quantizing-models-bitsandbytes for?
Solo developers and small teams shipping agent or API products that depend on self-hosted transformers models and hit GPU memory walls.
When should I use quantizing-models-bitsandbytes?
During Build backend work when configuring model load, fine-tuning, or experimentation and you need 4-bit loads, CPU offload, or checkpointing to proceed.
Is quantizing-models-bitsandbytes safe to install?
It is research-oriented community guidance; confirm license compatibility for base models and review the Security Audits panel on this page before running downloaded weights.
SKILL.md
READMESKILL.md - Quantizing Models Bitsandbytes
# Memory Optimization Complete guide to CPU offloading, gradient checkpointing, memory profiling, and advanced memory-saving strategies with bitsandbytes. ## Overview Memory optimization techniques for fitting large models: - **Quantization**: 50-75% reduction (covered in other docs) - **CPU offloading**: Move weights to CPU/disk - **Gradient checkpointing**: Trade compute for memory - **Optimizer strategies**: 8-bit, paged optimizers - **Mixed precision**: FP16/BF16 training ## CPU Offloading ### Basic CPU Offloading Move parts of the model to CPU RAM when not in use. ```python from transformers import AutoModelForCausalLM, BitsAndBytesConfig import torch config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16 ) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-70b-hf", quantization_config=config, device_map="auto", # Automatic device placement max_memory={0: "40GB", "cpu": "100GB"} # 40GB GPU, 100GB CPU ) ``` **How it works**: - Weights stored on CPU - Moved to GPU only when needed for computation - Automatically managed by `accelerate` **Trade-off**: ~5-10× slower but enables larger models ### Multi-GPU Offloading Distribute across multiple GPUs + CPU: ```python model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-405b-hf", quantization_config=config, device_map="auto", max_memory={ 0: "70GB", # GPU 0 1: "70GB", # GPU 1 2: "70GB", # GPU 2 3: "70GB", # GPU 3 "cpu": "200GB" # CPU RAM } ) ``` **Result**: 405B model (4-bit = ~200GB) fits on 4×80GB GPUs + CPU ### Disk Offloading For models too large even for CPU RAM: ```python model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-405b-hf", quantization_config=config, device_map="auto", offload_folder="./offload", # Disk offload directory offload_state_dict=True, max_memory={0: "40GB", "cpu": "50GB"} ) ``` **Trade-off**: Extremely slow (~100× slower) but works ### Manual Device Mapping For precise control: ```python device_map = { "model.embed_tokens": 0, # GPU 0 "model.layers.0": 0, "model.layers.1": 0, # ... "model.layers.40": 1, # GPU 1 "model.layers.41": 1, # ... "model.layers.79": "cpu", # CPU "model.norm": "cpu", "lm_head": "cpu" } model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-70b-hf", quantization_config=config, device_map=device_map ) ``` ## Gradient Checkpointing Recompute activations during backward pass instead of storing them. ### Enable for HuggingFace Models ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-13b-hf", quantization_config=config ) # Enable gradient checkpointing model.gradient_checkpointing_enable() ``` **Memory savings**: ~30-50% activation memory **Cost**: ~20% slower training ### With QLoRA ```python from peft import prepare_model_for_kbit_training # Enable gradient checkpointing before preparing for training model.gradient_checkpointing_enable() model = prepare_model_for_kbit_training( model, use_gradient_checkpointing=True ) ``` ### Configure Checkpointing Frequency ```python # Checkpoint every layer (maximum memory savings) model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False}) ``` ### Memory Breakdown Example: Llama 2 13B forward pass | Component | Without Checkpointing | With Checkpointing | |-----------|----------------------|-------------------| | Model weights | 26 GB | 26 GB | | Activations | 12 GB | **3 GB** | | Gradients | 26 GB | 26 GB | | Optimizer | 52 GB | 52 GB | | **Total** | 116 GB | **107 GB** | **Savings**: ~9GB for 13B model ## 8-Bit Optimizers Use 8-bit optimizer states instead of 32-bit. ### Standard AdamW Memory ``` Optimizer memory = 2 × model_params × 4 bytes (FP32) = 8 × model_par