Llm Inference
- 72 installs
- 7 repo stars
- Updated January 15, 2026
- eyadsibai/ltk
Helps with ai & agent building tasks.
About
llm-inference is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.
- llm-inference
- AI & Agent Building
- AI-coding skill
Llm Inference by the numbers
- 72 all-time installs (skills.sh)
- Ranked #5,635 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Jul 30, 2026 (Skillselion catalog sync)
npx skills add https://github.com/eyadsibai/ltk --skill llm-inferenceAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 72 |
|---|---|
| repo stars | ★ 7 |
| Last updated | January 15, 2026 |
| Repository | eyadsibai/ltk ↗ |
What it does
Helps with ai & agent building tasks.
Files
LLM Inference
High-performance inference engines for serving large language models.
---
Engine Comparison
| Engine | Best For | Hardware | Throughput | Setup |
|---|---|---|---|---|
| vLLM | Production serving | GPU | Highest | Medium |
| llama.cpp | Local/edge, CPU | CPU/GPU | Good | Easy |
| TGI | HuggingFace models | GPU | High | Easy |
| Ollama | Local desktop | CPU/GPU | Good | Easiest |
| TensorRT-LLM | NVIDIA production | NVIDIA GPU | Highest | Complex |
---
Decision Guide
| Scenario | Recommendation |
|---|---|
| Production API server | vLLM or TGI |
| Maximum throughput | vLLM |
| Local development | Ollama or llama.cpp |
| CPU-only deployment | llama.cpp |
| Edge/embedded | llama.cpp |
| Apple Silicon | llama.cpp with Metal |
| Quick experimentation | Ollama |
| Privacy-sensitive (no cloud) | llama.cpp |
---
vLLM
Production-grade serving with PagedAttention for optimal GPU memory usage.
Key Innovations
| Feature | What It Does |
|---|---|
| PagedAttention | Non-contiguous KV cache, better memory utilization |
| Continuous batching | Dynamic request grouping for throughput |
| Speculative decoding | Small model drafts, large model verifies |
Strengths: Highest throughput, OpenAI-compatible API, multi-GPU Limitations: GPU required, more complex setup
Key concept: Serves OpenAI-compatible endpoints—drop-in replacement for OpenAI API.
---
llama.cpp
C++ inference for running models anywhere—laptops, phones, Raspberry Pi.
Quantization Formats (GGUF)
| Format | Size (7B) | Quality | Use Case |
|---|---|---|---|
| Q8_0 | ~7 GB | Highest | When you have RAM |
| Q6_K | ~6 GB | High | Good balance |
| Q5_K_M | ~5 GB | Good | Balanced |
| Q4_K_M | ~4 GB | OK | Memory constrained |
| Q2_K | ~2.5 GB | Low | Minimum viable |
Recommendation: Q4_K_M for best quality/size balance.
Memory Requirements
| Model Size | Q4_K_M | RAM Needed |
|---|---|---|
| 7B | ~4 GB | 8 GB |
| 13B | ~7 GB | 16 GB |
| 30B | ~17 GB | 32 GB |
| 70B | ~38 GB | 64 GB |
Platform Optimization
| Platform | Key Setting |
|---|---|
| Apple Silicon | n_gpu_layers=-1 (Metal offload) |
| CUDA GPU | n_gpu_layers=-1 + offload_kqv=True |
| CPU only | n_gpu_layers=0 + set n_threads to core count |
Strengths: Runs anywhere, GGUF format, Metal/CUDA support Limitations: Lower throughput than vLLM, single-user focused
Key concept: GGUF format + quantization = run large models on consumer hardware.
---
Key Optimization Concepts
| Technique | What It Does | When to Use |
|---|---|---|
| KV Cache | Reuse attention computations | Always (automatic) |
| Continuous Batching | Group requests dynamically | High-throughput serving |
| Tensor Parallelism | Split model across GPUs | Large models |
| Quantization | Reduce precision (fp16→int4) | Memory constrained |
| Speculative Decoding | Small model drafts, large verifies | Latency sensitive |
| GPU Offloading | Move layers to GPU | When GPU available |
---
Common Parameters
| Parameter | Purpose | Typical Value |
|---|---|---|
| n_ctx | Context window size | 2048-8192 |
| n_gpu_layers | Layers to offload | -1 (all) or 0 (none) |
| temperature | Randomness | 0.0-1.0 |
| max_tokens | Output limit | 100-2000 |
| n_threads | CPU threads | Match core count |
---
Troubleshooting
| Issue | Solution |
|---|---|
| Out of memory | Reduce n_ctx, use smaller quant |
| Slow inference | Enable GPU offload, use faster quant |
| Model won't load | Check GGUF integrity, check RAM |
| Metal not working | Reinstall with -DLLAMA_METAL=on |
| Poor quality | Use higher quant (Q5_K_M, Q6_K) |
Resources
- vLLM: <https://docs.vllm.ai>
- llama.cpp: <https://github.com/ggerganov/llama.cpp>
- TGI: <https://huggingface.co/docs/text-generation-inference>
- Ollama: <https://ollama.ai>
- GGUF Models: <https://huggingface.co/TheBloke>
Related skills
Setup Matt Pocock SkillsScaffold the per-repo configuration that Matt Pocock’s engineering agent skills rely on so they understand the issue tracker, triage labels, and domain documentation la495k196k
Lark AppsConnect Claude, Cursor or custom agents directly to Lark (Feishu) for messaging, document automation, approval workflows and enterprise data access.402k
CavemanSlash token usage by roughly 75% while keeping every technical detail intact when working with Claude Code, Cursor or similar agents.391k94.7k
Lark Skill MakerQuickly turn any Lark/Feishu OpenAPI call or multi-step workflow into a reusable agent skill with its own SKILL.md.386k16k
Running Claude Code Via Litellm CopilotRun Claude Code at a fraction of the cost by routing requests through LiteLLM to the GitHub Copilot Chat API.270k72
Codex PetGenerate a complete Codex Pet spritesheet and metadata from one reference image without needing an OpenAI key or Codex Pro.246k10