Gemma4 Local Deploy

Gemma 4 12B 本地部署

目标：把 Gemma 4 12B 的 GGUF 版本部署成本机模型服务。默认用 llama.cpp / llama-server + Apple Metal + Q4_K_M + tmux 暴露 OpenAI-compatible API；用户明确要 QAT、256K 或对比演示时，切到 QAT Q4_0 profile；用户明确要 Ollama 时，再走 Ollama 导入路径。

默认选择

• 默认模型仓库：ggml-org/gemma-4-12B-it-GGUF
• 默认量化：Q4_K_M
• 默认模型名：gemma-4-12b-it
• 默认端口：127.0.0.1:8080
• 默认上下文：32768
• 12B 长上下文：用户明确要求更大上下文时，可改为 65536 或原生最高 131072
• QAT 仓库：google/gemma-4-12B-it-qat-q4_0-gguf
• QAT 量化：Q4_0，文件名通常是 gemma-4-12b-it-qat-q4_0.gguf
• QAT 上下文：用户要求 QAT、最大上下文或 256K 时，用 262144
• 默认后台方式：tmux 会话 gemma4-12b
• 默认关闭 thinking：--reasoning off，避免 OpenAI API 的 message.content 为空
• Ollama 路径：只在用户明确要 Ollama、需要接 Ollama 生态，或询问 ollama pull gemma4:12b 时使用

如果用户明确要更高质量，优先建议 Q6_K 或 Q8_0；不要默认上 bf16，除非用户接受更大内存和更慢加载。QAT 是训练时模拟量化以降低压缩后的质量损失，不等于无损；关键任务仍要做当前会话验证。

Profile 选择

先根据用户目标选择 profile。不要把 256K 当作默认值，也不要在用户只要日常本地服务时自动切 QAT。

选择后在最终回复里说清楚 profile、端口、上下文和为什么这么选。

执行流程

1. 搜索并确认现状

先查已有安装、进程、端口和模型缓存，避免重复部署：

command -v llama-server || true
llama-server --version || true
tmux has-session -t gemma4-12b 2>/dev/null && tmux display-message -p -t gemma4-12b '#S #{pane_pid}' || true
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
ls -lh "$HOME/Library/Caches/llama.cpp/"*gemma-4-12B-it*Q4_K_M*.gguf 2>/dev/null || true
find "$HOME/Library/Caches/llama.cpp" "$HOME/Models" \( -name '*gemma-4-12b-it-qat-q4_0*.gguf' -o -name '*gemma-4-12B-it-qat-q4_0*.gguf' \) 2>/dev/null || true

On Mac, also record hardware:

system_profiler SPHardwareDataType | sed -n '1,30p'
df -h "$HOME"

2. Install or upgrade llama.cpp

Use Homebrew on macOS:

brew install llama.cpp
# If already installed, upgrade only this package when possible.
brew upgrade llama.cpp
llama-server --version

Gemma 4 GGUF requires a llama.cpp build that recognizes general.architecture = gemma4. If loading fails with:

unknown model architecture: 'gemma4'

then upgrade llama.cpp and retry. A verified good local build was 9430; newer stable or HEAD is also acceptable.

3. Download/load the model

For the default daily-q4km-32k profile, first-run download can be done by llama-server -hf:

llama-server \
  -hf ggml-org/gemma-4-12B-it-GGUF:Q4_K_M \
  --no-mmproj \
  --ctx-size 32768 \
  --gpu-layers 99 \
  --parallel 1 \
  --reasoning off \
  --host 127.0.0.1 \
  --port 8080 \
  --alias gemma-4-12b-it

After the model is cached, prefer starting with the local file path. Typical cache path:

$HOME/Library/Caches/llama.cpp/ggml-org_gemma-4-12B-it-GGUF_gemma-4-12B-it-Q4_K_M.gguf

For the qat-q4_0-256k profile, use the Google QAT GGUF repo:

llama-server \
  -hf google/gemma-4-12B-it-qat-q4_0-gguf:Q4_0 \
  --ctx-size 262144 \
  --gpu-layers 99 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --reasoning off \
  --host 127.0.0.1 \
  --port 8080 \
  --alias gemma-4-12b-it-qat-q4_0

If the download should be explicit or reusable outside the llama.cpp cache:

mkdir -p "$HOME/Models/gemma4-qat"
huggingface-cli download google/gemma-4-12B-it-qat-q4_0-gguf \
  gemma-4-12b-it-qat-q4_0.gguf \
  --local-dir "$HOME/Models/gemma4-qat"

4. Run persistently with tmux

If port 8080 is free and no gemma4-12b session exists:

tmux new-session -d -s gemma4-12b 'llama-server -m "$HOME/Library/Caches/llama.cpp/ggml-org_gemma-4-12B-it-GGUF_gemma-4-12B-it-Q4_K_M.gguf" --ctx-size 32768 --gpu-layers 99 --parallel 1 --reasoning off --host 127.0.0.1 --port 8080 --alias gemma-4-12b-it'

If $HOME is not expanded inside single quotes in the target shell, use the absolute path instead.

For qat-q4_0-256k with an explicit local file:

tmux new-session -d -s gemma4-qat-256k 'llama-server -m "$HOME/Models/gemma4-qat/gemma-4-12b-it-qat-q4_0.gguf" --ctx-size 262144 --gpu-layers 99 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning off --host 127.0.0.1 --port 8080 --alias gemma-4-12b-it-qat-q4_0'

For compare-32k-vs-256k, keep separate session names and ports:

tmux new-session -d -s gemma4-left-32k 'llama-server -m "$HOME/Library/Caches/llama.cpp/ggml-org_gemma-4-12B-it-GGUF_gemma-4-12B-it-Q4_K_M.gguf" --ctx-size 32768 --gpu-layers 99 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning off --host 127.0.0.1 --port 8080 --alias gemma-4-12b-it'
tmux new-session -d -s gemma4-right-256k 'llama-server -m "$HOME/Models/gemma4-qat/gemma-4-12b-it-qat-q4_0.gguf" --ctx-size 262144 --gpu-layers 99 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning off --host 127.0.0.1 --port 8081 --alias gemma-4-12b-it-qat-q4_0'

Management commands:

tmux attach -t gemma4-12b
tmux kill-session -t gemma4-12b
tmux kill-session -t gemma4-qat-256k
tmux kill-session -t gemma4-left-32k
tmux kill-session -t gemma4-right-256k

5. Increase 12B context when requested

Do not tell the user 12B is limited to 32K. 32768 is the conservative default startup value. The 12B GGUF metadata can support a native training context of 131072.

Use this selection table:

Restart with a larger context, keeping --parallel 1 and using Flash Attention plus quantized KV cache to reduce long-context pressure:

tmux kill-session -t gemma4-12b 2>/dev/null || true
tmux new-session -d -s gemma4-12b 'llama-server -m "$HOME/Library/Caches/llama.cpp/ggml-org_gemma-4-12B-it-GGUF_gemma-4-12B-it-Q4_K_M.gguf" --ctx-size 131072 --gpu-layers 99 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning off --host 127.0.0.1 --port 8080 --alias gemma-4-12b-it'

If the model path is different, find it first:

find "$HOME/Library/Caches/llama.cpp" "$HOME/Models" -name '*gemma-4-12B-it*Q4_K_M*.gguf' 2>/dev/null

After restart, prove the actual context value instead of relying on the command line:

curl -fsS http://127.0.0.1:8080/v1/models | jq '.data[0].meta | {n_ctx, n_ctx_train, n_params, size}'

Expected long-context 12B result:

{
  "n_ctx": 131072,
  "n_ctx_train": 131072
}

If startup fails or memory pressure is high, retry --ctx-size 65536.

6. Verify before claiming success

Run all three checks from the current session:

curl -fsS http://127.0.0.1:8080/health
curl -fsS http://127.0.0.1:8080/v1/models
curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma-4-12b-it","messages":[{"role":"user","content":"用一句中文回答：现在可以问你问题吗？"}],"max_tokens":80,"temperature":0.2}'

Success requires:

• /health returns {"status":"ok"}
• /v1/models lists the chosen alias, for example gemma-4-12b-it or gemma-4-12b-it-qat-q4_0
• /v1/models shows the requested n_ctx when the user asked for larger context
• chat response has non-empty choices[0].message.content

7. Report resource usage

Use both process and macOS footprint views:

pid=$(pgrep -f 'llama-server .*gemma-4-12B-it-Q4_K_M.gguf' | head -1)
ps -p "$pid" -o pid,stat,%cpu,%mem,rss,vsz,etime,command
footprint -p "$pid" -summary 2>/dev/null | sed -n '1,80p'
memory_pressure | sed -n '1,20p'

Explain the difference clearly:

• GGUF Q4_K_M is a quantized model; its file is about 7GB, not 24GB full precision.
• ps RSS includes mapped model pages and often shows around 9-11GB for Q4 12B.
• footprint may show lower physical pressure because clean mmap pages can be discarded and reread.
• Apple Silicon uses unified memory; GPU work does not appear as a separate NVIDIA-style VRAM number.
• Larger --ctx-size increases KV/cache memory and may reduce tokens/sec even when the prompt is short.

8. Optional: Ollama route

Use this path only when the user asks for Ollama. Treat official Ollama registry state as live-changing: re-test before claiming support.

Install and start Ollama:

brew install ollama
ollama --version
lsof -nP -iTCP:11434 -sTCP:LISTEN || true
tmux new-session -d -s ollama-gemma4 'OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve'
curl -fsS http://127.0.0.1:11434/api/version

First try the official path:

ollama pull gemma4:12b

If it succeeds, run:

ollama run gemma4:12b "用一句中文回答：现在可以问你问题吗？"

If it fails with pull model manifest: file does not exist, fall back to GGUF import. Download or reuse a local GGUF:

mkdir -p "$HOME/Models/gemma4-12b"
huggingface-cli download ggml-org/gemma-4-12B-it-GGUF \
  gemma-4-12B-it-Q4_K_M.gguf \
  --local-dir "$HOME/Models/gemma4-12b"

Create a Modelfile:

cat > "$HOME/Models/gemma4-12b/Modelfile" <<EOF
FROM $HOME/Models/gemma4-12b/gemma-4-12B-it-Q4_K_M.gguf
EOF

Homebrew ollama builds may lack sidecar llama-server / llama-quantize binaries. If ollama create or ollama run reports either binary missing, create a stable working directory with symlinks to llama.cpp:

mkdir -p "$HOME/ollama-gemma4/build/lib/ollama"
ln -sf /opt/homebrew/bin/llama-server "$HOME/ollama-gemma4/build/lib/ollama/llama-server"
ln -sf /opt/homebrew/bin/llama-quantize "$HOME/ollama-gemma4/build/lib/ollama/llama-quantize"
tmux kill-session -t ollama-gemma4 2>/dev/null || true
tmux new-session -d -s ollama-gemma4 "cd '$HOME/ollama-gemma4' && OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve"

Import and run:

ollama create gemma4-12b-gguf-local -f "$HOME/Models/gemma4-12b/Modelfile"
ollama list
ollama run gemma4-12b-gguf-local "用一句中文回答：Ollama 能跑 Gemma 4 12B 吗？"

For Ollama success, report whether it was:

• official registry pull: ollama pull gemma4:12b
• manual GGUF import: ollama create gemma4-12b-gguf-local
• workaround needed: sidecar symlinks to llama.cpp

Troubleshooting

Final response shape

Answer in Chinese unless the user asks otherwise. Include:

• endpoint URL
• model id
• tmux/session management commands
• verification results from this session
• resource summary and any caveats

Cross-Check Agent

Use agents/openai.yaml only when the deployment plan or troubleshooting result needs an independent model review before execution.