Deploy the GGUF version of Gemma 4 12B as a local model service. By default use llama.cpp / llama-server、Apple Metal、Q4_K_M and tmux, listening only on loopback; switch routes only when the user explicitly requests QAT, 256K, a comparison demo, or Ollama.
Operating Contract
- Direct actions: read local hardware, disk, ports, processes, and model cache; when the user has already requested local deployment, install or upgrade the specified packages, download the selected model, create a dedicated model directory and tmux session, and bind only to
127.0.0.1。 - Escalate before: stopping existing processes that do not belong to this Skill, overwriting existing models or configurations, deleting user data, listening on public addresses, changing the firewall, or downloading large model variants the user did not select.
- Evidence-backed pushback: if the model tag, context, memory budget, or local capability specified by the user conflicts with the currently verifiable state, first show the command output and propose a runnable profile; do not fabricate support status.
- Feedback loop: check current state → select and restate the profile → execute one deployment route → complete health, model, and chat verification in the current session → report endpoint, resources, and limitations.
Default choices
- Default model repository:
ggml-org/gemma-4-12B-it-GGUF - Default quantization:
Q4_K_M - Default model name:
gemma-4-12b-it - Default endpoint:
http://127.0.0.1:8080 - Default context:
32768 - 12B long context: when the user explicitly requests it, choose
65536or131072 - QAT repository:
google/gemma-4-12B-it-qat-q4_0-gguf - QAT profile：
Q4_0、262144context - Default background session:
gemma4-12b - Disable thinking by default:
--reasoning off, to avoid the OpenAI API'smessage.contentbeing empty - Ollama: use only when the user explicitly requests Ollama or needs the Ollama ecosystem
QAT is quantization simulated during training and is not lossless. For critical tasks, still verify with real responses in the current session. When the user explicitly wants higher quality, prefer recommending Q6_K or Q8_0by default bf16。
Profile selection
| Profile | Use cases | Model / quant | Context | Port / alias |
|---|---|---|---|---|
daily-q4km-32k |
Default daily chat, coding, low-risk local API | ggml-org/...:Q4_K_M |
32768 |
8080 / gemma-4-12b-it |
long-q4km-128k |
Explicitly need longer context, but keep the default GGUF route | ggml-org/...:Q4_K_M |
65536 or 131072 |
8080 / gemma-4-12b-it |
qat-q4_0-256k |
Explicitly request QAT, Q4_0, 256K, or low-memory long context | google/...qat-q4_0-gguf:Q4_0 |
262144 |
8080 / gemma-4-12b-it-qat-q4_0 |
compare-32k-vs-256k |
Screen recording, demo, or A/B comparison of resources and speed | Left Q4_K_M, right QAT Q4_0 |
32768 + 262144 |
8080 + 8081 |
The final reply must state the selected profile, port, context, and rationale. Do not treat 256K as a daily default.
Execution flow
1. Search and confirm current state
First check existing installations, processes, ports, cache, hardware, and disk to avoid duplicate deployment:
command -v llama-server || true
llama-server --version || true
tmux has-session -t gemma4-12b 2>/dev/null && tmux display-message -p -t gemma4-12b '#S #{pane_pid}' || true
lsof -nP -iTCP:8080 -sTCP:LISTEN || true
ls -lh "$HOME/Library/Caches/llama.cpp/"*gemma-4-12B-it*Q4_K_M*.gguf 2>/dev/null || true
find "$HOME/Library/Caches/llama.cpp" "$HOME/Models" \( -name '*gemma-4-12b-it-qat-q4_0*.gguf' -o -name '*gemma-4-12B-it-qat-q4_0*.gguf' \) 2>/dev/null || true
system_profiler SPHardwareDataType | sed -n '1,30p'
df -h "$HOME"
These || true are only used to allow the expected finding of "not yet installed/not yet running"; the actual output must be shown, and a failed query must not be described as a successful deployment.
2. Execute one deployment route
daily-q4km-32k、long-q4km-128k、qat-q4_0-256korcompare-32k-vs-256k: first read and execute llama.cpp deployment route。- User explicitly requests Ollama: first read and execute Ollama deployment route。
- Do not mix the two routes at the same time, and do not start a second service without checking the port.
3. Verify and report
After deployment, you must read and execute Verification, resources, and troubleshooting. Success requires at least that the current session demonstrates:
/healthreturns a healthy status/v1/modelsor the Ollama model list contains the selected model- When the user requests a long context, the runtime reports the actual
n_ctx - The body of one chat response is non-empty
- The endpoint still listens only on the expected local address and port
Final response shape
By default, answer in Chinese and include:
- The actual endpoint URL and model id
- The selected profile, quantization, and context
- tmux/session management commands
- The verification results for the current session
- The actual resource summary, failed items, and limitations
When there is no verification data, write "unverified"; do not use planned values in place of runtime values.
Cross-check
When the deployment plan involves models, context, or resource judgments beyond the default profile, use
agents/openai.yaml for an independent review; the review cannot replace local verification in the current session.