colibri

Colibri

Assist with Colibri — the pure-C inference engine for running GLM-5.2 (744B Mixture-of-Experts) on consumer machines with ~25 GB RAM, featuring expert disk-streaming, MLA attention, and native speculative decoding (MTP).

When to use this skill

• Setup & build: Installing C build tools, running setup.sh, configuring OpenMP, cross-platform compilation (Linux, macOS, Windows via WSL)
• Model download & conversion: Running coli convert to fetch GLM-5.2-FP8 shards, convert to int4 container, prepare MTP heads (int8 quantization)
• Inference modes: Running chat mode (coli chat), API server setup, batch inference, configuring sampling and temperature
• Expert caching & storage: Tuning LRU cache sizes, pin-hot-store configuration, disk I/O optimization, thermal management
• Speculative decoding (MTP): Enabling/disabling multi-token prediction, measuring draft acceptance rates, optimizing verification latency
• GPU acceleration: Configuring CUDA pinned experts, GPU vRAM budgeting, hybrid CPU-GPU inference
• Performance profiling: Measuring tokens/second, analyzing expert-load patterns, memory residency tuning
• Integration & deployment: Embedding Colibri in applications, wrapping inference API, local-first LLM pipelines

When not to use this skill

• Python-first LLM frameworks: Use llama-index, crewai-multi-agent, or pydantic-ai for high-level abstractions
• Distributed/cloud inference: Use cloud LLM APIs or deepspeed for multi-GPU/multi-node training
• LLM fine-tuning: Use dspy, openrlhf-training, or moe-training for training workflows
• General AI infra: Use system-environment-setup for unrelated dev environments

Core architecture

Main components:

1. glm.c (~2,400 lines) — Faithful GLM-5.2 (glm_moe_dsa) forward pass with MLA attention, MTP speculative decoding, DeepSeek-V3 sigmoid router
2. Expert streaming — 21,504 routed experts (75 MoE layers × 256 experts) live on disk (~370 GB total at int4), streamed on demand with per-layer LRU cache
3. Storage hierarchy — Dense layers (attention, shared experts, embeddings) in RAM (~9.9 GB at int4), experts on NVMe/HDD, OS page cache as L2
4. MTP speculative decoding — GLM-5.2's native multi-token-prediction head (layer 78, int8 for lossless 39–59% draft acceptance) drafts and main model verifies in one forward
5. MLA attention — q/kv-LoRA + interleaved partial RoPE with compressed KV-cache (576 floats/token vs 32,768 dense — 57× smaller)

CLI modes:

• coli chat — Interactive chat with streaming, auto-detected RAM/cache/MTP
• coli convert — Download GLM-5.2-FP8, quantize to int4, prepare MTP int8 head
• coli plan — Inspect storage hierarchy, vRAM budget, expert placement before loading

Key parameters

Parameter	Scope	Default	Notes
COLI_MODEL	env	(required)	Path to converted int4 model directory
--ram <GB>	plan/chat	auto-detect	RAM budget for dense layers (min 10 GB)
--vram <GB>	plan/chat	0 (CPU-only)	GPU vRAM for pinned experts (requires CUDA build)
--cache <MB>	plan/chat	auto-detect	Per-layer LRU cache size for routed experts
--mtp on/off	chat	auto-detect	Enable/disable speculative decoding (off = single-token mode)
--temperature <0–2>	chat	0.95	Sampling temperature (0 = greedy)
--top-p <0–1>	chat	0.9	Nucleus sampling threshold

Performance metrics (community benchmarks)

• Tokens/second (CPU, int4 cache warm):
  • Single token: ~1.2 tok/s (minimal cache misses)
  • MTP enabled: ~2.2–2.8 tok/s (2.2–2.8× speedup, 39–59% draft acceptance)
  • MTP with cold cache: may regress due to extra expert loads (~660 → ~1100/token)
• Residency: Dense layers ~9.9 GB RAM; experts 370 GB disk
• Memory overhead: ~11 GB per-token token from routed experts; 57× smaller KV-cache than dense attention
• Conversion time: ~30–60 min (one-time, depends on disk speed and internet)

Integration patterns

• Local chat UI: Wrap coli chat in a terminal/web interface
• API server: Stream inference output over HTTP/gRPC
• Batch processing: Run multiple prompts with persistent model load
• Hybrid CPU-GPU: Pin hot experts to GPU vRAM, stream others from disk
• Edge/mobile: int4 model size (~370 GB) requires local NVMe; consider pruning/distillation for smaller devices