Purpose
FreeToken is an edge-native Mixture-of-Experts (MoE) inference serving engine, positioned to run frontier-scale open-weight models on personal and consumer-grade hardware. It aims to bring data-center-level model serving capabilities to users' existing desktop hardware, thereby enabling large-scale models to run locally and providing a fast, efficient inference experience. According to the official introduction, it can run frontier MoE models with 290B+ parameters locally on a gaming PC and achieve relatively fast interactive speeds.
Main Capabilities
- Fast edge-native runtime: Provides efficient MoE serving, including bandwidth-adaptive CPU-GPU cooperative execution (the $q^\star$ strategy), full-layer double-buffered prefill streaming, a global LRU expert cache, graph-compatible execution, and the FTW fast weight format.
- Semantics-aware caching: Features semantic anchor checkpoints for recurrent state and KV cache, enabling agent context editing (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
- Elastic memory management: Supports dynamically reallocating VRAM at runtime between the expert cache and KV memory, without restarting the engine or reloading weights.
- Broad MoE and ecosystem support: Supports a variety of frontier open-weight MoE models (such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2), covering different parameter scales and quantization formats (such as MXFP4, NVFP4, FP8, BF16), and provides Anthropic/OpenAI-compatible APIs for seamless integration with real coding and tool-calling agents (such as Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
- Diverse consumer-grade hardware: Scalably runs on consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
How to Use
- FreeToken for Windows or Linux can be downloaded at flashml.ai. The installer sets up the engine for the user and provides a graphical interface for running models, chatting, and tuning the engine.
- It is recommended to install FreeToken using uv or pip, and it can also be built from source. Specific installation commands, the list of supported models,as well as instructions for repairing old FTW checkpoints, are provided in more detailed pages in the official documentation.
- The official repository page provides entry points for downloads, the paper, developer Slack, community Discord, community WeChat, and more.
Use Cases
- Running frontier-scale open-weight MoE models locally on a personal computer or workstation.
- Using them with coding and tool-calling agents (such as Codex, Claude Code, etc.), accessed via Anthropic/OpenAI-compatible APIs.
- Research or development scenarios that require dynamic VRAM adjustment and semantics-aware caching to avoid redundant computation.
Limitations
The materials do not state specific supported context lengths, throughput, or latency figures, nor do they provide details such as pricing, cost, and platform availability. The actual runnable model scale and speed depend on the local hardware configuration. FreeToken uses the Apache License 2.0. If FreeToken is used in research, the official request is to cite its paper. The project is inspired by mini-sglang and learns from or reuses the design and code of projects such as SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM, and llama.cpp.
