SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FreeToken - Open-source on-device MoE large model inference system, supporting full-power operation

Author: AI小集 · Original source: AI工具集

What is FreeToken

FreeToken is an on-device MoE large model inference system jointly open-sourced by UC Berkeley, MIT, and other institutions. Through technologies such as full-layer double buffering, bandwidth-adaptive hybrid scheduling, Agent state reuse, and elastic VRAM hot scaling, the system unifies the CPU, memory, PCIe, and GPU of a personal computer into an elastic computing platform, enabling full-power operation of Qwen3.6-35B, DeepSeek-V4-Flash 284B, and other ultra-large MoE models on a single consumer-grade GPU, making local deployment of top-tier AI a reality.

FreeToken

Main features of FreeToken

  • On-device full-power inference: Supports full-power operation of Qwen3.6-35B, DeepSeek-V4-Flash 284B, and other ultra-large MoE models on consumer-grade single GPUs such as the RTX 4060/5090.
  • Full-layer double-buffered prefetching: While the GPU computes the current layer, the next layer's Expert weights are streamed and prefetched in the background via PCIe, completely eliminating I/O wait bubbles.
  • Bandwidth-adaptive scheduling: Probes PCIe bandwidth and CPU compute power in real time, dynamically offloading missed Experts to the GPU cache or computing them in place on the CPU, pushing hardware throughput to its limit.
  • Agent state reuse: Sets lightweight checkpoints at special Token boundaries, so when editing context, incremental Prefill starts only from the nearest anchor, avoiding recomputation of the full history.
  • Elastic VRAM hot scaling: When background applications seize VRAM, seamlessly shrinks the GPU Cache and hands computation over to the CPU, achieving smooth degradation with 0 downtime and no OOM.
  • Extreme speed and low latency: First Token latency reduced by 42–58%, Agent multi-turn interaction TTFT reduced by 65–80%, overall 2–4 times faster than Ollama.

Technical principles of FreeToken

  • Full-layer double-buffered prefetching:During the Prompt processing stage, FreeToken achieves complete overlap of computation and data movement. When the GPU is computing layer l, the Expert weights of layer l+1 have already been streamed via background PCIe prefetching, essentially eliminating I/O wait bubbles, so that the Prefill process is not severely blocked by pulling Experts layer by layer from system memory.
  • Bandwidth-adaptive hybrid scheduling:At runtime, the system probes the machine's actual PCIe bandwidth and instantaneous CPU compute power in real time, dynamically calculating the optimal offload ratio. Some Experts hit the GPU's LRU cache, while the missed portion is intelligently offloaded according to real-time bandwidth, with some transmitted to the GPU via PCIe and some computed in parallel in place on the CPU, pushing hardware throughput to the theoretical limit under that topology.
  • Agent-oriented intelligent state reuse:For scenarios where Coding Agents or tool-calling models frequently fine-tune context, FreeToken sets lightweight checkpoints at special Token boundaries. When context is edited, the system only needs to restore from the nearest valid anchor and incrementally Prefill the newly added suffix, without recomputing the entire context, greatly reducing redundant computation overhead in multi-turn tool calls and chain-of-thought iterations.
  • Elastic VRAM dynamic hot scaling:In response to the reality that personal computers often have other applications seizing VRAM, FreeToken supports dynamically hot-adjusting the size of the Expert Cache in the GPU without restarting the Serving service. When available VRAM drops sharply, it seamlessly shrinks the cache and hands more missed Experts over to CPU computation, ensuring smooth degradation of the inference service without triggering a CUDA Out of Memory crash.
Dig into GitHub

Follow WeChat and reply "open source" to joinAI Open Source Project Community

How to Use FreeToken

  • Desktop App Installation: Visit the FreeToken official website https://www.flashml.ai/ to download the Windows or Linux desktop app. After installation, you can start the inference service through the GUI interface.
  • Quick Command-Line Installation: Execute via the command line uv pip install "freetoken[accel]" to install the CLI tool with one click.
  • Model Loading and Automatic Scheduling: Load supported MoE model weights, and the system will automatically complete CPU-GPU heterogeneous scheduling and bandwidth-adaptive configuration.
  • Hardware Configuration Recommendations: When running ultra-large models such as DeepSeek-V4-Flash, it is recommended to equip 32GB of VRAM and pair it with 192GB or more of memory to ensure stable and smooth operation.
  • Elastic VRAM Auto-Management: If background games or rendering seize VRAM during use, FreeToken will automatically hot-scale up or down, with no manual intervention or service restart required.

FreeToken's Core Advantages

  • Break Through Hardware Barriers: A consumer-grade single GPU (such as RTX 4060/5090) can run ultra-large MoE models with 35B–753B parameters at full capacity, with no data center cluster required.
  • Leading Inference Speed: 2–4 times faster than Ollama, 1.46 times faster than llama.cpp; a laptop RTX 4060 running Qwen3.6-35B can reach 39.3 tok/s.
  • Extremely Low First Token Latency: Through full-layer double buffering and bandwidth-adaptive scheduling, it reduces the TTFT of long prompts by 42–58%.
  • Agent Interaction Optimization: Based on lightweight checkpoints at token boundaries and state reuse, TTFT in multi-turn tool-calling scenarios is reduced by 65–80%.
  • Elastic VRAM Without Crashes: When background applications seize VRAM, GPU Cache can be hot-scaled and computation transferred to the CPU, achieving smooth degradation with 0 downtime and no OOM.
  • Full-Stack Heterogeneous Collaboration: Unifies CPU, system memory, PCIe bus, and GPU into an elastic inference platform, automatically converging to the theoretical maximum throughput of the hardware topology.

FreeToken's Project Address

  • Project Official Website：https://www.flashml.ai/
  • GitHub Repository：https://github.com/FlashML-org/FreeToken
  • arXiv Technical Paper：https://arxiv.org/pdf/2608.16157

FreeToken's Comparison with Similar Competitors

Comparison Dimension FreeToken Ollama
Positioning Full-stack inference system dedicated to on-device MoE General-purpose local large model runtime framework
MoE Model Support Native deep optimization, supports full-capacity operation of 20+ MoE models Some MoE models lack support or cannot start (such as DSV4-Flash)
Inference Speed Faster than Ollama by 2–4 times(RTX 4060 running Qwen3.6-35B reaches 39.3 tok/s) Baseline speed, with obvious PCIe bandwidth bottlenecks
First Token Latency Reduces TTFT through double-buffered prefetching 42–58% Under long prompts, experts are fetched layer by layer, resulting in higher latency
Agent Multi-Turn Interaction State reuse reduces TTFT by 65–80% After context modification, recomputation is required, causing latency accumulation
VRAM Elasticity When background processes seize VRAM, it can hot-scale,0 downtime and no OOM Insufficient VRAM directly triggers a CUDA OOM crash
Hardware Collaboration Unified elastic scheduling of CPU + memory + PCIe + GPU Mainly depends on GPU, CPU offloading efficiency is limited
Installation method GUI App / CLI uv pip install "freetoken[accel]" ollama run One-command pull from the command line

FreeToken application scenarios

  • Local AI Coding Agent: Run coding agents such as Claude Code and OpenClaw locally on a laptop or gaming PC, supporting multi-turn tool calls and chain-of-thought iteration, with intelligent code completion and debugging possible without an internet connection.
  • On-device inference of ultra-large MoE models: Individual users can run frontier models such as DeepSeek-V4-Flash (284B) and GLM-5.2 (753B) at full capacity on consumer single cards like the RTX 4060/5090, breaking the dependence on data centers.
  • High-concurrency multi-turn dialogue service: In dialogue scenarios such as customer service and knowledge Q&A that require frequent context modification, Token boundary checkpoints and state reuse are used to avoid redundant computation and significantly reduce multi-turn response latency.
  • Elastic AI assistant for parallel gaming/work: When running 3D rendering, games, or development tools in the background, FreeToken can automatically shrink the GPU cache and hand computation over to the CPU, ensuring that the AI inference service is not interrupted or crashed.
  • Edge device research and prototype validation: Researchers can quickly deploy and test the latest open-source MoE models on ordinary workstations in the laboratory, without applying for server resources, lowering the barrier to large-model research.