ds4

Purpose

ds4 (DwarfStar) aims to be the best way to run a small number of excellent large language models on consumer-grade hardware (that is, hardware ordinary people can actually own). To this end, the project builds a small native inference engine, prioritizing DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, text inference on CUDA), and additionally supporting GLM 5.2/5.3, GLM 5.3 Flash, DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA). The code is self-contained and deliberately kept narrow in scope; it is not a general-purpose GGUF runner: you must use GGUF files produced by the project itself.

Main capabilities

• Hardware support: Metal is the primary target, aimed at Macs with 96GB and above; smaller machines can use SSD streaming, and running very large models such as full GLM 5.x (non-Flash) on a 128GB system also requires SSD streaming. NVIDIA CUDA primarily targets DGX Spark, while also supporting multi-GPU systems that other backends do not support (for example, running DeepSeek V4 Flash on Ada Lovelace cards). ROCm supports Strix Halo systems, such as Framework Desktop.
• Running on consumer-grade hardware: Capable models can be run on devices such as MacBooks, DGX Spark, and Strix Halo; even with insufficient memory, SSD streaming allows them to run at a decent speed.
• Multi-GPU and multi-machine: Multiple CUDA cards can serve as a multi-user LLM server; a Flash configuration with eight L40S reaches an aggregate generation speed of about 126 t/s across 16 sessions. Two 128GB Macs connected via RDMA can run 4-bit DeepSeek Flash or GLM 5.3 Flash using tensor parallelism. Pipeline parallelism can also be used to combine multiple systems, accumulating memory to run larger models.
• Quantization and caching: DeepSeek V4 Flash/PRO and GLM 5.2 can tolerate aggressive routed expert quantization; compressed KV cache and fast local SSD make long contexts practical.

Usage

Choose a build version; the platform guides cover prerequisites, memory estimates, and hardware-specific setup. On a 96 or 128GB machine, the first run can download DeepSeek V4 Flash Q2; downloaded files are placed in the gguf/ directory. After building and downloading the model: the default model is ds4flash.gguf, specified explicitly via -m FILE ; commands are usually run from the repository root, and when starting from another location use --chdir /path/to/ds4. The server listens by default on http://127.0.0.1:8000. The interactive CLI preserves multi-turn conversations and can be used with /help、/read FILE、/ctx N、/quit; Ctrl+C interrupts generation and returns to the prompt; each binary can be used with --help to view the full options.

• Native coding agent:ds4-agent runs inference directly, without a separate HTTP server, keeping token history together with live model state, showing prefill progress, and using the model's native tool format. DeepSeek and GLM have their own templates. You can use /hints on to occasionally view explanations of coding choices. Sessions are saved in ~/.ds4/kvcache. For Pi, OpenCode, Codex CLI, or Claude Code, use ds4-server and follow the client setup guide.
• Models, images, and speculative decoding: DeepSeek Vision Experimental uses a different checkpoint from Flash 0731; GLM 5.3 Flash and Qwen3.8 Flash Next add vision capabilities to the same text model through a separate encoder. DeepSeek V4.1 Flash text and vision run on Metal, and text can also run on DGX Spark. With --vision FILE passing in the matching encoder, you can use /read image.png in the CLI or in the native agent view_image. The smaller Q2 version of Qwen3.8 has main weights of 41.73 GiB and is the entry-level option for a 64GB Mac; GGUF also includes a 95.37 GiB raw BF16 n-gram, read directly from disk rather than loaded into memory, and needs to be placed on a fast SSD, with an 8K context recommended to start. Speculative decoding is optional: GLM and Qwen use --mtp, and V4 Flash DSpark requires matching GGUF support.
• Output and power consumption: Thinking is enabled by default; use --nothink or /nothink to answer directly,--think or /think to re-enable. In V4.1, --think-level 25 or /think 25is also accepted (1 to 100 sets reasoning effort, 0 disables it). Regular sampling defaults to temperature 1, top-p 1, min-p 0.05, and--temp 0 selects greedy output. In DeepSeek V4, --power N trades throughput for lower sustained GPU load, default 100; V4.1 and GLM currently require --power 100. DeepSeek V4 Flash and GLM 5.3 Flash also support direction guidance, using --dir-steering-file FILE to load vectors.--prefix-file FILE can preload complete USER:/ASSISTANT: conversation pairs before a live conversation.
• Capability evaluation:ds4-eval runs embedded capability regression tests against real GGUF, which is a DwarfStar integration check rather than an official leaderboard score. The default suite is core，--suite all runs the core and hard test cases,--list-cases lists the tests,--plain selects non-interactive output, and--regrade-trace FILE scores existing trajectories.

Applicable Scenarios and Limitations

ds4 is suitable for users who want to run a small number of frontier open-source models locally on their own consumer hardware, for multi-GPU CUDA environments that need a multi-user LLM server, and for scenarios using two Macs or Spark to do tensor parallelism over RDMA. Its model support is deliberately opportunistic; the project follows the best open-source weights suited to local machine sizes (especially 128GB laptops and 256/512GB workstations), and a model may be removed when a better alternative appears. Key points to note: the project is still in a rapidly changing stage and is beta quality; although a large QA run is performed before each release, instability and regressions can still occur. In addition, the software has relied heavily on AI coding agents during development, with humans leading the ideas, testing, and debugging; if you do not accept AI-developed code, it is not suitable for you. The project does not link against GGML, but its existence benefits from the path opened by llama.cpp and GGML, and some source-level content (GGUF quantization layouts and tables, CPU quantization/dot-product logic, and some kernels) is retained or adapted from that project under the MIT license. The software can be modified and customized for its use, and the author encourages users with coding agents to use the agent as an interface for discovering the project, making changes, and creating personalized configurations.

Detailed Guides

The project provides detailed guides covering the following topics: models and vision (Flash, PRO, GLM, Qwen, and matching encoders), Qwen3.8 Flash Next (model settings, MTP, vision, and validation), SSD streaming (running models larger than memory and setting cache size), cross-machine inference (dual-Mac TP/RDMA and layer pipelining), speculative decoding (DSpark, GLM, and Qwen MTP and sampling), serving (API, images, batching, and disk KV cache), coding agent clients (Pi, OpenCode, Codex CLI, and Claude Code), performance (reproducible measurement and recorded baselines), and testing and development (regression tests, debugging, and model building tools). Read CONTRIBUTING.md before sending a pull request.