NInfer is a C++/CUDA single-GPU inference engine written from scratch, with the goal of achieving high-performance inference for selected model checkpoints on a single NVIDIA GeForce RTX 5090. It supports Qwen3.5 Dense/MoE and Qwen4Exp (Qwen3.8-Flash-Next), and can handle text, image/video prompts, and native MTP. At runtime it uses one GPU, one resident model, and pins 1 to 8 execution channels at startup.
Main capabilities include: text generation (thinking and non-thinking modes), image/multi-image/video and mixed multimodal messages, chunked prefill, exact-batch CUDA Graph decoding, batch decoding with batch size limited at startup, MTP speculative decoding with a draft window of 1 to 5, BF16/INT8/FP8/NVFP4 and K8V4 KV storage, offline causal perplexity scoring, private and shared exact prefix reuse (Device/Host State and KV retention), model-aware sampling defaults and explicit sampler overrides, as well as OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages interfaces (including streaming, tools, local response state, token counting, and usage statistics). The 35B-A3B target also supports DFlash (draft window 1 to 15) and CPU/GPU MoE expert offloading.
Usage: requires x86-64 Linux, RTX 5090, a CUDA toolkit supporting sm_120a, CMake 3.28+, a C++20 compiler, Ninja, pkg-config, FFmpeg development libraries, hwloc>=2.5, liburing>=2.5, libcurl>=7.85. To build the product binary: cmake --preset release; the development preset cmake --preset dev enables tests and benchmarks. There is no install target or packaged binary, so it must be run from the source build tree. Official artifacts can be downloaded with the Hugging Face CLI, and the text/agent server or one-off CLI requests can be launched, or it can be run via a Docker image.
Limitations: supports only one RTX 5090 and one resident model; no priority/QoS, multi-GPU, or distributedserving; tool calls are returned to the client only and do not execute tools; in-tree C++ headers are not distributed as an installed SDK; requires v3 artifacts, and older v2 can be upgraded locally. The project is developed as a personal interest and is licensed under the Apache License 2.0.
