Turbopilot is an open-source, large language model-based code completion engine that can run locally on a CPU. It is designed as a self-hosted Copilot clone, using the library behind llama.cpp to run the 6-billion-parameter Salesforce Codegen model with only 4GiB of memory. The project is heavily inspired by fauxpilot.
Purpose The main purpose of Turbopilot is to provide code autocompletion locally without relying on cloud services or a GPU. It is suitable for developers who want a GitHub Copilot-like experience in a local environment, especially for users with limited memory (4-8GiB) or without a dedicated GPU.
Main Capabilities
- Supports multiple models: including the Salesforce Codegen series, StableCode 3B Instruct, Wizardcoder, Starcoder, Santacoder, etc.
- Local operation: runs entirely on CPU, no GPU required.
- Memory optimization: through quantized models, a 6-billion-parameter model can run in 4GiB of memory.
- Multi-platform support: provides Docker images and precompiled binaries, supporting Linux, Windows, etc.
- CUDA support: supports CUDA inference starting from v0.0.5, and can be accelerated via GPU.
- API compatibility: the API is roughly compatible with OpenAI and can be used with the fauxpilot plugin for VS Code.
- Multi-model support: supports multiple model types, which can be specified via the -m parameter.
How to Use
- Obtain a model: pre-converted, pre-quantized models can be downloaded from Huggingface. StableCode is recommended for low-memory users (4-8GiB), and WizardCoder is recommended for high-configuration users (16+GiB memory, dedicated GPU, or Apple Silicon).
- Run the server: download the latest binary and extract it to the project root directory, or use the Docker image. The server listens on port 18080 by default and can be changed via -p. On multi-core systems, -t can be used to control the number of CPUs.
- Use the API: install the vscode-fauxpilot plugin, configure it in settings.json, and then enable Fauxpilot. The plugin will send API requests to the local server. API endpoints:http://localhost:18080/v1/engines/codegen/completions。
- Docker operation: the model needs to be downloaded separately, then run the Docker container. The CUDA version requires nvidia-docker, and --gpus=all must be passed. GPU_LAYERS can be used to set full offloading of computation to the GPU.
Suitable Scenarios
- Code autocompletion in a local development environment.
- Machines without a GPU or with limited memory.
- Scenarios requiring self-hosting and data privacy protection.
- Experimenting with and exploring local code completion models.
Limitations
- The project has been archived (September 30, 2023) and is no longer maintained.
- The current version's autocompletion speed is relatively slow; it is a proof of concept, not a stable tool.
- Only single-GPU devices are supported; multiple devices are not supported.
- Older Codegen models need to be requantized.
- Official VS Code Copilot plugin support is still under development.
Acknowledgments The project is based on GGML and llama.cpp, inspired by fauxpilot, uses the vscode-fauxpilot plugin for the frontend, uses the Salesforce Codegen model, and uses CrowCPP for the model server.
