Purpose
Chat with MLX is an all-in-one large language model chat console based on the Apple MLX framework, aimed at Apple Silicon Mac users. Its goal is to let users converse securely with local models and their own data in a privacy-enhanced way, while also serving as a comprehensive chat interface for MLX.
Main Capabilities
- Privacy-enhanced AI: Converse securely and locally with commonly used models and data.
- MLX chat console: Provides an all-in-one LLM chat UI for Apple MLX.
- Easy integration: Can conveniently integrate any HuggingFace and MLX-compatible open-source model.
- Default models: Built in are Llama-3, Phi-3, Yi, Qwen, Mistral, Codestral, Mixtral, StableLM, as well as Dolphin and Hermes variants.
- Dataset mode support: You need to select Document or YouTube dataset mode for it to work, combining LangChain and ChromaDB to implement RAG retrieval-augmented conversation.
How to Use
Installation and startup:
pip install chat-with-mlx
After manual Pip installation, start the application:
chat-with-mlx
Usage notes (from the project description):
- When the model is downloaded via option 1, the only way to stop the download is to press control + C in the terminal at the same time.
- If you want to switch files, you must manually click STOP INDEXING, otherwise the vector database will add the second document to the current database.
- You must select a dataset mode (Document or YouTube) for it to work.
- Phi-3-small cannot perform streaming output in completion.
Underlying Framework MLX
MLX is an array framework for machine learning research on Apple chips, introduced by Apple's machine learning research team. Its key features include:
- Familiar API: MLX's Python API is highly similar to NumPy, while also providing fully functional C++, C, and Swift APIs that remain consistent with the Python API; it also provides high-level packages such as mlx.nn and mlx.optimizers, whose APIs are close to PyTorch, making it easier to build more complex models.
- Composable function transformations: Supports composable function transformations for automatic differentiation, automatic vectorization, and computational graph optimization.
- Lazy computation: Computation in MLX is lazy, and arrays are only actually materialized when needed.
- Dynamic graph construction: The computational graph is constructed dynamically, and changing the shapes of function arguments does not trigger slow compilation, making debugging simple and intuitive.
- Multiple devices: Operations can run on any supported device (currently CPU and GPU).
- Unified memory: A notable difference from other frameworks is the unified memory model. Arrays in MLX reside in shared memory, and operations can be executed on any supported device type without transferring data.
Applicable Scenarios
Suitable for users who use an Apple Silicon Mac, want to have privacy-preserving conversations locally with open-source large language models and their own document data (Document or YouTube dataset mode), and need MLX framework compatibility.
Limitations
- Platform limitation: Based on Apple MLX, only applicable to Apple Silicon Mac.
- Phi-3-small does not support streaming output in completion.
- When switching files, you must manually click STOP INDEXING, otherwise the vector database will continue to append documents.
- You must select a dataset mode (Document or YouTube) for it to work properly.
- During model downloadit can only be stopped by pressing control + C in the terminal.
- Other specific information such as fees, pricing, platform support, and performance is not yet provided.
