Mamba-Chat is the first chat language model based on the state space model (SSM) architecture rather than Transformer. This model builds on the work of Albert Gu and Tri Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (paper) and its model implementation. This repository provides model training/fine-tuning code modified based on the Huggingface Trainer class.
Purpose
Mamba-Chat is intended to explore chat language models with non-Transformer architectures. It is based on the Mamba-2.8B model and was fine-tuned on 16,000 samples from the HuggingFaceH4/ultrachat_200k dataset, and can be used for dialogue generation tasks.
Main capabilities
- A chat language model based on the state space model architecture.
- Provides training and fine-tuning code.
- Includes two interaction methods: a CLI chatbot and a Gradio app.
- Supports fine-tuning on specific hardware (such as a 24GB GPU).
Usage
- Clone the repository and install dependencies.
- Run the CLI chatbot or Gradio app to converse with the model.
- Use the provided code to fine-tune the Mamba base model on a subset of the Ultrachat dataset.
Applicable scenarios
- Studying the application of state space models in dialogue tasks.
- Developers trying chat models with non-Transformer architectures.
- Conducting model fine-tuning experiments under limited hardware resources.
Limitations
- The model is based on Mamba-2.8B and is relatively large, which may require a higher hardware configuration.
- Fine-tuning requires a GPU with at least 24GB of VRAM.
- Specific performance and results are not yet provided.
