hands-on-llms is an open-source course designed to teach knowledge related to LLMs, LLMOps, and vector databases by building a real-time financial advisor LLM system. The course provides source code, video lectures, and reading materials, and is completely free (code and videos); the Medium articles require payment.
Purpose
This course is suitable for developers who want to learn how to design, train, and deploy production-grade LLM systems. Through a three-pipeline architecture, learners will master the complete process from data ingestion and model fine-tuning to inference serving.
Main Capabilities
The course covers the following core components:
Training Pipeline
- Load a proprietary question-answering dataset
- Fine-tune an open-source LLM using QLoRA
- Log training experiments and inference results in Comet ML's experiment tracker and LLMOps dashboard
- Store the best model in Comet ML's model registry
- Deploy the training pipeline using Beam as serverless GPU infrastructure
- Located in the modules/training_pipeline directory
- VRAM requirement: 10 GiB (must support CUDA-enabled Nvidia GPUs)
Real-Time Feature Pipeline
- Ingest financial news from Alpaca
- Use Bytewax to clean and transform news documents into embeddings in real time
- Store the embeddings in the Qdrant vector database
- Automatically deploy to an AWS EC2 machine through a CI/CD pipeline built with GitHub Actions
- Located in the modules/streaming_pipeline directory
Inference Pipeline
- Use LangChain to create a chain:
- Download the fine-tuned model from the Comet model registry
- Receive user questions as input
- Query the Qdrant vector database and augment the prompt with relevant financial news
- Use the initial query, vector database context, and chat history to call the fine-tuned LLM to obtain financial advice
- Persist the chat history to memory
- Log prompts and answers to Comet ML's LLMOps monitoring feature
- Deploy as a RESTful API using Beam as serverless GPU infrastructure, wrapped in a Gradio UI for demonstration
- Located in the modules/financial_bot directory
- VRAM requirement: 8 GiB (must support CUDA's Nvidia GPU)
Financial Q&A Dataset
Use GPT-3.5 to generate a financial Q&A dataset for fine-tuning open-source LLMs so that they specifically use financial terminology and answer financial questions. This method of using a large LLM to generate a dataset to train a small LLM is called distillation fine-tuning.
How to Use
- Set up external services:
- Alpaca: Create a free account and generate an API key (1 data connection free)
- Qdrant: Create a free account and generate an API key (use the free tier)
- Comet ML: Create a free account and generate an API key and project (use the free tier)
- Beam: Create a free account, install the CLI, and configure credentials (about 10 hours of free compute time, then pay-as-you-go; if you have an Nvidia GPU with >8 GB VRAM, you can skip this)
- AWS: Create an account and generate credentials, install AWS CLI v2.11.22 and configure it (pay-as-you-go, only deploy a t2.small EC2 VM, about $0.023/hour; if you do not want to deploy the feature pipeline, you can skip this)
- Clone the repository and install and use it according to the README of each module:
- q_and_a_dataset_generator
- training_pipeline
- streaming_pipeline
- inference_pipeline
- Refer to the video lectures and articles for learning.
Applicable Scenarios
- Learning LLM fine-tuning and deployment
- Understanding LLMOps practices
- Building RAG applications
- Real-time data processing and vector database integration
- Financial domain LLM application development
Limitations
- The training pipeline requires 10 GiB of VRAM, and the inference pipeline requires 8 GiB of VRAM (must support CUDA Nvidia GPU), but this can be bypassed through Beam serverless infrastructure
- External services have free quota limits, and payment is required after exceeding them
- Medium articles require a paid subscription ($5/month)
- The course has been archived; it is recommended to check out the new LLM Twin open-source course
- The code and videos are free, but Medium articles require payment
License
The course is an open-source project under the MIT license, and you can clone or fork it for any purpose as long as you distribute the license and acknowledge the work.acknowledge the work.
