RouteLLM

Purpose

RouteLLM is a framework for serving and evaluating large language model (LLM) routers. It aims to address the cost-quality trade-off in LLM deployment: route complex queries to stronger but more expensive models, and simple queries to cheaper, less capable models, thereby significantly reducing costs while maintaining response quality.

Main Capabilities

• Cost Savings: The provided pre-trained routers can reduce costs by up to 85% on common benchmarks such as MT Bench while maintaining 95% of GPT-4 performance.
• Performance Comparison: Benchmark tests show that these routers can achieve the same performance as commercial solutions while reducing costs by more than 40%.
• Out-of-the-box Routers: Supports 4 routers trained on the gpt-4-1106-preview and mixtral-8x7b-instruct-v0.1 model pair: mf (recommended), sw_ranking, bert, causal_llm, as well as random routing random.
• Easy to Extend: New routers can be easily added and their performance compared across multiple benchmarks.
• OpenAI Compatible: Can be used as a drop-in replacement for the OpenAI client, or launch an OpenAI-compatible server.
• Model Support: Supports a variety of open-source and closed-source models through LiteLLM, and can also route to local models.
• Threshold Calibration: Each routed request has a cost threshold that controls the trade-off between cost and quality, and supports calibrating the threshold based on the Chatbot Arena dataset.
• Evaluation Framework: Includes an evaluation framework that can measure the performance of different routing strategies on benchmarks, currently supporting the mmlu, gsm8k, and mt-bench benchmarks.

Usage

1. Replace the OpenAI client: Initialize the RouteLLM controller, specify the strong model and weak model (defaulting to gpt-4-1106-preview and anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1), and set the API key. When making generation requests, specify the router and threshold through the model field, in the format router-[ROUTER NAME]-[THRESHOLD].
2. Launch an OpenAI-compatible server: Use --routers to specify the list of available routers, and--config to specify the configuration file path. The client specifies the router and threshold through the model field.
3. Threshold Calibration: Calibrate based on incoming query samples and the desired percentage of routing to the strong model. By default, calibration based on the Chatbot Arena dataset is supported.
4. Evaluate Routers: Use the evaluation command, with --routers to specify the router to evaluate, and--benchmark to specify the benchmark. Results will be output to the console and a performance plot will be generated.
5. Train a New Router: Implement the calculate_strong_win_rate method in the abstract Router class, and add the new router to the ROUTER_CLS dictionary.

Suitable Scenarios

• Scenarios where LLMs need to be deployed but costs need to be controlled.
• Applications where query complexity is unevenly distributed and simple queries make up a large proportion.
• R&D scenarios where you want to compare the effects of different routing strategies.
• Scenarios where the strong and weak model pair needs to be changed without retraining.

Limitations

• For the mf and sw_ranking routers, OPENAI_API_KEY is still currently required when generating embeddings.
• Threshold calibration is based on existing datasets. The actual percentage routed to each model will vary depending on the actual queries, so it is recommended to calibrate using a dataset similar to the incoming queries.
• By default, evaluation uses GPT-4 and Mixtral as the model pair. To change the model pair, use the --strong-model and --weak-model flags.
• The specific steps for local model support are not provided, but the relevant guide can be consulted.