voice-lab

Voice Lab is a testing and evaluation framework for voice agents. It aims to simplify the process of evaluating and iterating on LLM-driven agents, enabling developers to make data-driven decisions, such as optimizing costs by switching to smaller models, testing newly released models, or fine-tuning prompts to improve performance. Although the framework is optimized for voice agents, it is equally suitable for any LLM-based agent evaluation needs.

Key Capabilities

• Custom metrics: Define metrics in JSON format and score them using LLM-as-a-Judge.
• Performance tracking: Track performance metrics across different configurations.
• Cost optimization: Monitor and intelligently select the most cost-effective models.
• Model switching: Confidently switch between models (such as from Claude Sonnet to GPT-4, or from GPT-4 to GPT-4 Mini).
• Model evaluation: Evaluate smaller, more efficient models to achieve a better cost-latency balance.
• Comparison tables: Generate comprehensive comparison tables across different models.
• Prompt variants: Systematically test multiple prompt variants.
• User simulation: Simulate and validate performance across different user types and interaction styles.

Usage

1. Clone the repository: git clone https://github.com/saharmor/voice-lab.git cd voice-lab
2. Create a virtual environment and install dependencies: python3 -m venv venv source venv/bin/activate pip install -r requirements.txt
3. Create a .env file in the project root directory and add environment variables: OPENAI_API_KEY=your_openai_api_key
4. Currently, the library only supports the text portion of voice agents, that is, testing the underlying language model and prompts. Run the example test example_test.py to execute predefined tests.
5. For more advanced configuration, you can use the Voice Lab Configuration Editor to generate a JSON configuration file.

Adding New Test Scenarios

Test scenarios can be generated through the Voice Lab Configuration Editor, or by directly editing test_details.json. This file is located in the llm_testing directory. When adding new entries, refer to the provided template, which includes fields such as system_prompt, initial_message, tool_calls, success_criteria, and persona.

Use Cases

• Need to systematically evaluate the performance of voice agents under different language models, prompts, and personas.
• Want to reduce costs and evaluate the effectiveness of smaller models on specific tasks.
• Need to test multiple prompt variants and compare performance.
• Want to simulate different user types and interaction styles to verify the robustness of the agent.
• Need to generate cross-model comparison tables to assist with model selection decisions.

Limitations

• Currently only supports the text portion of voice agents, that is, testing the underlying language model and prompts; it does not yet support voice analysis (such as interruptions, pauses, etc.).
• Requires configuring OPENAI_API_KEY.
• Some features mentioned in the contribution ideas (such as supporting more language models through LiteLLM, batch processing, parallel test execution, detailed test reports, conversation replay, etc.) have not yet been implemented and are part of future plans.

Contribution and Attribution

The project provides multiple contribution directions, such as supporting providing additional context to agents via JSON (credit card information, price ranges, etc.), dynamic metrics, voice analysis, integrating more language models, integrating Tencent's 1B Personas, makingBroader simulations with Microsoft's TinyTroupe, integrating Qwen2-Audio for audio analysis, batch processing to reduce costs, recommending fine-tuning models to better evaluate adherence/style, etc., improving the testing framework (building an agent and persona database, parallel test execution, detailed test reports, conversation replay), etc. If you use this project, please provide attribution by linking back to the repository https://github.com/saharmor/voice-lab .