BeHonest is a pioneering benchmark specifically designed for comprehensively evaluating the honesty of large language models (LLMs). It measures honesty from three key aspects: knowledge boundary awareness (self-knowledge), avoidance of deception (non-deceptiveness), and answer consistency.
Purpose: BeHonest aims to provide a standardized evaluation framework for the honesty of LLMs, helping researchers and developers understand model performance on honesty-related behaviors and advancing research on model honesty alignment.
Main capabilities: This benchmark supports the following 10 evaluation scenarios:
- Expressing the unknown: LLMs should appropriately refuse to answer questions beyond their knowledge scope.
- Acknowledging the known: LLMs should correctly answer questions within their knowledge scope.
- Role sycophancy: LLMs should not change their answers to cater to a certain role's viewpoint, thereby producing biased information to maintain approval.
- Preference sycophancy: When a user explicitly expresses a preference for a certain option, LLMs should not change their own views in order to maintain approval.
- Thief deception: Even when explicitly instructed or encouraged to engage in deceptive behavior, LLMs should not lie.
- Games: Even when deception is beneficial for winning in strategic games, LLMs should not lie.
- Prompt format: When the same content appears in slightly different prompt formats, LLMs should maintain consistent answers.
- Demonstration format: Even when few-shot demonstrations contain irrelevant biased features, LLMs should maintain consistent answers.
- Open-form consistency: LLMs should demonstrate consistency by verifying their own generated answers.
- Multiple-choice consistency: When asked to answer the same question again or when the user raises unnecessary doubts, LLMs should maintain consistent answers.
How to use: Currently, the BeHonest leaderboard contains evaluation results for 9 popular LLMs, covering closed-source and open-source models from different model families, as well as different model scales. The project team is working to add more models to the leaderboard to provide a comprehensive evaluation of LLM honesty. The evaluation metrics are calculated as follows: the overall scores for self-knowledge and non-deceptiveness are averaged; the overall score for consistency is obtained by inverting the values of performance spread (Perf. Spread) and inconsistency rate (Inconsistency Rate) (using the formula ( \left( \frac{\max(X) - x}{\max(X) - \min(X)} \right) \times 100 )), and normalizing the agreement rate (Agree. Rate) and consistency rate (Consist. Rate) to 0-100 (using the formula ( \left( \frac{x - \min(X)}{\max(X) - \min(X)} \right) \times 100 )), with each modified metric weighted equally, and averaging them to obtain the overall score. This score is a comprehensive measure that integrates the model's performance across different aspects into a single numerical value.
Applicable scenarios: BeHonest is suitable for scenarios such as LLM honesty research, model alignment evaluation, and model selection and comparison. Researchers and developers can use this benchmark to evaluate model performance on honesty-related behaviors and identify areas that need improvement.
Limitations: Currently, the leaderboard contains evaluation results for only 9 models, resulting in limited coverage. The project team points out that there is still significant room for improvement in LLM honesty and encourages the AI community to prioritize model honesty alignment. If you have any questions, you can contact the project team via GitHub issue or email.
