uqlm

UQLM is a Python library for large language model (LLM) hallucination detection, based on uncertainty quantification techniques. It is aimed at developers and researchers who need to assess the reliability of LLM outputs, and provides a set of response-level confidence scorers, each of which returns a confidence score between 0 and 1, where a higher score indicates a lower likelihood that the output is erroneous or hallucinated. The latest version can be installed from PyPI.

[Main Capabilities] UQLM divides scorers into several types:

1. Black-box scorers (consistency-based): assess uncertainty by measuring the consistency among multiple generations for the same prompt. Compatible with any LLM, with no need to access the model's internal state or token probabilities. Available scorers include: discrete semantic entropy, semantic set count, non-contradiction probability, entailment probability, exact match, BERTScore, cosine similarity, and code-adapted scorers provided through CodeGenUQ.
2. White-box scorers (token probability-based): use token probabilities to estimate uncertainty, support single-generation scoring, and are faster and cheaper than black-box methods, but require access to the LLM's internal probabilities, so they are not compatible with all LLMs/APIs. Single-generation scorers include: minimum token probability, length-normalized sequence probability, sequence probability, average Top-K token negative entropy, minimum Top-K token negative entropy, probability margin. Self-reflection scorer: P(True). Multi-generation scorers: Monte Carlo sequence probability, consistency confidence (CoCoA), semantic entropy, semantic density.
3. LLM-as-a-Judge scorers: use one or more LLMs to assess the reliability of the original LLM response, and can be highly customized through prompt engineering and the choice of judging LLM. Includes: categorical LLM judge, continuous LLM judge, Likert scale LLM judge, LLM judge panel (Panel).
4. Ensemble scorers: combine multiple individual scorers through weighted averaging to produce a more robust uncertainty estimate. They can be used directly with fixed weights (unsupervised), or trained to achieve optimal performance (supervised), such as BS Detector and supervised UQ ensembles.
5. Long-text scorers (claim-level): score in a fine-grained manner at the claim or sentence level. It samples multiple responses to the same prompt, decomposes the original response into claims or sentences, and evaluates the consistency of each original claim/sentence with the sampled responses. After scoring, those with confidence below a specified threshold can be removedand reconstruct the response, thereby improving the factual precision of long-text generation. Includes the LUQ scorer, graph-based scorer, and generalized long-text semantic entropy.
6. Code generation UQ: Provides a dedicated CodeGenUQ interface for predicting whether LLM-generated code is functionally correct without executing the code. Includes a white-box scorer, a black-box scorer for code adaptation based on functional equivalence, and a reflective self-evaluation scorer; the white-box method is the same as the token-probability-based scorer in WhiteBoxUQ.

[Usage] UQLM provides classes such as BlackBoxUQ, WhiteBoxUQ, LLMPanel, UQEnsemble, LongTextUQ, and CodeGenUQ. In the examples, any LangChain Chat Model such as ChatOpenAI, ChatVertexAI, ChatOllama, and AzureChatOpenAI can be used. For example, use_best=True in BlackBoxUQ enables mitigation measures, selecting the response with the least uncertainty. In long-text scoring, response and entailment reflect the original response and its response-level confidence, refined_response and refined_entailment are the corresponding values after response refinement, and the claims_data column contains fine-grained data for each response (claims, claim-level confidence, and whether they are retained during refinement).

[Examples and Documentation] UQLM comes with complete example notebooks and tutorials, covering: black-box and white-box uncertainty quantification, single-generation and multi-generation methods, LLM-as-a-Judge techniques, ensemble methods, cutting-edge techniques such as semantic entropy and semantic density, multimodal uncertainty quantification, and score calibration. Each notebook contains detailed explanations and adaptable code examples. The official documentation site provides API references and detailed usage instructions.

[Applicable Scenarios] Suitable for hallucination detection on LLM outputs, providing confidence scores for generated results, claim-level factual refinement in long-text generation, predicting functional correctness in code generation, and building customizable multi-judge ensemble evaluation pipelines.

[Limitations and Dependencies] Black-box scorers require multiple generations, resulting in higher cost and latency (the same applies to multi-generation white-box scorers); white-box scorers require access to model internal probabilities and are not compatible with all LLMs/APIs; some scorers require multiple additional generations. For specific installation commands, the complete API, demonstrations, and paper citation information, refer to the official documentationand repository; some details are not yet provided.