This skill allows implementing production-grade evaluation systems using LLMs as judges. It covers from the evaluation taxonomy to the mitigation of systematic biases (position, length, self-enhancement).
Mental Model: Evaluation is not an event, it is a pipeline. "LLM-as-a-Judge" is not a single technique, but a decision framework based on the nature of the data (objective vs. subjective).
🚩 Fragility and Autonomy
- Fragility: Medium. The definition of rubrics requires alignment with the business/project objectives.
- Freedom: High in the technical implementation of the validation pipelines.
🚀 When to Activate
- When building automatic evaluation pipelines for LLM outputs.
- To compare responses from multiple models (A/B testing of prompts).
- When establishing consistent quality standards in development teams.
- To debug evaluation systems with inconsistent results.
🧠 Core Concepts
1. Evaluation Taxonomy
| Approach | Best for... | Reliability | Common Failure |
|---|---|---|---|
| Direct Scoring | Objective Criteria (Factuality, Format) | High | Scale drift (calibration drift) |
| Pairwise Comparison | Subjective Criteria (Tone, Style, Creativity) | Very High | Position Bias |
2. The Landscape of Biases (Mitigation)
- Position Bias: Tendency to prefer the first response.
- Mitigation: Evaluate twice by swapping positions (Swap) and verify consistency.
- Length Bias: Preference for long responses.
- Mitigation: Explicit instructions to ignore length and penalize irrelevant verbosity.
- Self-Enhancement: Models that score their own outputs higher.
- Mitigation: Use different models for generation and evaluation.
🛠️ Implementation in Gemini Elite Core
Pattern: Direct Scoring (Gemini 3 + Thinking)
Use thinking-pro to reason about the evidence before issuing the score.
## Task
Evalúa la calidad de la respuesta según los criterios.
## Criterios
- Precisión Factual (Peso: 1.0)
- Concisión (Peso: 0.5)
## Instrucciones
1. Busca evidencia específica en la respuesta.
2. Razona sobre cómo la evidencia cumple o no el criterio.
3. Asigna un score (1-5).
4. Sugiere una mejora atómica.
## Formato de Salida (JSON)
{ "score": number, "reasoning": string, "improvement": string }
Pattern: Pairwise with Swap
It is mandatory to perform the "Swap Test" to ensure the integrity of the evaluation.
- Pass 1: Model evaluates [Response A, Response B].
- Pass 2: Model evaluates [Response B, Response A].
- Consistency: If the winner changes, it is marked as a TIE with low confidence.
📏 Rubric Generation
A well-defined rubric reduces variance by 40-60%.
- Level 1 (Poor): No evidence, logically inconsistent.
- Level 3 (Adequate): Meets the instruction, correct tone, logically passable.
- Level 5 (Excellent): Exceeds expectations, deep insights, perfect format.
🔗 Integrations
context-fundamentals: Structure of evaluation prompts.tool-design: Creation of programmatic validation tools.debug-master: Identification of systemic failures in the model's logic.
📚 Internal References
references/bias-mitigation.md: Advanced Swap and PoLL (Panel of LLMs) techniques.references/implementation-patterns.md: Technical patterns of "Reasoning-First" and "Swap Test".references/metrics-guide.md: Selection of metrics (Factuality, Tone, Format).