SuperAPI AI Directory

evaluation

General & other

View repository

Updated 10-16 05:48

About this skill

📊 Evaluation Methods for Agent Systems

This skill allows you to systematically evaluate the performance of autonomous agents. Unlike traditional software, agents are non-deterministic and can reach a goal through multiple valid paths. Evaluation should focus on the outcomes (Outcomes) and the efficiency of the process.

Mental Model: Evaluation is not binary (pass/fail); it is a multidimensional measurement of the probability of success and the quality of reasoning.

🚩 Fragility and Autonomy

  • Fragility: Low. These principles are robust and fundamental.
  • Freedom: High in the design of rubrics specific to each project.

🚀 When to Activate

  • When building test frameworks for agents.
  • To validate whether a change in context engineering improves or worsens performance.
  • When comparing different models (e.g. Flash vs Pro) for a specific task.
  • To establish "Quality Gates" in production pipelines.

🧠 Core Concepts

1. The Three Pillars of Performance (95% of Variance)

Studies show that three factors explain almost all of an agent's performance:

  1. Token Budget (80%): More tokens (within the signal limit) usually allow for deeper exploration.
  2. Tool Calls (10%): The ability to interact with and observe the environment is critical.
  3. Model Choice (5%): Superior models (e.g. Gemini 3 Ultra) multiply the efficiency of the other two factors.

2. Evaluation Challenges

  • Non-Determinism: Two identical executions can produce different but equally valid paths.
  • Context-dependent failures: An agent may fail only when the context is very long or noisy.
  • Composite Dimensions: Quality includes factual accuracy, tone, brevity, and tool efficiency.

🛠️ Implementation in Gemini Elite Core

Pattern: Multidimensional Rubric

Define criteria with specific weights according to their importance to the project.

Dimension Weight Description
Factual Accuracy 0.35 The facts match the source of truth.
Completeness 0.25 Covers all requested aspects.
Efficiency 0.20 Uses the minimum number of tools necessary.
Format 0.20 Follows the expected JSON/Markdown schema.

Pattern: Complexity Stratification

Divide your test set into levels:

  • Simple: A single tool call, factual search.
  • Medium: Multiple tools, comparison logic.
  • Complex: Multi-step reasoning, significant ambiguity.
  • Strategic: Extended interaction, deep data synthesis.

🔗 Integrations

  • context-fundamentals: Evaluation of how the model uses the provided context.
  • context-degradation: Detection of performance drops in long contexts.
  • advanced-evaluation: Technical implementation of "LLM-as-a-Judge".

📚 Internal References

  • references/metrics.md: Detailed definitions of metrics, implementation of rubrics and evaluation runners.