mixseek-evaluator-config

MixSeek Evaluation Configuration Generation

Overview

Generates the MixSeek-Core evaluation configuration file (evaluator.toml) and judgment configuration file (judgment.toml). It defines the evaluation criteria, scoring method, and final judgment logic for Submissions in the TUMIX tournament.

Prerequisites

• The workspace must be initialized (seemixseek-workspace-init)
• The environment variable MIXSEEK_WORKSPACE must be set (recommended)

Generated Files

File	Purpose	Location
evaluator.toml	Scoring settings for Submissions	configs/evaluators/
judgment.toml	Settings for final judgment	configs/judgment/

Usage

Step 1: Hearing Requirements

Confirm the following with the user:

1. Evaluation focus: What to prioritize in evaluation (clarity, coverage, relevance, etc.)
2. Weighting: The importance of each metric (uniform or custom)
3. Judgment style: Deterministic (temperature=0) or diversity-oriented

Step 2: Proposing Metric Settings

Select from the standard metrics:

Metrics	Description	Use Case
ClarityCoherence	Clarity and consistency	Tasks focused on readability
Coverage	Coverage	Tasks focused on comprehensiveness
LLMPlain	General-purpose LLM evaluation	Tasks requiring custom evaluation criteria
Relevance	Relevance	Tasks focused on accuracy

Step 3: Generating Configuration Files

evaluator.toml:

default_model = "google-gla:gemini-2.5-pro"
temperature = 0.0

[[metrics]]
name = "ClarityCoherence"
weight = 0.34

[[metrics]]
name = "Coverage"
weight = 0.33

[[metrics]]
name = "Relevance"
weight = 0.33

judgment.toml:

model = "google-gla:gemini-2.5-pro"
temperature = 0.0
timeout_seconds = 60

Step 4: Saving Files

$MIXSEEK_WORKSPACE/configs/evaluators/evaluator.toml
$MIXSEEK_WORKSPACE/configs/judgment/judgment.toml

Important: Custom path (configs/evaluators/やconfigs/judgment/), you must alwaysorchestrator.tomlexplicitly specify the path withconfigs/evaluator.toml、configs/judgment.toml). If not specified, the default path (

# orchestrator.toml
[orchestrator]
evaluator_config = "configs/evaluators/evaluator.toml"
judgment_config = "configs/judgment/judgment.toml"

Step 5: Validating Configuration Files (Required)

Always run validation after generation.

# Evaluator設定の検証
uv run python skills/mixseek-config-validate/scripts/validate-config.py \
    $MIXSEEK_WORKSPACE/configs/evaluators/evaluator.toml --type evaluator

# Judgment設定の検証
uv run python skills/mixseek-config-validate/scripts/validate-config.py \
    $MIXSEEK_WORKSPACE/configs/judgment/judgment.toml --type judgment

If validation succeeds, report the results to the user. If it fails, check the error details and fix the configuration.

Standard Metric Details

ClarityCoherence (Clarity and Consistency)

Evaluates the readability and logical consistency of the response.

Evaluation Aspects:

• Clarity of sentence structure
• Logical flow
• Appropriate use of technical terms
• Clarity of conclusions

Recommended Use Cases:

• Generation of explanatory text
• Report writing
• Educational content

Coverage

Evaluates the comprehensiveness of responses to questions.

Evaluation Aspects:

• Coverage of all aspects of the question
• Inclusion of related topics
• Sufficiency of examples
• Presence of supplementary information

Recommended uses:

• Research task
• FAQ creation
• Technical documentation

Relevance

Evaluates how accurately the answer addresses the question.

Evaluation criteria:

• Direct answer to the question
• Elimination of unnecessary information
• Maintaining focus
• Fit with the context

Recommended uses:

• Q&A
• Customer support
• Evaluation of search results

LLMPlain (general-purpose LLM evaluation)

system_instructionThe LLM evaluates based on the custom evaluation criteria defined in

Features:

• Has no predefined evaluation logic
• system_instructioncan be fully customized with
• Used when special evaluation criteria are needed

Recommended uses:

• Domain-specific evaluation (legal, medical, etc.)
• Project-specific quality standards
• Aspects that cannot be covered by other metrics

Configuration example:

[[metrics]]
name = "LLMPlain"
weight = 0.5
system_instruction = """
セキュリティ観点から回答を評価してください:
1. 機密情報の漏洩リスク
2. 安全なコーディング実践
3. 脆弱性の有無
0-100のスコアで評価してください。
"""

Example

Equal weighting configuration

User: 評価設定を作成して

Agent: 評価設定を提案します。

       メトリクス（均等重み付け）:
       - ClarityCoherence: 33.4%
       - Coverage: 33.3%
       - Relevance: 33.3%

       この構成でよろしいですか？

User: はい

Agent: 設定ファイルを生成しました:
       ✓ configs/evaluators/evaluator.toml
       ✓ configs/judgment/judgment.toml

Custom weighting configuration

User: 正確性を重視した評価設定を作って

Agent: 関連性（Relevance）を重視した設定を提案します。

       メトリクス:
       - Relevance: 50%（重視）
       - ClarityCoherence: 30%
       - Coverage: 20%

       この構成でよろしいですか？

User: はい

Example generated configuration file

evaluator.toml (custom weighting):

# MixSeek Evaluator Configuration
# Generated by mixseek-evaluator-config skill

default_model = "google-gla:gemini-2.5-pro"
temperature = 0.0
timeout_seconds = 300
max_retries = 3

[[metrics]]
name = "Relevance"
weight = 0.5

[[metrics]]
name = "ClarityCoherence"
weight = 0.3

[[metrics]]
name = "Coverage"
weight = 0.2

judgment.toml:

# MixSeek Judgment Configuration
# Generated by mixseek-evaluator-config skill

model = "google-gla:gemini-2.5-pro"
temperature = 0.0
timeout_seconds = 60
max_retries = 3

Weighting rules

Weighting has the following rules:

1. Specify all or omit all: You cannot specify weights for only some metrics
2. Total 1.0: The sum of all weights must be 1.0 (±0.001)
3. Equal when omitted: If weights are omitted, they are automatically distributed equally

# 有効: 全て指定
[[metrics]]
name = "ClarityCoherence"
weight = 0.5

[[metrics]]
name = "Coverage"
weight = 0.5

# 有効: 全て省略（均等配分）
[[metrics]]
name = "ClarityCoherence"

[[metrics]]
name = "Coverage"

# 無効: 一部のみ指定
[[metrics]]
name = "ClarityCoherence"
weight = 0.5  # ❌

[[metrics]]
name = "Coverage"
# weight省略 ❌

Troubleshooting

Weight total error

Error: Weights must sum to 1.0

Solution:

• Adjust so that the sum of all weights is 1.0
• Or omit all weights and distribute equally

Metric name error

Error: Unknown metric name

Solution:

• Use a valid metric name: ClarityCoherence, Coverage, LLMPlain, Relevance
• Be careful about uppercase and lowercase

Unstable judgment

Solution:

• judgment.tomlのtemperatureSet to 0.0 (deterministic)
• seedSet to a fixed value

References

• TOML schema details: references/TOML-SCHEMA.md
• Standard metrics: references/METRICS.md
• Orchestrator configuration: skills/mixseek-orchestrator-config/