bootstrap

Bootstrap

Cold-start an evaluation system when you have nothing. In 30 minutes you get a working v0 grader and a clear path to a calibrated, trustworthy evaluation.

Requires OpenJudge (pip install py-openjudge) for the grader generators (SimpleRubricsGenerator / IterativeRubricsGenerator). The interview, stratification, and calibration-roadmap methodology is SDK-independent.

Checklist

You MUST create a task for each item and complete them in order:

1. Understand the product — one-shot interview, not question-by-question
2. Generate v0 grader — use OpenJudge SimpleRubricsGenerator
3. Synthesize eval inputs — 30 inputs with 60/30/10 stratification
4. Run v0 evaluation — GradingRunner with the generated grader
5. Output roadmap — exactly how to reach 50 labels → calibrate

Step 1: Product Interview (One Shot)

Ask the user to describe their system in one go:

To bootstrap your evaluation, I need to understand what you're building.
Please describe (all at once):

- What does your system do? Who uses it?
- What are 3 examples of perfect outputs?
- What are 3 things the system must never do?
- What failures worry you most?

Don't drip-feed these questions. One prompt, one answer. If the user provides a spec doc or design document instead, read that directly.

Step 2: Generate v0 Grader

Use OpenJudge's SimpleRubricsGenerator to create a zero-shot grader from the product description:

import asyncio
from openjudge.models.openai_chat_model import OpenAIChatModel
from openjudge.generator.simple_rubric.generator import (
    SimpleRubricsGenerator,
    SimpleRubricsGeneratorConfig,
)
from openjudge.runner.grading_runner import GradingRunner

# OpenAIChatModel reads OPENAI_API_KEY / OPENAI_BASE_URL from the environment.
# For Aliyun DashScope (Bailian): set OPENAI_BASE_URL to
# https://dashscope.aliyuncs.com/compatible-mode/v1 and OPENAI_API_KEY to your key.
model = OpenAIChatModel(model="qwen-plus")  # or "gpt-4o", etc.

config = SimpleRubricsGeneratorConfig(
    grader_name="Initial Quality Grader",
    model=model,
    task_description="<summarize from the interview>",
    scenario="<usage context from interview>",
    min_score=0,
    max_score=1,
)

generator = SimpleRubricsGenerator(config)
grader = await generator.generate(
    dataset=[],
    sample_queries=[
        "<example query 1 from interview>",
        "<example query 2 from interview>",
        "<example query 3 from interview>",
    ],
)

Why zero-shot instead of asking the user to write criteria? At this stage, the user doesn't know what "good" means operationally. The generator produces a reasonable starting point. The user refines it after seeing v0 results.

Step 3: Synthesize Eval Inputs

Generate 30 test inputs with stratification. Use 3 different prompt templates for diversity:

Template 1: "Generate a typical {domain} query for a {user_type}"
Template 2: "Create an ambiguous {domain} query where intent is unclear"
Template 3: "Generate an edge-case {domain} query that's unusual but realistic"

Target distribution:

• 60% common/typical queries (18 inputs)
• 30% boundary/ambiguous queries (9 inputs)
• 10% edge-case/unusual queries (3 inputs)

Critical: Generate inputs ONLY. Never generate labels. The labels come from running the actual system and getting human judgments.

# The dataset format for GradingRunner
dataset = [
    {
        "query": "What's the status of my order #12345?",
        "response": "<will be filled by running the system>",
    },
    # ... 30 inputs
]

Step 4: Run v0 Evaluation

Plug the generated grader into GradingRunner:

from openjudge.runner.grading_runner import GradingRunner
from openjudge.graders.schema import GraderScore, GraderError

runner = GradingRunner(
    grader_configs={"v0_quality": grader},
    max_concurrency=8,
)

results = await runner.arun(dataset)

scores = [r.score for r in results["v0_quality"] if isinstance(r, GraderScore)]
errors = [r for r in results["v0_quality"] if isinstance(r, GraderError)]
print(f"V0 Results: avg={sum(scores)/len(scores):.2f}, errors={len(errors)}")

Step 5: Output Roadmap

The v0 grader is uncalibrated — you don't know its TPR/TNR yet. Give the user an exact path to trustworthiness:

Your v0 evaluation is ready. Here's the path to a calibrated system:

Phase 1 (now): Run the v0 grader on 30 inputs to get a baseline.
  → The grader is UNCALIBRATED. Treat scores as directional, not definitive.

Phase 2 (1-2 weeks): Collect 50 human-labeled examples (25 pass + 25 fail).
  → For each system output, have a human mark pass/fail against the criterion.
  → Store labels in labels/<grader_name>.jsonl

Phase 3: When you have 50 labels, run 03-align-human to:
  → Measure TPR/TNR of the v0 grader
  → Detect biases (position, verbosity, self-enhancement)
  → Get a human-reduction roadmap

Phase 4: When TPR >= 0.8 and TNR >= 0.8:
  → The grader is calibrated and can be used as a production gate

Quick Mode vs Deep Mode

• Quick mode (default): Steps 1-5 above. 30 minutes to v0. Use when stakes=low or when exploring.
• Deep mode: If the user has 20+ labeled examples, use IterativeRubricsGenerator instead of SimpleRubricsGenerator for data-driven grader creation:

from openjudge.generator.iterative_rubric.generator import (
    IterativeRubricsGenerator,
    IterativePointwiseRubricsGeneratorConfig,
)

config = IterativePointwiseRubricsGeneratorConfig(
    grader_name="Data-Driven Grader",
    model=model,
    task_description="<from interview>",
    min_score=0, max_score=1,
    max_epochs=3,
    batch_size=10,
)
generator = IterativeRubricsGenerator(config)
grader = await generator.generate(dataset=labeled_data)  # 20+ labeled examples

Red Flags — STOP and Re-evaluate

• "I'll generate both inputs and labels with the LLM to save time" → STOP. LLM-generating labels creates a self-consistency loop. TPR will look great until you test on real data, then it collapses.
• "The v0 grader looks good, let's deploy it as a gate" → STOP. Uncalibrated graders have unknown TPR/TNR. They might pass everything or fail everything.
• "I'll skip the roadmap, the user knows what to do next" → STOP. The roadmap IS the deliverable. Without it, bootstrap just produces an untrustworthy grader.

Common Mistakes

• Over-interviewing. One prompt with 4 questions. Don't ask follow-ups unless the answers are genuinely unclear.
• Too many principles in v0. SimpleRubricsGenerator works best with a focused task description. Don't try to evaluate 10 dimensions in v0 — start with the 2-3 most important ones.
• Skipping stratification in synthetic inputs. If all 30 inputs are typical queries, you'll never see how the system handles edge cases.
• Presenting v0 scores as truth. Always prefix v0 results with "UNVERIFIED — these scores are directional only."

Next Skills

After 08-bootstrap:

• 03-align-human: Once 50 human labels are collected, calibrate the grader.
• 01-eval-design: If you want a properly stratified dataset beyond the v0 30 inputs.
• 02-metric-design: If you need multiple graders for different dimensions.