synthetic-scenarios

Synthetic scenario generation

A coding agent follows the procedure below to generate synthetic evaluation data for an AI agent. The procedure produces a collection of realistic requests, each paired with an expected outcome computed from the database or policy documents, not from a model's own answer.

When to use

Use the procedure when a specification and an authoritative data source already exist, but representative requests and traces do not. The procedure generates requests against an existing application; it does not create the application's database, policies, or expected outcomes.

Inputs

• The path to the agent's specification (a human-readable document that states the agent's scope, roles, permissions, and tool contracts).
• A database or corpus that the procedure can query to compute expected outcomes, e.g., a SQLite database with orders and an eligibility function.

Key concepts

A dimension is one named source of variation across requests. Each dimension represents a way the requests can differ from one another. For example, a customer support agent might vary along the following dimensions: the user's role (shopper, merchant, support staff), the user's intent (refund, order status, policy question), the state of the order (eligible, expired, above the approval threshold), and the difficulty of the request (straightforward or ambiguous). Defining dimensions before generating requests prevents the procedure from producing many phrasings of common requests and few of the unusual requests that test policy boundaries.

A tuple assigns one value to each dimension, e.g., (shopper, refund, expired, ambiguous). A model then drafts a natural request from the tuple. The tuple constrains what the request must represent, while the model supplies the wording.

A scenario is one planned request together with enough context to run the agent and, when possible, an expected outcome. A scenario for a refund request records the authenticated role, the opening message, the order being discussed, and whether the eligibility function says the refund should be approved or denied.

Procedure

1. Read the specification and think about what kinds of requests the agent will receive. Propose 5 to 8 dimensions with values that are specific to the agent's task. For example, a support agent's dimensions might include the user's role, the user's intent, and the state of the order being discussed. Show the proposed dimensions to the human before generating anything.
2. Create a coverage set in which every dimension value appears repeatedly. Create a separate challenge set from difficult but valid combinations. Do not select a challenge merely because a model failed on the exact request; use the underlying dimension (e.g., a policy override or missing data) to generate new cases.
3. Draft one natural opening message per tuple. For multi-turn scenarios, put exact user utterances in the followups field; the runner sends every string verbatim, so do not place persona notes or generation instructions in the field.
4. Check for duplicate requests, unrealistic phrasing, and missing dimension values. Assign scenario_group as coverage or challenge.
5. Compute the expected outcome per scenario from ground truth (e.g., a SQL query against the database, or a deterministic eligibility function). A model assertion is not ground truth. When code or a policy document cannot determine the expected outcome, mark the scenario for later human judgment.
6. Run a pilot of 30 scenarios on the selected model. Confirm failures against the recorded expected results, then identify dimensions associated with the difficult cases. If the first pilot produces too few failures, use a lower capability model from the same provider. Do not create a failure taxonomy during scenario generation.
7. Generate new challenge scenarios from the difficult dimensions. Preserve the coverage set, because a collection containing only failures cannot show ordinary behavior.
8. Emit JSONL, one scenario per line.

Output schema

{
  "id": "support-0042",
  "scenario_group": "challenge",
  "data_quality_case_id": null,
  "tuple": {"role": "shopper", "intent": "refund", "order_id": 4127},
  "opening_message": "I want to return order 4127.",
  "followups": [],
  "expected": {
    "evaluation": "objective",
    "outcome": "refund_approved",
    "reason": "Order is within the return window and below the approval threshold.",
    "source": {
      "type": "eligibility_function",
      "reference": "is_refund_eligible(order_id=4127)"
    }
  }
}

For an objective case, expected.source.type is sql, eligibility_function, policy_document, or data_quality_table. expected.source.reference identifies the query, function, or document used to determine the answer.

Use the following form when a person must judge the resulting trace:

{
  "expected": {
    "evaluation": "human_judgment",
    "criterion": "The response should explain the policy clearly without inventing facts.",
    "source": {
      "type": "specification",
      "reference": "SPEC.md, RESP-3"
    }
  }
}

Use objective when the scenario tests an answer or action that data, code, or a policy document fixes exactly. Use human_judgment when the scenario tests a response quality that has several acceptable answers. A model response cannot be the source of an expected result, because the response may contain the error the evaluation needs to find.

The id field links the scenario to its traces through a span attribute (e.g., scenario_id). Use data_quality_case_id: null for an ordinary scenario. A scenario involving a documented data defect uses the matching identifier.