autoctx-hillclimb

Auto Context Generation - Hill-Climbing Workflow

This skill guides the process of improving a ContextSet iteratively by analyzing evaluation failures (Gap Analysis) and applying automated structural refinements (Mutations).

[!IMPORTANT] Constraint: In this workflow, you are ONLY allowed use context types templates and facets. Do not attempt to use other context types.

Workflow

Follow these steps exactly in order:

1. Setup & Loop Identification

Validation:
- Check if autoctx/experiments/ directory and autoctx/state.md exist. If missing, warn the user that the workspace might not be initialized (suggest activating the autoctx-init skill).
- Once an experiment is selected, verify it contains an eval_reports/ folder. If missing, suggest running the Evaluation workflow first.
Identify Experiment:
- Read the local autoctx/state.md to identify the active experiment.
- If not found, ask the user to select an experiment folder from autoctx/experiments/.
Determine Loop Version:
- Scan the autoctx/experiments/<experiment_name>/hillclimb/ folder for files matching improved_context_v*.json.
- Determine the loop version vN by finding the maximum N and using N+1. If the folder is empty, start at v1.
Locate Base Context:
- For v1:
  - Check autoctx/state.md to see if a specific base context path was recorded for this experiment (e.g., during the Evaluation setup for user-provided contexts).
  - If not found in state.md, default to the baseline generated by Bootstrap in the experiment folder.
  - If still not found, ask the user for the absolute path to their base context file and record it as the Base Context in autoctx/state.md.
- For vN (where N > 1), the base context is improved_context_v(N-1).json.
- Verify the base context file exists. If missing, STOP and ask the user for the correct path.

2. Phase 1: Gap Analysis

Validation:
- Determine the target evaluation run folder under eval_reports/. If multiple folders exist, find the most recent one by modified time. Prefer the latest run by default, but list other available runs as well (peeking into their summary.csv or configs.csv to show timestamps/metrics for visual context). Ask the user to confirm the selection.
- Verify that the selected eval_reports/<job_id_folder>/ contains expected files (e.g., scores.csv, summary.csv). If missing or empty, STOP and inform the user.
Read Evaluation Results: Use the read_evaluation_result MCP tool passing the path to eval_reports/<job_id_folder>/.

Generate Gap Analysis Report (Batched):

The tool returns a summary and a batch of failure cases (default limit 10).
Iterate through the failure cases by calling the tool with increasing offset (0, 10, 20, ...) until all failed queries are analyzed.
First Batch (offset=0): Initialize the report file with the # Gap Analysis Report - vN header and ## Summary section, followed by the analysis of the first batch under ## Failed Queries Detail.
Subsequent Batches: Call the tool with the next offset, analyze the new failures, and append them to the ## Failed Queries Detail section.

Use the following structure for the report:

# Gap Analysis Report - vN

## Summary
- **Total Queries**: 10
- **Passed**: 7
- **Failed**: 3
- **Pass Rate**: 70%

## Failed Queries Detail

### Query 1: "How many users registered in 2023?"
- **Error Category**: `[FilterError]`
- **Expected SQL**: `SELECT count(*) FROM users WHERE year = 2023`
- **Actual SQL**: `SELECT count(*) FROM users` (Missing filter)
- **Root Cause**: The LLM did not know about the `year` column or how to filter by year for this entity.
- **Proposed Mutation**: Add a facet for "Users by Year".

### Query 2: "Show me top selling products"
- **Error Category**: `[OrderingError]`
- **Expected SQL**: `SELECT name FROM products ORDER BY sales DESC LIMIT 5`
- **Actual SQL**: `SELECT name FROM products LIMIT 5`
- **Root Cause**: Missing ordering instruction in context.
- **Proposed Mutation**: Update the template for "Product Sales" to include ordering.

### Query 3: "Get users older than 30"
- **Error Category**: `[GoldenDataError]`
- **Expected SQL**: `SELECT * FROM users WHERE age >> 30` (Syntax error `>>` in golden SQL)
- **Actual SQL**: `SELECT * FROM users WHERE age > 30`
- **Root Cause**: Invalid syntax in golden dataset.
- **Proposed Mutation**: None. Flag to user to fix the evaluation dataset.

Save Report: You MUST physically write the report file to autoctx/experiments/<experiment_name>/hillclimb/gap_analysis_vN.md. If you are processing in batches, ensure you append to this file until all failed queries are documented. Do not merely output it in chat; it must exist on the file system.
Log in State Tracking:
- Update autoctx/state.md to record the mapping for Loop vN (Base Context <-> Eval Report <-> Gap Analysis).
Human-in-the-Loop Review:
- Inform the user that the Gap Analysis report has been successfully written to disk.
- Ask the user if they want to review, make any corrections, or add manual feedback directly to the file before proceeding to Phase 2 (Context Mutation).
- Wait for user confirmation before starting Phase 2.

3. Phase 2: Context Mutation

Validation: Verify that gap_analysis_vN.md exists and contains findings. Verify the base ContextSet file exists. If missing, STOP and inform the user.
Analyze Gap Report & Determine Fixing Strategy:
- Read gap_analysis_vN.md to identify what needs to be fixed.
- Fixing Strategy Guidelines:
  - Conciseness: Try to use less context to cover more scenarios. Avoid adding redundant or hyper-specific templates for every single edge case.
  - Generalizability: Prefer solutions that generalize well (e.g., use a facet for a column definition rather than a specific template for every query using that column).
  - Supported Types: Support mutations for template, facet, and value_search types.
Apply Mutations:
- Copy the Base Context: Copy the base ContextSet file to the new destination: autoctx/experiments/<experiment_name>/hillclimb/improved_context_vN.json.
- Generate New Items: For any new context items identified in the fixing strategy (for "add" operations):
  - Invoke the context-generation-guide skill to produce the final parameterized items.
  - Provide the identified candidates to that skill.
  - That skill will handle phrase extraction, parameterization, and constructing the valid JSON structure.
- Validate New Items:
  - Templates: Run generated SQL examples via <source>-execute-sql (use dummy values for placeholders) to verify syntax.
  - Others: Cross-check table/column references against the schema via <source>-list-schemas.
- Apply Mutations: Call the mutate_context_set MCP tool passing the new file path as file_path and mutations as mutations_json to mutate the context set.
Log in State Tracking:
- Update autoctx/state.md to include the output path of improved_context_vN.json for Loop vN.

4. Validation & Upload Advice

Summarize Improvements: Tell the user what was changed (e.g., added 2 facets, updated 1 template).
Upload Instructions:
- Read Database Details: Read autoctx/tools.yaml (or db_config.yaml) to fetch the specific project, location, and instance/cluster details for the active database.
- Generate URL: Call the generate_upload_url tool passing the extracted values to provide the direct console link to the user.
- Present the local file path to improved_context_vN.json and the generated console link together in a single clear message.
Instruct Next Step Evaluation:
- Instruct the user to run evaluation using the evaluating workflow on this new ContextSet to see if metrics improve. This will start Loop N+1.

Output

Upon successful completion, the workspace must contain:

autoctx/experiments/<experiment_name>/hillclimb/gap_analysis_vN.md
autoctx/experiments/<experiment_name>/hillclimb/improved_context_vN.json
Updated autoctx/state.md summarizing the run loop.

Logging State Example (`autoctx/state.md`)

When updating autoctx/state.md, please append or update the Hill-Climbing Run Log section:

# Context Authoring Experiment State Tracking

## Active Experiment: my-exp-1

## Hill-Climbing Run Log

### Loop: v1
- **Base Context**: `baseline_context.json`
- **Eval Report Path**: `autoctx/experiments/my-exp-1/eval_reports/<job_id_uuid>/` (containing `configs.csv`, `evals.csv`, etc.)
- **Gap Analysis**: `autoctx/experiments/my-exp-1/hillclimb/gap_analysis_v1.md`
- **Mutated Context**: `autoctx/experiments/my-exp-1/hillclimb/improved_context_v1.json`