iterative-retrieval

Iterative Retrieval Pattern

Solves the "context problem" in multi-agent workflows—subagents often don't know which context they need before they start working.

When to Activate

When generating subagents that need codebase context that cannot be fully predicted in advance
When building multi-agent workflows where context needs to be progressively refined
When encountering "context too large" or "missing context" failures in agent tasks
When designing RAG-like retrieval pipelines for code exploration
When optimizing Token usage in agent orchestration

The Problem

Generated subagents usually only carry limited context. They don't know:

Which files contain the relevant code
What patterns exist in the codebase
What terminology the project uses

Standard approaches often fail:

Send everything: exceeds context limits.
Send nothing: the agent lacks critical information.
Guess the requirements: often guesses wrong.

The Solution: Iterative Retrieval

A 4-phase loop for progressively refining context:

┌─────────────────────────────────────────────┐
│                                             │
│   ┌──────────┐      ┌──────────┐            │
│   │   派发   │─────▶│   评估   │            │
│   │ DISPATCH │      │ EVALUATE │            │
│   └──────────┘      └──────────┘            │
│        ▲                  │                 │
│        │                  ▼                 │
│   ┌──────────┐      ┌──────────┐            │
│   │   循环   │◀─────│   优化   │            │
│   │   LOOP   │      │  REFINE  │            │
│   └──────────┘      └──────────┘            │
│                                             │
│        最多 3 个周期，然后继续              │
└─────────────────────────────────────────────┘

Phase 1: DISPATCH

An initial broad query to collect candidate files:

// 从高层意图开始
const initialQuery = {
  patterns: ['src/**/*.ts', 'lib/**/*.ts'],
  keywords: ['authentication', 'user', 'session'],
  excludes: ['*.test.ts', '*.spec.ts']
};

// 派发给检索智能体
const candidates = await retrieveFiles(initialQuery);

Phase 2: EVALUATE

Evaluate the relevance of the retrieved content:

function evaluateRelevance(files, task) {
  return files.map(file => ({
    path: file.path,
    relevance: scoreRelevance(file.content, task),
    reason: explainRelevance(file.content, task),
    missingContext: identifyGaps(file.content, task)
  }));
}

Scoring criteria:

High (0.8-1.0): directly implements the target functionality
Medium (0.5-0.7): contains related patterns or types
Low (0.2-0.4): indirectly related
None (0-0.2): irrelevant, exclude

Phase 3: REFINE

Update the search criteria based on the evaluation results:

function refineQuery(evaluation, previousQuery) {
  return {
    // 添加在高度相关文件中发现的新模式
    patterns: [...previousQuery.patterns, ...extractPatterns(evaluation)],

    // 添加在代码库中发现的术语
    keywords: [...previousQuery.keywords, ...extractKeywords(evaluation)],

    // 排除已确认为不相关的路径
    excludes: [...previousQuery.excludes, ...evaluation
      .filter(e => e.relevance < 0.2)
      .map(e => e.path)
    ],

    // 针对特定的缺口
    focusAreas: evaluation
      .flatMap(e => e.missingContext)
      .filter(unique)
  };
}

Phase 4: LOOP

Repeat using the refined criteria (up to 3 cycles):

async function iterativeRetrieve(task, maxCycles = 3) {
  let query = createInitialQuery(task);
  let bestContext = [];

  for (let cycle = 0; cycle < maxCycles; cycle++) {
    const candidates = await retrieveFiles(query);
    const evaluation = evaluateRelevance(candidates, task);

    // 检查是否已有足够的上下文
    const highRelevance = evaluation.filter(e => e.relevance >= 0.7);
    if (highRelevance.length >= 3 && !hasCriticalGaps(evaluation)) {
      return highRelevance;
    }

    // 优化并继续
    query = refineQuery(evaluation, query);
    bestContext = mergeContext(bestContext, highRelevance);
  }

  return bestContext;
}

Practical Examples

Case 1: Bug Fix Context

任务："修复身份验证令牌过期 bug"

周期 1:
  派发 (DISPATCH)：在 src/** 中搜索 "token", "auth", "expiry"
  评估 (EVALUATE)：发现 auth.ts (0.9), tokens.ts (0.8), user.ts (0.3)
  优化 (REFINE)：添加 "refresh", "jwt" 关键词；排除 user.ts

周期 2:
  派发 (DISPATCH)：搜索优化后的术语
  评估 (EVALUATE)：发现 session-manager.ts (0.95), jwt-utils.ts (0.85)
  优化 (REFINE)：上下文已足够（2 个高度相关文件）

结果：auth.ts, tokens.ts, session-manager.ts, jwt-utils.ts

Case 2: Feature Implementation

任务："为 API 端点添加速率限制 (rate limiting)"

周期 1:
  派发 (DISPATCH)：在 routes/** 中搜索 "rate", "limit", "api"
  评估 (EVALUATE)：无匹配项 —— 代码库使用的是 "throttle" 术语
  优化 (REFINE)：添加 "throttle", "middleware" 关键词

周期 2:
  派发 (DISPATCH)：搜索优化后的术语
  评估 (EVALUATE)：发现 throttle.ts (0.9), middleware/index.ts (0.7)
  优化 (REFINE)：需要路由器模式

周期 3:
  派发 (DISPATCH)：搜索 "router", "express" 模式
  评估 (EVALUATE)：发现 router-setup.ts (0.8)
  优化 (REFINE)：上下文已足够

结果：throttle.ts, middleware/index.ts, router-setup.ts

Integrating with Agents

Use in agent prompts:

在为此任务检索上下文时：
1. 从广泛的关键词搜索开始
2. 评估每个文件的相关性（0-1 等级）
3. 识别仍然缺失的上下文
4. 优化搜索条件并重复执行（最多 3 个周期）
5. 返回相关性 >= 0.7 的文件

Best Practices

Go from broad to deep, gradually narrowing the scope — don't make the initial query too specific.
Learn the codebase terminology — the first cycle usually reveals naming conventions.
Track what's missing — explicit gap identification is the key to driving refinement.
Stop while you're ahead — 3 highly relevant files are better than 10 mediocre ones.
Exclude decisively — low-relevance files usually don't suddenly become relevant.

Related Resources

The Longform Guide — Subagent Orchestration section
continuous-learning Skills — patterns for improving over time
~/.claude/agents/ agent definitions in