Recommended execution model & parallelization
Model: sonnet Required (opus costs 3-5x more. Accuracy difference is small)
haiku prohibited: sonnet or higher is required for semantic linking of procedures
Parallelization: Split chapters into N parts and launch N Agents in parallel
| Number of chapters | Recommended parallelism | Chapters per Agent |
|---|---|---|
| ~12 chapters | 2 Agent | 6 chapters each |
| ~24 chapters | 4 Agent | 6 chapters each |
| ~36 chapters | 4-6 Agent | 6-9 chapters each |
| 36 chapters+ | 6 Agent | 6-10 chapters each |
Agent launch template:
Agent(
model: "sonnet",
mode: "bypassPermissions",
run_in_background: true,
prompt: "Process chapters NN-NN of [bundle_path]..."
)
Note: visual-indexer is the most cost-heavy because it Reads all images.Always use sonnet + parallel. Running all chapters sequentially with opus is forbidden.
Prerequisite: visual-indexer must be complete. procedure-extractor references the frame information in visual-index.json to link procedures with screenshots.
WHAT
Extract UI walkthroughs and operation procedures from lecture transcripts, link them with screenshots classified by visual-indexer, and generate procedures.json .
WHY
- Lecture videos contain a mix of "concept explanations" and "operation demonstrations"
- concept-extractor extracts concepts, definitions, and quotes, but operation procedures are out of scope
- Procedures only become reproducible once they are linked with screenshots
- This skill differs from concept-extractor in that it is multimodal (it actually reads screenshots)
Boundary with concept-extractor
| Target | Responsible skill | Example |
|---|---|---|
| Concepts, definitions, theories | concept-extractor | "What is wall-bashing (aizuchi)..." |
| Quotes, citations | concept-extractor | "AI is a tool" |
| Screen operations, setup procedures | procedure-extractor | "Click here to ~" |
| Tool operation demos | procedure-extractor | "Enter it in the search field on note and..." |
Input priority
| Priority | Source | Use |
|---|---|---|
| 1 | chapters/{id}/manual.md |
Contextual understanding of procedures (edited and structured) |
| 2 | chapters/{id}/transcript.txt |
Procedure details (operation wording remains) |
| 3 | chapters/{id}/visual-index.json |
Screenshot linking (required) |
| 4 | resources-manifest.json |
Linking resource references |
It is best to read both the manual and the transcript. Use the manual to grasp the structure of the procedures, and use the transcript to pick up the specific wording of operations.
BLOCKER gates
If the following are not satisfied, abort processing and report to the user.
| Gate | Check content | Abort message |
|---|---|---|
| B1 | manifest.json exists | manifest.jsonが見つかりません。course-bundleを先に実行してください。 |
| B2 | visual-index.json for the target chapter exists | visual-index.jsonが見つかりません。visual-indexerを先に実行してください。 |
| B3 | transcript.txt for the target chapter exists | transcript.txtが見つかりません。 |
Processing flow
1. manifest.json を読む
2. 対象チャプターの visual-index.json を読む ← BLOCKER B2
3. 対象チャプターの transcript.txt を読む ← BLOCKER B3
4. 対象チャプターの manual.md があれば読む（構造化済みマニュアル）
5. transcript_ts.json があれば読む（タイムスタンプ付きセグメント）
6. resources-manifest.json があれば読む（リソース参照用）
7. トランスクリプト（+マニュアル）から手順的言語パターンをスキャン
8. 関連するアクションをprocedureにグルーピング
9. 各ステップをスクリーンショットと紐付け（→ Screenshot Correlation）
10. 各ステップのリソース参照を紐付け（→ リソース参照の紐付け）
11. procedures.json を生成
12. 自己検証を実行
Step 5: Detection of procedural language patterns
Detect the following patterns from the transcript. For details, see references/visual-correlation.md .
Action verb patterns
| Category | Pattern examples |
|---|---|
| Click-related | click, tap, press, press and, click |
| Input-related | input, enter, type, write, paste |
| Selection-related | select, choose, check, on/off |
| Movement-related | open, access, move, jump, transition |
| Setting-related | set, change, switch, turn on |
| Demonstrative + verb | "here ~", "on this screen ~", "this ~" |
Grouping rule
| NG | OK | Reason |
|---|---|---|
| One giant procedure (20 steps) | Split into small procedures of 5-8 steps | Long procedures are hard to understand |
| Extract conceptual explanations as procedures | Extract only clear actions (verb + object) | Concepts are the domain of concept-extractor |
| List steps without context | Make the purpose of the procedure explicit with title and outcome | Procedures alone do not convey "what for" |
| Ignore resource references and record only action statements | Link external resources with resource_ref | Resources can be included in assets/ when turning into a skill |
| Extract only from the transcript even though a manual exists | Use both the manual and the transcript | Manuals are already structured, making procedures easier to grasp |
Guidelines for splitting:
- Timing when the screen changes
- Timing when the purpose changes ("search" -> "purchase")
- Consider splitting when exceeding 8 steps
Step 7: Screenshot Correlation
Logic for linking with screenshots.
Linking flow
1. transcript_ts.json からアクション動詞の出現タイムスタンプを特定
2. visual-index.json で該当タイムスタンプ付近のフレームを検索
- 対象: type が browser_screenshot または terminal_screenshot のフレーム
- 範囲: アクション前後 ±5秒
3. 候補フレームを Read ツールで実際に読み、内容を確認
4. トランスクリプトの操作内容と画面内容が一致するフレームを screenshot_ref に設定
Linking rules
| NG | OK | Reason |
|---|---|---|
| Specify a file that does not exist in screenshot_ref | Reference only frames that actually exist from visual-index | Broken references cause errors during skill composition |
| Force screenshot_ref on all steps | Steps without screenshots are screenshot_ref: null |
Not every action has a screen |
| Forcefully limit one screenshot to one step | Multiple steps may reference the same screenshot | Consecutive operations on the same screen are natural |
| Match mechanically using only timestamps | Actually read and verify the contents of the image | Timestamp discrepancies occur frequently |
When transcript_ts.json is absent
If there is no timestamp information, substitute with the following:
- Estimate the timeline of the entire chapter from the position in the transcript (percentage from the beginning)
- Cross-check with the frame order in visual-index.json
- Actually read the image content and cross-check with the context of the transcript
Step 10: Linking resource references
For steps that use external resources within the procedure, set resource_ref .
Detection patterns
- "Copy from Notion", "Use the template", "Refer to the PDF"
- "Download" "the prompts here"
Linking method
- Read resources-manifest.json
- Detect resource mentions within the procedure
- Match with the corresponding resource in the manifest
- Set the path in resource_ref (null if no match)
Output format: procedures.json
For each chapter, procedures.json .
{
"chapter_id": "06",
"procedures": [
{
"id": "p06-01",
"title": "noteでプロンプトを検索する方法",
"steps": [
{ "step": 1, "action": "NotionからPredictionXプロンプトをコピー", "screenshot_ref": "frame_011.jpg", "resource_ref": "prompts/prediction-x.md" },
{ "step": 2, "action": "note.comにアクセス", "screenshot_ref": "frame_012.jpg", "resource_ref": null },
{ "step": 3, "action": "検索欄に「プロンプト」と入力", "screenshot_ref": "frame_013.jpg", "resource_ref": null },
{ "step": 4, "action": "有料のみフィルタを選択", "screenshot_ref": "frame_014.jpg", "resource_ref": null }
],
"outcome": "有料のプロンプト一覧が表示される",
"tags": ["note", "プロンプト検索", "壁打ち"],
"prerequisites": ["c03-02"],
"difficulty": "beginner"
}
]
}
Field specifications
| Field | Required | Description |
|---|---|---|
| chapter_id | Yes | Chapter number (matches manifest.json) |
| procedures[].id | Yes | p{chapter_id}-{連番} format |
| procedures[].title | Yes | A concise title representing the purpose of the procedure |
| procedures[].steps[].step | Yes | Step number (starting from 1) |
| procedures[].steps[].action | Yes | Specific operation details |
| procedures[].steps[].screenshot_ref | Yes | Frame file name or null |
| procedures[].steps[].resource_ref | No | Resource file path or null(corresponds to resources-manifest.json) |
| procedures[].outcome | Yes | What happens when this step is completed |
| procedures[].tags | Yes | Specific tags (tool names, feature names) |
| procedures[].prerequisites | No | Prerequisite concept IDs (c{xx}-{yy} format) |
| procedures[].difficulty | Yes | beginner / intermediate / advanced |
difficulty criteria
| Level | Criteria |
|---|---|
| beginner | Completes on a single screen, no special knowledge required |
| intermediate | Multiple screen transitions, understanding of setting items required |
| advanced | External integration, API settings, conditional branching present |
Step 9: Self-verification
After generation, check all of the following. If even one fails, fix it.
- Does each procedure have a clear title and outcome?
- Do all screenshot_refs point to frames that exist in visual-index.json?
- Is the order of steps logical? (Are there no later steps coming without prerequisites?)
- Are the tags specific? (Are there no vague tags like "Other"?)
- Is there no conceptual content mixed in? (That is the job of concept-extractor)
- Is one procedure 8 steps or fewer?
- procedures.jsonが
schemas/procedures.schema.jsonDoes it comply with ? - Do the files for all screenshot_refs actually exist? (Check with Bash)
- Do the paths of steps with resource_ref set exist in resources-manifest.json?
- Did you reference the manual in chapters that have a manual?
screenshot_ref verification command
# procedures.jsonから全screenshot_refを抽出し、存在チェック
jq -r '.procedures[].steps[].screenshot_ref // empty' procedures.json | while read f; do
[ -f "$FRAMES_DIR/$f" ] || echo "MISSING: $f"
done
Presentation to the user
After processing is complete, report in the following format:
## 手順抽出完了: Chapter {id}
- 抽出数: {n} procedures, {m} total steps
- スクショ紐付け: {linked}/{total} steps にスクショあり
- 難易度分布: beginner={b}, intermediate={i}, advanced={a}
### 抽出した手順一覧
| ID | タイトル | ステップ数 | スクショ数 |
|----|---------|----------|----------|
| p{id}-01 | ... | 5 | 3 |
| p{id}-02 | ... | 4 | 4 |
出力先: {path}/procedures.json
What this skill does not do
| What it does not do | Reason | Skill to use instead |
|---|---|---|
| Extraction of concepts and definitions | Text-only work | concept-extractor |
| Video frame extraction | Preprocessing | video-to-frames |
| Screenshot classification and indexing | Preprocessing | visual-indexer |
| Video download | Preprocessing | video-downloader |
| Manual document generation | Post-processing | (Separate skill) |