SuperAPI AI Directory

visual-indexer

General & other

View repository

Updated 10-15 23:40

About this skill

visual-indexer

A skill that catalogs all screenshots in the course-bundle and performs classification, OCR, and metadata assignment.


Recommended execution model & parallelization

Model: sonnet Required (opus costs 3-5x more. The accuracy difference is small)

Reading all frames in a single execution is the most costly. Strictly use sonnet in parallel.

Parallelization: Split chapters into N parts and launch N Agents in parallel

Number of chapters Recommended parallelism Chapters per Agent
~12 chapters 2 Agent 6 chapters each
~24 chapters 4 Agent 6 chapters each
~36 chapters 4-6 Agent 6-9 chapters each
36 chapters+ 6 Agent 6-10 chapters each

Agent launch template:

Agent(
  model: "sonnet",
  mode: "bypassPermissions",
  run_in_background: true,
  prompt: "Process chapters NN-NN of [bundle_path]..."
)

Note: visual-indexer reads all images, so it is the most cost-heavy.Always use sonnet + parallel. Running all chapters sequentially with opus is forbidden.


WHAT — What it does

Reads each screenshot image contained in every chapter of the course one by one, and performs the following:

  1. Frame classification — Classify images using the 7 core types + course-specific types (custom_types)
  2. OCR extraction — Extract all text within the image
  3. Identification of visual elements — Record visual elements such as shapes, arrows, boxes, etc.
  4. Keyword assignment — Tag so that downstream skills can search easily
  5. Index generation — By chapter visual-index.json + overall visual-catalog.json

WHY — Why it is necessary

  • procedure-extractor needs OCR information and frame classification to link procedure images
  • concept-extractor uses slide images as supporting material
  • Reduce downstream processing costs by excluding talking_head and transition
  • A catalog is needed to get an overview of the image assets of the entire course

HOW — Execution procedure

Phase 0: Preparation & verification

BLOCKER: manifest.jsonが存在するか確認
  → 存在しない場合は course-ingest を先に実行するよう案内して停止
  1. manifest.json and obtain the chapter list and screenshot directories
  2. Check the existence of the screenshot directory for each chapter
  3. Count all frames and create a processing plan

Phase 0.5: Discovery of course-specific types

Sample-read the first 5 frames of each chapter:

  1. Determine whether they fit the core types
  2. If there are 2 or more screens that do not fit → propose course-specific types
  3. Present the type definitions to the user or main session (only at the beginning of the chapter)
  4. Once confirmed, visual-index.json の custom_types register in
  5. For subsequent frames, select from core types + course-specific types

Note: For full automation, the Agent itself decides and registers in Phase 0.5.

Phase 1: Chapter-by-chapter frame processing

For each chapter, execute the following:

Step 1.1: Retrieve frame list

ls chapters/{chapter_id}/screenshots/

Step 1.2: Batch processing (5-10 frames at a time)

BLOCKER: 各画像は必ずReadツールで読み取ってから分類する
  → ファイル名からの推測は NG

For each frame:

  1. Read the image with the Read tool
  2. Frame classification — classify with the following taxonomy:

Core types(common to all courses):

type Description How it is used downstream
title_slide Chapter title Manual heading image
content_slide Concept diagrams/slides Reinforcement for concept-extractor
browser_screenshot Browser screen Procedure images for procedure-extractor
terminal_screenshot Command line Procedure images for procedure-extractor
diagram Flowcharts/diagrams Visualization of conceptual relationships
talking_head Speaker video only Low value, can skip
transition Blank/transition frames Skip

Repeated screens that do not fit the core types should be course-specific types (custom_types) (described later).

Course-specific types (custom_types)

Screens that do not fit the core types but appear repeatedly within a course should becourse-specific typesand created dynamically.

Creation rules

  1. First 3-5 framesto consider type candidates

  2. "Does not fit the core types" and "appears 2 or more times" → register as a course-specific type

  3. Name in snake_case, including keywords from the course

    • Example: tmux_pane, vscode_editor, ae_timeline, figma_canvas
  4. Per courseUp to 5 types(to prevent infinite proliferation)

  5. Do not create types that appear only once; assign them to the closest core type

  6. OCR extraction — store all text in the image in the text_content array

  7. Visual element identification — visual_elements record in the array

  8. Keyword assignment — store keywords representing the content in the keywords array

  9. Confidence determination — confidence を high / medium / low set with

  10. UI operation determination — for browser/terminal screens, set is_ui_walkthrough set

Step 1.3: Generate visual-index.json

Once all frames in the chapter have been processed, visual-index.json generate:

{
  "chapter_id": "03",
  "custom_types": {
    "tmux_pane": "tmuxで複数paneに分割されたClaude Code画面(タイトルバーにtmux表記、画面分割あり)"
  },
  "frames": [
    {
      "filename": "frame_005.jpg",
      "type": "content_slide",
      "text_content": ["AIマインド", "壁打ち", "アイデア", "専門知識"],
      "visual_elements": ["equation_diagram", "four_boxes", "arrow"],
      "keywords": ["4原則", "マインド"],
      "confidence": "high"
    }
  ]
}

Output destination: chapters/{chapter_id}/visual-index.json

Schema: schemas/visual-index.schema.json Must conform to.

Phase 2: Generate visual-catalog.json

After processing all chapters, generate the catalog for the entire course:

{
  "course_name": "YouTube Booster",
  "total_frames": 245,
  "course_custom_types": {
    "tmux_pane": {"definition": "tmuxで複数paneに分割されたClaude Code画面", "frame_count": 18},
    "vscode_editor": {"definition": "VS Codeのエディタ画面(ファイルツリー+エディタ+ステータスバー)", "frame_count": 8}
  },
  "type_distribution": {
    "content_slide": 89,
    "browser_screenshot": 45,
    "talking_head": 67,
    "diagram": 23,
    "title_slide": 12,
    "transition": 9
  },
  "chapters": [
    {
      "chapter_id": "03",
      "frame_count": 31,
      "high_value_frames": ["frame_005.jpg", "frame_012.jpg"],
      "visual_index_path": "chapters/03/visual-index.json"
    }
  ]
}
  • high_value_frames: content_slide, browser_screenshot, diagram among confidence: "high" with
  • type_distribution: Aggregate the frame types of all chapters

Output destination: project root's visual-catalog.json

Phase 3: Self-verification

All of the following checklist items must pass:

  • All frames have type is set
  • text_content with empty content_slide is missing (slides must always have text)
  • talking_head frame count is reasonable (usually 20-40%)
  • visual-catalog.json の total_frames matches the actual number of files
  • Each visual-index.json が schemas/visual-index.schema.json complies with
  • type_distribution total matches total_frames matches
  • custom_types count is 5 or fewer
  • Each custom_type has a definition statement
  • Each custom_type has 2 or more frames using it
  • custom_type naming is snake_case
検証失敗時:
  → 不一致があれば該当チャプターを再処理
  → text_contentが空のcontent_slideがあれば画像を再読み取り

BLOCKER gate

Gate Condition Action when not met
G1 manifest.json exists Guide the user to run course-ingest first
G2 Screenshot directory exists Guide the user to run video-to-frames first
G3 Each image was actually read with the Read tool Guessing from filenames is prohibited; always read them
G4 All frames were processed (all of them, not a sample) If there are unprocessed frames, process them additionally
G5 All self-verification checklist items pass Fix failed items and re-verify

NG / OK judgment table

NG OK Reason
Classify based only on filenames Read images with the Read tool before classifying Filenames contain no content information
Miss talking_head and classify everything as content_slide Frames with only a person are clearly classified as talking_head Later skills end up processing unnecessary images
Leave text_content empty Extract all text in the image OCR information is needed for procedure-extractor linking
Process only a sample of frames Process all frames without omission If an omitted frame was a procedure image, the procedure is missing
Do not distinguish between content_slide and diagram Clearly separate slide format from flowchart format concept-extractor's processing method differs
Set confidence to high for everything Set images you are unsure about to medium/low Information is needed to prioritize in later stages
Omit visual_elements Record arrows, boxes, tables, code, etc. Affects downstream image selection accuracy
Forcing screens that don't fit into core types Create course-specific types Misclassification breaks downstream skills
Create a type that appears only once Assign to the closest core type Prevent infinite proliferation of types
Create more than 5 course-specific types in one course Consolidate into the top 5 Prevent catalog bloat
Course-specific type naming uses generic terms (editoretc.) Contains course keywords (vscode_editoretc.) Does not conflict with other courses

Frame classification criteria

For details, see references/frame-types.md Refer to. Overview:

  • title_slide: Large text, chapter numbers, background design. Few text elements
  • content_slide: Bullet points, diagrams, concept explanations. Large amount of text
  • browser_screenshot: Browser UI elements (address bar, tabs, etc.) are visible
  • terminal_screenshot: Black background with monospace font, command line
  • diagram: Arrows, flows, connecting lines. Arrangement of structured shapes
  • talking_head: Person's video occupies most of the screen. No/few text elements
  • transition: Blank screen, fade, blur

Edge cases

  • Small speaker video overlaid on slide → content_slide(slide is primary)
  • Browser screen + commentary text overlay → browser_screenshot
  • Half slide, half speaker → judge by text amount; if there is a lot of text, content_slide

Output file list

File Location Content
visual-index.json chapters/{id}/visual-index.json Frame information by chapter
visual-catalog.json Project root Catalog for the entire course

Linked skills

Skill Relationship
course-ingest Prerequisite: generate manifest.json
video-to-frames Prerequisite: generate screenshots
procedure-extractor Subsequent: use frame classification and OCR
concept-extractor Can run in parallel: use content_slide for reinforcement

Batch processing notes

  • When reading images with the Read tool, process 5-10 images per batch
  • Process all frames even if there are a large number (over 100)
  • Report processing progress by chapter: Chapter 03: 31/31 frames processed
  • Frames where an error occurred are recorded in confidence: "low" and note the error details