video-frame-extractor

Video Reverse-Engineering Tool - Frame Extraction and Visual Analysis

Task Objective

This Skill is used to: extract keyframes from a video, use a visual model to analyze the content of each frame, and generate structured descriptions and prompts
Capabilities include:
- Video frame extraction (supports interval frame extraction and uniform sampling)
- Visual model API calls (supports GPT-4V/Claude-3.5-Sonnet, etc.)
- Batch image analysis
- Structured result output (JSON format)
Trigger conditions: the user needs to analyze video content, extract storyboard references, or generate creative prompts

Prerequisites

Dependency description:

opencv-python>=4.8.0
pillow>=10.0.0
requests>=2.28.0

Visual model configuration: A visual model API key needs to be configured (such as GPT-4V, Claude-3.5-Sonnet). Environment variable format:

export VISION_API_KEY="your_api_key"
export VISION_API_BASE="https://api.openai.com/v1"  # 或其他API地址
export VISION_MODEL="gpt-4-vision-preview"  # 模型名称

Coze Bot configuration (recommended): Use your published Coze Bot API for visual analysis. Environment variable format:
```
export COZE_BOT_ID="7572557757883383858"  # 您的Bot ID
export COZE_API_KEY="cztei_qHZQ0A5OSJjsmfZWmVb8bqu2BTbtB240YGbDYLhZpsIr8jER4aL4Aevyii8rnKfNs"  # 您的API Key
```
Advantages of Coze Bot: it integrates the official frame extraction plugin and visual model, with no additional configuration required

Operation Steps

Standard Process

Step 1: Video frame extraction

Input: video file path or URL
Call script: scripts/video_frame_extractor.py
Parameters:
- --input: video file path or URL
- --output: output image directory
- --interval: frame extraction interval (seconds), default is 1 frame per second
- --max_frames: maximum number of frames to extract, default is 10 frames
Output: the extracted image sequence

# 示例: 抽取视频关键帧
python scripts/video_frame_extractor.py \
  --input ./input/video.mp4 \
  --output ./output/frames \
  --interval 3 \
  --max_frames 10

Step 2: Visual analysis

Option A: Use Coze Bot API (recommended)

Input: extracted frame image directory
Call script: scripts/coze_bot_client.py
Parameters:
- --image_dir: image directory
- --prompt: analysis prompt (optional)
- --output: output JSON file path
Output: description and analysis results for each image

# 示例: 使用Coze Bot分析抽帧图片
python scripts/coze_bot_client.py \
  --image_dir ./output/frames \
  --prompt "分析场景内容、人物状态、构图特点,适合AI视频创作" \
  --output ./output/analysis.json

Option B: Use another visual model API

Input: extracted frame image directory
Call script: scripts/visual_analyzer.py
Parameters:
- --input: image directory or single image path
- --output: analysis result JSON file path
- --prompt: analysis prompt (optional, default analyzes scene/characters/composition)
Output: description and analysis results for each image

# 示例: 使用GPT-4V分析抽帧图片
python scripts/visual_analyzer.py \
  --input ./output/frames \
  --output ./output/analysis.json \
  --prompt "分析场景内容、人物状态、构图特点,适合AI视频创作"

Step 3: Result integration

The agent reads the analysis result JSON
Generate prompts based on the analysis results
Can be used for video creation reference or storyboard design

Optional Parameters

Frame extraction parameters:

--interval: frame extraction interval (seconds); the smaller it is, the denser the frame extraction
--max_frames: maximum number of frames to extract, controls the output quantity
--start_time: start time (seconds)
--end_time: end time (seconds)
--resolution: output image resolution, default 1080P

Analysis parameters:

--prompt: custom analysis prompt
--detail: analysis detail level (brief/standard/detailed)
--batch_size: batch analysis size, default 5 images

Resource Index

Required scripts:
- scripts/video_frame_extractor.py - video frame extraction tool
- scripts/coze_bot_client.py - Coze Bot API calling tool (recommended)
- scripts/visual_analyzer.py - Other visual model API calling tools
Domain references:
- references/analysis-guide.md - Visual analysis guidance and prompt templates

Notes

The visual model API key must be configured in advance, otherwise the analysis function cannot be called
The frame extraction interval and maximum number of frames need to be set reasonably according to the video duration
Visual analysis results depend on model capabilities, and different models may output in different formats
It is recommended to first test with small-scale frame extraction, and then process on a large scale after confirming the results

Usage examples

Example 1: Complete reverse-engineering workflow (using Coze Bot)

# 1. 抽取视频关键帧
python scripts/video_frame_extractor.py \
  --input ./input/source_video.mp4 \
  --output ./output/frames \
  --interval 2 \
  --max_frames 8

# 2. 使用Coze Bot分析抽帧内容
python scripts/coze_bot_client.py \
  --image_dir ./output/frames \
  --prompt "详细描述场景、人物、构图,适合AI视频创作参考" \
  --output ./output/analysis.json

# 3. 智能体读取analysis.json,生成创作提示词
# 智能体将根据分析结果组织内容,生成可用于AI创作的提示词

Example 2: Frame extraction only (no analysis)

python scripts/video_frame_extractor.py \
  --input ./input/video.mp4 \
  --output ./output/frames \
  --interval 5 \
  --max_frames 5

Example 3: Single image analysis

# 使用Coze Bot
python scripts/coze_bot_client.py \
  --image ./output/frames/frame_00001.jpg \
  --prompt "分析这张图片的场景风格和构图"

# 或使用其他视觉模型
python scripts/visual_analyzer.py \
  --input ./output/frames/ \
  --output ./analysis.json \
  --prompt "分析场景风格和构图"

Output format

Frame extraction output:

./output/frames/
├── frame_00001.jpg
├── frame_00002.jpg
├── frame_00003.jpg
└── ...

Analysis output (JSON):

{
  "total_frames": 10,
  "analysis": [
    {
      "frame_file": "frame_00001.jpg",
      "timestamp": "00:00:00",
      "description": "场景描述内容...",
      "elements": ["人物", "背景", "道具"],
      "style": "风格描述..."
    },
    {
      "frame_file": "frame_00002.jpg",
      "timestamp": "00:00:03",
      "description": "场景描述内容...",
      "elements": ["人物", "背景", "道具"],
      "style": "风格描述..."
    }
  ]
}

Technical description

Frame extraction technology:

Use OpenCV's VideoCapture to read the video
Evenly sample keyframes by time interval
Supports multiple video formats (MP4/MOV/AVI, etc.)

Visual analysis:

Supports Coze Bot API (recommended, already integrates frame extraction and visual models)
Supports mainstream visual model APIs (GPT-4V/Claude-3.5-Sonnet, etc.)
Batch processing to improve efficiency
Structured output, convenient for subsequent processing

Coze Bot advantages:

Integrates the official frame extraction plugin, with high frame extraction quality
Built-in visual model, no additional configuration required
One-stop service, integrated frame extraction + analysis
Supports streaming responses to obtain results in real time

Performance optimization:

Configurable frame extraction interval to reduce redundant frames
Batch API calls to reduce the number of requests
Result caching to avoid repeated analysis