volcengine-video-understanding

Volcano Video Understanding

Use the ByteDance Volcano Ark video understanding API (doubao-seed-2-0-pro-260215 and other models) for in-depth understanding and analysis of videos.

Recommended method: Files API upload + Responses API analysis

Supports up to 512MB video files
Automatic video preprocessing (FPS sampling)
Files can be reused (stored for 7 days)

Features

Video upload: Upload local videos via the Files API (recommended, up to 512MB)
Content understanding: Analyze video scenes, people, actions, and emotions
Video Q&A: Answer user questions based on video content
Video description: Automatically generate video descriptions and summaries

Prerequisites

You need to set the ARK_API_KEY environment variable.

Configuration method (recommended)

Copy the configuration template:

cp .canghe-skills/.env.example .canghe-skills/.env

Edit the .canghe-skills/.env file and fill in your API Key:

ARK_API_KEY=your-actual-api-key-here

Or use environment variables

export ARK_API_KEY="your-api-key"

Loading priority

System environment variables (process.env)
Current directory .canghe-skills/.env
User home directory ~/.canghe-skills/.env

Usage

1. Basic video analysis (Files API method - recommended)

cd ~/.openclaw/workspace/skills/volcengine-video-understanding
python3 scripts/video_understand.py /path/to/video.mp4 "描述这个视频的内容"

2. Video Q&A

python3 scripts/video_understand.py /path/to/video.mp4 "视频中出现了哪些人物？"

3. Sentiment analysis

python3 scripts/video_understand.py /path/to/video.mp4 "分析视频中人物的情感变化"

4. Specify model and frame rate

python3 scripts/video_understand.py /path/to/video.mp4 "总结视频要点" \
  --model doubao-seed-2-0-pro-260215 \
  --fps 2

5. Save results to file

python3 scripts/video_understand.py /path/to/video.mp4 "描述视频" --output result.json

Parameter description

Parameter	Default value	Description
`video_path`	Required	Video file path
`instruction`	Required	Analysis instruction/question
`--model`	doubao-seed-2-0-pro-260215	Model ID
`--fps`	1	Video sampling frame rate (preprocessing)
`--output`	-	Result output file path

Supported models

doubao-seed-2-0-pro-260215 (default)
doubao-seed-2-0-lite-250728
doubao-seed-1-6-251015
Other Seed series video understanding models

Analysis examples

Example 1: Video content description

python3 scripts/video_understand.py ~/Desktop/video.mp4 "详细描述这个视频的内容，包括场景、人物和动作"

Example 2: Video summary

python3 scripts/video_understand.py ~/Desktop/video.mp4 "用3句话总结这个视频的要点"

Example 3: Action recognition

python3 scripts/video_understand.py ~/Desktop/video.mp4 "视频中的人物在做什么动作？按时间顺序描述"

Example 4: Scene analysis

python3 scripts/video_understand.py ~/Desktop/video.mp4 "分析视频中的场景变化和环境特征"

Technical details

Invocation flow

Upload video: Upload the local video file via the Files API, specifying the FPS preprocessing configuration
Wait for processing: Wait for video preprocessing to complete (status changes to processed)
Create task: Call the Responses API for video understanding
Get results: Return the analysis results

API format

Files API upload：

curl https://ark.cn-beijing.volces.com/api/v3/files \
  -H "Authorization: Bearer $ARK_API_KEY" \
  -F 'purpose=user_data' \
  -F 'file=@video.mp4' \
  -F 'preprocess_configs[video][fps]=1'

Responses API analysis：

{
  "model": "doubao-seed-2-0-pro-260215",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_video",
          "file_id": "file-xxxx"
        },
        {
          "type": "input_text",
          "text": "用户指令"
        }
      ]
    }
  ]
}

FPS setting suggestions

FPS	Applicable scenarios
0.3-0.5	Slow-paced videos, static scenes, saving tokens
1	General video analysis (default)
2-3	Fast actions, detailed analysis

Limitations

Video format: MP4 (recommended), MOV, AVI
File size: Maximum 512MB (Files API method)
Storage duration: Uploaded files are stored for 7 days by default
Processing time: Depending on video length and complexity, usually 10-60 seconds

Python API Usage

from scripts.video_understand import analyze_video

result = analyze_video(
    file_path="/path/to/video.mp4",
    instruction="描述视频内容",
    model="doubao-seed-2-0-pro-260215",
    fps=1
)

# 提取回答
text = ""
for item in result.get("output", []):
    if item.get("type") == "message":
        for content in item.get("content", []):
            if content.get("type") == "output_text":
                text = content.get("text", "")
                break

print(text)

Error Handling

Common errors and solutions:

Error	Cause	Solution
API Key error	Not set or incorrect	Check the ARK_API_KEY environment variable
File does not exist	Incorrect path	Check the file path
Upload failed	File too large or unsupported format	Check the file size (<512MB) and format
Processing timeout	Video too long or complex	Shorten the video or lower the FPS

About this skill