SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsdoubao-multimodal

doubao-multimodal

Agents & models

View repository

Updated 10-15 23:32

About this skill

Doubao Multimodal Understanding

Bun + TypeScript CLI wrapping the Doubao-Seed multimodal chat completion endpoint. Resolves a single audio/video source (URL or local path), normalizes it for Ark (download remote → cache, upload local → TOS pre-signed URL), splits oversized media, fans out concurrent Ark calls, and merges the results.

Script Directory

{baseDir} = this SKILL.md's directory. Main entry: {baseDir}/scripts/main.ts. Run with bun run {baseDir}/scripts/main.ts .... Dependencies live in {baseDir}/scripts/package.json (run bun install inside that folder once).

Required Environment

ARK_API_KEY=...                 # 火山方舟 API Key (必填)
ARK_MODEL=...                   # 多模态 endpoint id 或 model 名，例如 doubao-seed-2-0-lite-260428 或 doubao-seed-1-6-flash-250928
ARK_BASE_URL=...                # 可选，默认 https://ark.cn-beijing.volces.com/api/v3
ARK_REASONING_EFFORT=minimal    # 可选
Required when uploading a local file (not needed for remote URL input)

TOS_ACCESS_KEY_ID=... TOS_ACCESS_KEY_SECRET=... TOS_BUCKET=... TOS_REGION=cn-beijing TOS_ENDPOINT=tos-cn-beijing.volces.com # optional TOS_KEY_PREFIX=doubao-multimodal # optional


> 注意：部分 Doubao-Seed endpoint 仅支持 `input_audio` 字段（不支持 `audio_url`）。命中 400 `invalid value: audio_url` 时，加 `--audio-part-type input_audio`。

## Tasks

| `--task` | 说明 | 视频自动转音频 |
|----------|------|---------------|
| `asr` | 纯文字转写（无标点格式约束） | ✅ |
| `asr-timestamp` | 每字带 `开始-结束-字符;` 模板 | ✅ |
| `multispeaker-asr` | `[spk0]...[spk1]...` 输出 | ✅ |
| `ast` | 语音翻译，配合 `--target-language` | ✅ |
| `diarize` | 说话人日志：`[spk][start-end]文本` | ✅ |
| `subtitle-align` | 已知字幕做字符级打轴，需 `--subtitle*` | ✅ |
| `caption` | 音频/视频整体分析（markdown 报告） | ❌ 视频时保留画面 |
| `video-timeline` | 视频时间轴 JSON：`start_end_time/event/people/emotion`（仅视频） | ❌ 视频时保留画面 |
| `keyframe-extract` | 技术博客配图关键帧 JSON：`timestamp/timestamp_sec/description/suggested_caption/reason`（仅视频，可选 `--transcript*`） | ❌ 视频时保留画面 |
| `understand` | 自定义 prompt 的通用理解，需 `--prompt*` | ❌ 视频时保留画面 |

视频自动转音频 = 当任务只看声音时，CLI 会先用 ffmpeg 抽 16kHz mono WAV，再走音频流程；其余任务保留 `video_url` 直接送给 Ark。

## Source Resolution

- `--url <u>`：远程音视频。CLI 会先下载到 `{out-dir}/cache/downloads/<sha>.<ext>`，确保后续 ffmpeg/上传走本地。
- `--path <p>`：本地音视频。CLI 会上传到 TOS，得到预签名 GET URL 给 Ark。
- 类型推断：先看扩展名，再看远程 `Content-Type`；判定不出来就报错，要求 `--type audio|video`。

## Splitting Rules

| 类型 | 阈值 | 单段限制 |
|------|------|----------|
| 视频 | 时长 > 20 min 或 文件 > 50 MB | ≤ 20 min, ≤ 50 MB |
| 音频 | 时长 > 120 min 或 文件 > 50 MB | ≤ 120 min, ≤ 50 MB |

**视频切片**：先 `ffprobe` 取分辨率，源高度 > 720 时下采样到 720p（`scale=-2:720`，保持宽高比，不上采样）。编码 H.264 veryfast，CRF 28（恒定质量）+ maxrate 1500 kbps / bufsize 3000 kbps（动作画面才会冲到上限，PPT/演讲类内容通常 200-400 kbps）+ AAC @ 96 kbps。初始按 `min(20min, 50MB ÷ 假设均值 496 kbps × 0.9 ≈ 12.4min)` 估算切段，所以一段 21 min 的演讲视频通常切 2 段（实测每段 ~17 MB）。如果某段编码后仍 > 50 MB（高动作内容），**不再降码率**，而是把这段时间二分继续切（`partNNNa / partNNNb / partNNNaa ...`），递归到 ≤ 30 s 兜底。

**音频切片**：H.264 不参与，转 mp3 16k mono @ 48 kbps，超过 50 MB 才会按比例降码率重切（音频码率本就低，降码率对 ASR 影响很小）。

多段时按 `--concurrency`（默认 3）并发送到 Ark，最后按时间顺序拼接 `=== Part N (start-end) ===` 分隔；`Segment.index` 在所有子段拍平后从 0 重新编号。

## ffmpeg

启动时自动检测 `ffmpeg` + `ffprobe`。缺失时给出安装命令（macOS Homebrew / apt / dnf / pacman），**默认会停下来等用户回 `y` 确认**；用 `--yes` 可跳过确认（适合 CI/Skill 自动化场景，仅在已得到用户授权后使用）。

## Quick Start

```bash
# 远程音频 ASR
bun run {baseDir}/scripts/main.ts \
  --task asr \
  --url "https://example.com/clip.wav" --type audio \
  --audio-part-type input_audio \
  --out-dir ./out/asr

# 本地视频 → 自动抽音频 → 多说话人 ASR
bun run {baseDir}/scripts/main.ts \
  --task multispeaker-asr \
  --path "/abs/path/to/meeting.mp4" \
  --audio-part-type input_audio \
  --out-dir ./out/meeting

# 视频整体理解，自定义 prompt
bun run {baseDir}/scripts/main.ts \
  --task understand \
  --path "/abs/path/to/replay.mp4" \
  --prompt "你是 CS2 教练，先描述这一回合的关键事件，再点评经济与道具使用。" \
  --out-dir ./out/cs2-replay

# 视频时间轴 JSON 分析（事件/人物/情绪）
bun run {baseDir}/scripts/main.ts \
  --task video-timeline \
  --url "https://ark-public.tos-cn-beijing.volces.com/carcrash.mp4" --type video \
  --out-dir ./out/timeline

# 为技术博客挑配图关键帧（输出 timestamp + 描述 + 建议图注，自己用 ffmpeg 截图）
bun run {baseDir}/scripts/main.ts \
  --task keyframe-extract \
  --url "https://example.com/talk.mp4" --type video \
  --out-dir ./out/keyframes

# 同上，附带转录稿提升选帧准确度
bun run {baseDir}/scripts/main.ts \
  --task keyframe-extract \
  --path ./talk.mp4 \
  --transcript-file ./talk.transcript.txt \
  --out-dir ./out/keyframes-tx

# AST 翻译成英文
bun run {baseDir}/scripts/main.ts \
  --task ast --url "https://x.com/zh.wav" --type audio \
  --target-language English --audio-part-type input_audio \
  --out-dir ./out/ast

# 字幕打轴（已知文本）
bun run {baseDir}/scripts/main.ts \
  --task subtitle-align --path ./clip.wav \
  --subtitle-file ./clip.txt \
  --audio-part-type input_audio \
  --out-dir ./out/align

Output Layout

out-dir/
  <task>.json          # 完整结构：每段 start/end/text + log_id/response_id/usage + usage_total + merged
  <task>.txt           # 仅 merged 文本，便于直接消费
  cache/
    downloads/         # 远程 URL 下载缓存（按 sha 命名）
    audio/             # ffmpeg 抽出的音频（仅 ASR-only 视频任务）
    segments/          # 切片后的音/视频片段

<task>.json Field quick reference:

  • usage_total.{prompt_tokens,completion_tokens,total_tokens,audio_tokens,cached_tokens,reasoning_tokens}: the total token count accumulated across all segments, for cost accounting.
  • segments[i].usage: the same field as above for a single segment.
  • segments[i].log_id: from the response header x-tt-logid, essential when reporting issues to Volcano technical support.
  • segments[i].response_id / response_model: the id in the response body and the actual underlying model name hit (the endpoint maps to doubao-seed-x-x-xxx）。

Custom Prompt template application

--prompt/--prompt-file is mainly for understand ; the system + user prompts for other tasks are hardcoded to versions consistent with the official demo. To fully override, you can additionally add --system-prompt / --system-prompt-file。

--language is currently just a language hint slot reserved for the prompt (many task templates don't use it). If you need to strictly specify the output language, please state it clearly in --prompt or --system-prompt .

Common Pitfalls

  • When a remote URL is signed, make sure the expiration time is ≥ the expected Ark call duration; the CLI will re-download to local, so the original URL expiring does not affect the Ark call.
  • TOS credential errors are the most common:SignatureDoesNotMatch is most likely TOS_ACCESS_KEY_SECRET with extra/missing characters pasted.
  • understand + video: if you only care about audio, please switch to the corresponding ASR/diarize task, otherwise video tokens will be wasted.
  • Slicing is done evenly by duration. For long-range tasks (>2h), it is recommended to combine with the task-side chunked-summary workflow; do not feed all parts directly to subsequent models.

Prompt References (load on demand)

Each task's original system / user prompt, example output, parsing snippets, and CLI usage are split into separate reference files.Look at this index first, then read the corresponding file once you hit a specific task; do not read them all at once.

File Task One-line purpose
references/asr.md asr Plain-text ASR: English system forces output of only the transcription text, no punctuation/formatting
references/asr-timestamp.md asr-timestamp Character-level timestamp ASR:{start}-{end}-{char}; template
references/multispeaker-asr.md multispeaker-asr Multi-speaker ASR:[spk0]...[spk1]... output in order, no timestamps
references/ast.md ast Speech translation:--target-language controls the target language, returns only the translation
references/diarize.md diarize Speaker diarization:[spkN][start-end]文本, multi-speaker ASR + timestamps
references/subtitle-align.md subtitle-align Subtitle alignment: given the subtitle text, the model outputs only the timestamp of each character
references/caption.md caption Overall audio/video analysis:### 概述/内容分析/说话人/声音事件/视觉信息 markdown report
references/video-timeline.md video-timeline Video timeline JSON:{timeline_events: [{start_end_time, event, people, emotion}]}
references/keyframe-extract.md keyframe-extract Blog illustration keyframe JSON:{keyframes: [{timestamp, timestamp_sec, description, suggested_caption, reason}]}, optional transcript
references/understand.md understand General understanding: user prompt fully customizable, visuals + audio participate together

When to read a reference：

  • Choosing a task: the table above + the ## Tasks table above is enough, no need to read files.
  • Writing parsing code / troubleshooting output format / modifying prompts → read the corresponding reference.
  • Adding a new task → copy a reference, update the task list in prompts.ts + main.ts + the table above.