vlog-auto-edit

AI Agent Automatically Edits Travel Vlogs

Author: nyx Research Institute · GitHub · Bilibili @nyx研究所 · Xiaohongshu @nyx研究所 · X @znyupup_music

Take a bunch of travel footage shot on a phone and use AI to automatically edit it into a complete vlog. Minimal dependencies: ffmpeg + Python (whisper or FunASR + Pillow) + vision API. Only ffmpeg is installed at the system level; everything else is installed via pip.

Trigger Conditions

The user has a batch of travel/daily video footage and wants to automatically edit it into a vlog
The user says "help me edit a video", "automatic editing", "vlog editing"

Prerequisites

1. System Tools

Tool	Purpose	Installation
ffmpeg	Video trimming/encoding/concatenation/frame extraction/volume detection	`brew install ffmpeg` (macOS) / `apt install ffmpeg` (Linux)
Python 3.9+	Script glue	Built into macOS/Linux

2. Python Dependencies (check first, skip if already present)

First check whether the required packages are already installed on the system; do not blindly create a venv:

# 检测 whisper
python3 -c "import whisper; print('✅ whisper已安装:', whisper.__file__)"

# 检测 Pillow
python3 -c "from PIL import Image; print('✅ Pillow已安装')"

⚠ Important: do not use 2>/dev/null to swallow errors! You need to see the actual error to determine whether it is truly not installed or a PATH issue.

⚠ macOS note: Check the packages already installed on the system before deciding whether a venv is needed. If the system python3 can already import whisper and from PIL import Image, just use it directly; do not blindly create a venv and end up unable to find the existing packages.

Only install if the check fails:

# 方案A: 直接装到用户环境（推荐）
pip install openai-whisper Pillow

# 方案B: 如果用户环境有冲突，再用 venv
python3 -m venv .venv && source .venv/bin/activate
pip install openai-whisper Pillow

⚠ Do not create a new venv and reinstall everything for every project! The whisper model file is 1.4GB, and pip install also takes several minutes.

⚠ Choose one of two ASR engines: Whisper or FunASR

This workflow supports two ASR engines. Prefer whisper; if whisper cannot be installed, use FunASR:

	Whisper (OpenAI)	FunASR (Alibaba DAMO Academy)
Installation	`pip install openai-whisper`	`pip install funasr modelscope torchaudio`
Model	medium (1.4GB, downloaded from GitHub)	paraformer-zh (1.05GB, downloaded from ModelScope)
Additional Models	None	VAD model + punctuation model (automatically downloaded on first run, ~50MB total)
Chinese Performance	Good	Good (comparable to whisper medium, slightly more detail)
Speed	14s audio ≈ 5-10s (CPU)	14s audio ≈ 1.9s (CPU, RTF=0.132) ⚡
Timestamp Granularity	Sentence level (one start/end per sentence)	Character level (one timestamp per character)
Model Download Source	GitHub (unstable in China, prone to failure)	ModelScope (extremely fast in China, ~15MB/s)
Hidden Dependencies	torch	torch + torchaudio (⚠ must be installed separately)

Detect installed ASR engines:

python3 -c "import whisper; print('✅ whisper可用')" 2>/dev/null \
  || python3 -c "from funasr import AutoModel; print('✅ funasr可用')" 2>/dev/null \
  || echo "❌ 没有可用的ASR引擎，需要安装一个"

⚠ FunASR installation notes:

You must install torchaudio!pip install funasr modelscope will not automatically install torchaudio, and importing will report No module named 'torchaudio'
Correct installation command:pip install funasr modelscope torchaudio
On first run, 3 models (ASR + VAD + punctuation) will be automatically downloaded from ModelScope, about 1.1GB total, with fast speeds in China
Models are cached in ~/.cache/modelscope/hub/models/iic/ directory
When importing, there will be a urllib3 SSL warning (NotOpenSSLWarning), which can be ignored and does not affect functionality

Title card scheme auto-selection:

ffmpeg -filters 2>&1 | grep drawtext
# 有drawtext → 直接用ffmpeg，不需要Pillow
# 没有 → 用Pillow生成透明PNG再overlay（macOS brew ffmpeg通常没编freetype）

Optional (for use in the reference analysis stage):

which yt-dlp && echo "✅ yt-dlp已安装" || pip install yt-dlp
python3 -c "import scenedetect" 2>/dev/null && echo "✅ scenedetect已安装" || pip install scenedetect

3. Visual understanding model (API)

You need a visual model API that can understand image content, used to analyze the material footage.

Requirements:

Support OpenAI Chat Completions format (messages + image_url)
Support base64 image input (frames extracted locally from material, then encoded and uploaded)
Chinese understanding capability (needs to describe image content, annotate shot types, etc.)

Recommended models:

Model	Cost	Description
Zhipu GLM-4.6V-Flash	Free	Register https://open.bigmodel.cn and it's ready to use, good Chinese understanding
GPT-4o	Paid	Best results
Qwen-VL	Paid	Alibaba Cloud, good Chinese

Call example:

import base64, json, urllib.request

API_URL = 'YOUR_VISION_API_ENDPOINT'   # 替换为你的视觉模型端点
API_KEY='***'               # 替换为你的API Key
MODEL = 'YOUR_MODEL_NAME'             # 替换为你的模型名

with open('frame.jpg', 'rb') as f:
    img_b64 = base64.b64encode(f.read()).decode()

payload = {
    'model': MODEL,
    'messages': [{'role': 'user', 'content': [
        {'type': 'image_url', 'image_url': {'url': f'data:image/jpeg;base64,{img_b64}'}},
        {'type': 'text', 'text': '简洁描述画面内容，标注：镜头类型(远/全/中/近/特写)、拍摄手法(固定/手持/移动)、画面氛围。格式：内容|类型|手法|氛围'}
    ]}],
    'max_tokens': 200
}

req = urllib.request.Request(API_URL, json.dumps(payload).encode(),
    {'Content-Type': 'application/json', 'Authorization': f'Bearer {API_KEY}'})
with urllib.request.urlopen(req, timeout=30) as resp:
    result = json.loads(resp.read())
    print(result['choices'][0]['message']['content'])

⚠ Notes on visual API calls:

Input format: ffmpeg extracts frames as JPEG → base64 encoding → data:image/jpeg;base64,{b64} placed into image_url
Frame extraction size: original resolution is not needed, scaling down to 720p is sufficient (saves upload bandwidth and tokens)
Endpoint distinction: some platforms use different endpoints for visual models and text models, confirm you're using the right one
Rate limiting: easily rate-limited during peak hours, add retry + sleep

Not needed (pitfalls encountered)

❌ Jianying/CapCut — v10.4+ project files are encrypted, cannot be operated programmatically
❌ moviepy — the ffmpeg command line is already sufficient, an extra layer of abstraction is instead inflexible
❌ capcut-cli (GitHub) — based on plaintext JSON, ineffective for the encrypted new versions
❌ ImageMagick — Pillow is already sufficient, no need to install another image tool

Workflow (7 stages)

Stage 1: Material inventory

Goal: understand what material you have.

# 批量获取素材信息
for f in footage/*.{MOV,mp4,MP4}; do
  echo "=== $f ==="
  ffprobe -v quiet -print_format json -show_format -show_streams "$f" \
    | python3 -c "import json,sys; d=json.load(sys.stdin); \
      s=d['streams'][0]; f=d['format']; \
      print(f\"  时长: {float(f['duration']):.1f}s\"); \
      print(f\"  分辨率: {s['width']}x{s['height']}\"); \
      print(f\"  编码: {s['codec_name']}\")"
done

Output a material list: quantity, total duration, resolution distribution, shooting time range.

Stage 2: Reference research (optional but strongly recommended)

Goal: establish an understanding of "what a good vlog looks like". Skipping this step and editing directly will produce poor results.

Steps:

Find 2-3 high-quality vlogs of the same type (Bilibili/YouTube)
yt-dlp Download 720p video + audio
whisper Transcribe narration (extract narrative structure)
scenedetect Detect shot cuts (measure pacing)
ffmpeg Extract keyframes + visual API analysis (understand image composition)

Verified patterns:

Shot pacing: an average of 3-4 seconds per shot is the golden pace
Content ratio: food 40-45% + scenery 30% + people 15% + daily life 15%
Shot types: mainly close-up/medium shots (60-65%), interspersed with wide/long shots (35-40%)
Narrative structure: exciting opening → departure/arrival → connected by location → emotional climax ending
Alternating long and short: short shots (<2s) for montage, long shots (>8s) for narration

Editing tutorial key points:

Find the focus, don't make a running account
Three-act structure: 25% suspense + 50% development + 25% ending
Put the exciting parts in the opening
Emotion accounts for 51% (Walter Murch's six rules)

Stage 3: Three-dimensional material analysis

Goal: Let the AI understand the content of each material. Perform three-dimensional analysis on each material:

a) Audio analysis — ASR transcription (Whisper or FunASR)

# 提取音频（16kHz单声道WAV，两个引擎通用的最佳输入格式）
ffmpeg -i footage/INPUT.MOV -vn -acodec pcm_s16le -ar 16000 -ac 1 /tmp/audio.wav

Option 1: Whisper transcription (preferred)

import whisper, json
model = whisper.load_model('medium')
result = model.transcribe('/tmp/audio.wav', language='zh')
segments = [{"start": s['start'], "end": s['end'], "text": s['text']} for s in result['segments']]
# 输出: [{"start": 0.5, "end": 2.3, "text": "我们现在在中央大街"}, ...]

Option 2: FunASR transcription (alternative when Whisper cannot be installed)

import warnings
warnings.filterwarnings("ignore")
from funasr import AutoModel

model = AutoModel(
    model="paraformer-zh",      # Paraformer-large，中文ASR主模型(1.05GB)
    vad_model="fsmn-vad",       # 语音活动检测，自动切分静音段
    punc_model="ct-punc",       # 自动加标点
    log_level="ERROR",
    disable_update=True         # 跳过版本检查，加快启动
)

result = model.generate(input="/tmp/audio.wav")
text = result[0]["text"]        # 完整识别文本（已加标点）
timestamps = result[0].get("timestamp", [])  # 字级别时间戳 [[start_ms, end_ms], ...]

# ⚠ FunASR时间戳是字级别(毫秒)，需要聚合成句级别才能和whisper格式统一
# 聚合策略：按标点符号（。！？，）切分成句子
segments = []
if timestamps and text:
    puncs = set("。！？，,.!?")
    current_start = timestamps[0][0] / 1000.0
    current_text = ""
    chars = list(text)
    for i, (char, ts) in enumerate(zip(chars, timestamps)):
        current_text += char
        if char in puncs or i == len(chars) - 1:
            segments.append({
                "start": round(current_start, 1),
                "end": round(ts[1] / 1000.0, 1),
                "text": current_text.strip()
            })
            if i < len(chars) - 1:
                current_start = timestamps[i + 1][0] / 1000.0
            current_text = ""
# 输出格式与whisper一致: [{"start": 0.6, "end": 2.9, "text": "我们现在在中央大街的入口，"}, ...]

⚠ Note on FunASR output differences:

FunASR returns the full text + character-level timestamps, unlike whisper which directly provides sentence-level segments
The aggregation code above splits character-level timestamps into sentences by punctuation, aligning the output format with whisper
If the material has no speech, FunASR returns empty text "", and timestamps is empty []
FunASR is 3-5x faster than whisper (14s of audio takes only 1.9s), a clear advantage for batch processing

b) Volume detection — ffmpeg

ffmpeg -i footage/INPUT.MOV -af volumedetect -f null /dev/null 2>&1 | grep volume
# mean_volume: 平均音量(dB)  max_volume: 峰值音量(dB)
# mean < -40dB 基本无声  mean > -20dB 有明显声音/语音

c) Visual analysis — ffmpeg frame extraction + visual understanding model

# 抽帧策略（按时长分段）
# ≤20s → 3帧(首/中/尾)
# 20-60s → 5帧
# >60s → 每15s一帧

# 抽首/中/尾三帧示例（缩到720p节省带宽）
ffmpeg -i footage/INPUT.MOV -vf "select=eq(n\,0),scale=1280:-1" -frames:v 1 -q:v 2 frame_start.jpg
ffmpeg -i footage/INPUT.MOV -vf "select=eq(n\,MIDDLE),scale=1280:-1" -frames:v 1 -q:v 2 frame_mid.jpg
ffmpeg -i footage/INPUT.MOV -vf "reverse,scale=1280:-1" -frames:v 1 -q:v 2 frame_end.jpg

d) Output format(one JSON per material)

{
  "filename": "clip_001.MOV",
  "duration": 15.3,
  "resolution": "1920x1080",
  "visual": [
    {"time": "0:00", "description": "...", "shot_type": "近景", "camera": "手持", "mood": "温馨"}
  ],
  "audio": {
    "has_speech": true,
    "mean_volume_db": -20.5,
    "max_volume_db": -3.2,
    "transcript": [{"start": 0.5, "end": 2.3, "text": "..."}]
  }
}

Stage 3.5: Material refinement preprocessing

Goal: Before giving to the LLM, automatically annotate each material's "recommended effective interval", so the LLM can focus on narrative arrangement.

Why this step is needed: Raw phone footage commonly has: recording commands at the start, the same sentence repeated multiple times, phone-holding shake in the first 1-2s, and long meaningless footage after finishing speaking. If this raw data is given directly to the LLM, the plan output by the LLM still needs to be patched item by item.

Preprocessing does four things:

a) Recording command detection

RECORDING_CUES = [
    "开始了", "好了开始", "开始录了", "好了 开始",
    "走了", "好了", "来了",  # 仅在前3s内出现时算口令
]

def detect_cues(transcript_segments, duration):
    """检测录制口令，返回skip zones"""
    skip_zones = []
    for seg in transcript_segments:
        text = seg['text'].strip()
        if seg['start'] < 3.0 and any(text.startswith(c) or text == c for c in RECORDING_CUES):
            skip_zones.append({
                'type': 'recording_cue',
                'range': [seg['start'], seg['end']],
                'text': text,
                'action': 'skip'
            })
        if seg['end'] > duration - 3.0 and text in ["好了", "走了", "好了 走"]:
            skip_zones.append({
                'type': 'recording_cue',
                'range': [seg['start'], seg['end']],
                'text': text,
                'action': 'skip'
            })
    return skip_zones

b) Repeated speech detection

from difflib import SequenceMatcher

def detect_repeats(segments, threshold=0.6):
    """检测重复语音，标记建议跳过的重复段"""
    repeats = []
    for i, a in enumerate(segments):
        for j, b in enumerate(segments):
            if j <= i: continue
            ratio = SequenceMatcher(None, a['text'], b['text']).ratio()
            if ratio >= threshold and len(a['text']) > 4:
                skip = a if len(a['text']) <= len(b['text']) else b
                repeats.append({
                    'type': 'repeat',
                    'range': [skip['start'], skip['end']],
                    'text': skip['text'],
                    'kept': b['text'] if skip == a else a['text'],
                    'action': 'suggest_skip'
                })
    return repeats

c) Start shake / ending drag detection

def detect_trim_points(segments, duration, mean_volume_db):
    """基于语音+音量推荐起止点"""
    suggested_start = 0.0
    suggested_end = duration

    if segments:
        first_speech = segments[0]['start']
        last_speech_end = segments[-1]['end']
        if first_speech > 2.0:
            suggested_start = max(0, first_speech - 0.5)
        if duration - last_speech_end > 3.0:
            suggested_end = last_speech_end + 1.5
    else:
        if duration > 5.0:
            suggested_start = 1.0

    return suggested_start, suggested_end

d) Pure visual duration limit

Mode	Pure visual upper limit	Command handling	Repetition handling	Start shake
strict	8-10s	Remove all	Keep only 1 instance	Remove 1.5s
normal	12-15s	Remove clear commands	Remove obvious repetitions	Remove 1.0s
loose	20-25s	Only remove "started"	Keep most	Remove 0.5s

e) Output format

Append the preprocessing results to the material analysis data:

━━━ clip_012.mp4 | 时长25.8s | 1920x1080 ━━━
  【画面】...
  【语音】...
  【精修建议】模式: normal
    原始区间: [0.0 - 25.8]
    推荐区间: [2.4 - 25.8]
    ⊘ [0.0-2.4] 跳过: 录制口令 "开始了"
    有效语音: [2.62-25.8]

⚠ Important principle: Preprocessing only annotates and suggests, it does not hard-trim. The LLM retains the final decision.

Stage 4: LLM narrative arrangement

Goal: Give the preprocessed material data to the LLM to generate an editing plan.

Input: material analysis + refinement suggestions (recommended intervals) + reference research conclusions Output: editing plan JSON (structure see schema below)

edit_plan JSON Schema

{
  "title": "视频标题",
  "structure": [
    {
      "section": "段落名 — 副标题",
      "description": "本段落内容概述",
      "clips": [
        {
          "file": "素材文件名.mp4",
          "start": 0.0,
          "end": 12.0,
          "note": "画面内容简述",
          "subtitle": "保留的语音文字"
        }
      ]
    }
  ],
  "bgm_suggestion": "BGM风格建议（含genre/mood/instruments/tempo/bpm）",
  "editing_notes": "整体剪辑说明"
}

Constraints:

The same material can appear in multiple clips (different time intervals)
start/end cannot cut off the middle of speech (Stage 4.5 will validate)
The first section is recommended to be an "opening" type introductory paragraph
The last section is recommended to be a "closing" type paragraph

For the detailed prompt template see templates/edit_plan_prompt.md。

Stage 4+: Storyboard visualization + Dashboard (browser preview)

Goal: Turn the LLM's editing plan into a webpage with both images and text, making it convenient for users to review intuitively.

The Dashboard contains two panels:

📋 Material overview — Material thumbnail grid + duration/speech/volume annotations + filters (All/Used/Unused/With speech) + click to view details
✅ Final product quality check — Periodic frame extraction from the final video (every 30 seconds) + segment locating + click to enlarge

Use gen_dashboard.py script:

# 最小用法（只生成素材面板）
python3 scripts/gen_dashboard.py \
  --analysis clip_analysis.json \
  --plan edit_plan.json \
  --footage footage/ \
  --out output/

# 完整两面板（含成品QC帧）
python3 scripts/gen_dashboard.py \
  --analysis clip_analysis.json \
  --plan edit_plan.json \
  --footage footage/ \
  --video output/final.mp4 \
  --out output/

Supports two analysis data formats (auto-detected):

Standard format clip_analysis.json: {filename, duration, visual: [...], audio: {...}}
Compact format clips_compact.json: {file, dur, visual: "string", speech: [...]}

After the user confirms the storyboard plan, proceed to the rendering stage.

Stage 4.5: Speech truncation validation

Goal: Automatically validate the plan output by the LLM to ensure no speech is truncated in the middle.

def validate_speech(plan_clips, speech_data):
    """检查每个clip的start/end是否截断了语音"""
    issues = []
    for clip in plan_clips:
        f = clip['file']
        cs, ce = clip['start'], clip['end']
        if f not in speech_data: continue
        for ss, se, txt in speech_data[f]:
            if ss < ce and se > cs:  # 语音与clip有交集
                if ss < cs - 0.3:    # 语音开始在clip之前
                    issues.append(f"截断开头: {f} clip从{cs}开始，但语音\"{txt}\"从{ss}开始")
                if se > ce + 0.5:    # 语音结束在clip之后
                    issues.append(f"截断结尾: {f} clip在{ce}结束，但语音\"{txt}\"到{se}结束")
    return issues

When validation fails: automatically fix start/end, no need to go back to the LLM for re-arrangement.

Automatic fix logic

def fix_speech_cuts(plan, speech_data, margin=0.3):
    """
    自动修复语音截断问题。直接修改plan中的start/end。
    
    参数:
        plan: edit_plan dict (会被原地修改)
        speech_data: dict, {filename: [(start, end, text), ...]}
        margin: float, 语音边界容差(秒)
    
    返回:
        fixes: list of str, 修复日志
    """
    fixes = []
    
    for sec in plan["structure"]:
        for clip in sec["clips"]:
            f = clip["file"]
            cs = float(clip["start"])
            ce = float(clip["end"])
            
            if f not in speech_data:
                continue
            
            for ss, se, txt in speech_data[f]:
                if ss >= ce or se <= cs:
                    continue
                
                # 开头截断：clip.start > speech.start + margin
                if ss < cs - margin and se > cs:
                    old_start = cs
                    new_start = max(0, ss - 0.1)
                    clip["start"] = round(new_start, 1)
                    fixes.append(
                        f"  修复开头: {f} [{old_start}→{new_start}] "
                        f"语音\"{txt[:20]}\"从{ss}s开始"
                    )
                    cs = new_start
                
                # 结尾截断：clip.end < speech.end - margin
                if se > ce + margin and ss < ce:
                    old_end = ce
                    new_end = se + 0.2
                    clip["end"] = round(new_end, 1)
                    fixes.append(
                        f"  修复结尾: {f} [{old_end}→{new_end}] "
                        f"语音\"{txt[:20]}\"到{se}s结束"
                    )
                    ce = new_end
    
    return fixes


def validate_and_fix(plan, speech_data):
    """
    端到端校验+修复入口。
    
    用法:
        import json
        # 加载plan
        plan = json.load(open('edit_plan.json'))
        # 加载语音数据 (格式: {filename: [{start, end, text}, ...]})
        raw = json.load(open('clip_analysis.json'))
        speech_data = {}
        for clip_info in raw:
            fname = clip_info['filename']
            if clip_info['audio']['has_speech']:
                speech_data[fname] = [
                    (s['start'], s['end'], s['text'])
                    for s in clip_info['audio']['transcript']
                ]
        
        # 校验
        all_clips = [c for sec in plan['structure'] for c in sec['clips']]
        issues = validate_speech(all_clips, speech_data)
        print(f'发现 {len(issues)} 个截断问题')
        
        # 修复
        fixes = fix_speech_cuts(plan, speech_data)
        print(f'自动修复 {len(fixes)} 处')
        
        # 二次校验确认
        all_clips2 = [c for sec in plan['structure'] for c in sec['clips']]
        issues2 = validate_speech(all_clips2, speech_data)
        assert len(issues2) == 0, f'仍有 {len(issues2)} 个问题未修复!'
        print('✅ 二次校验通过')
        
        # 保存修复后的plan
        json.dump(plan, open('edit_plan_fixed.json', 'w', ensure_ascii=False), indent=2)
    """
    all_clips = [c for sec in plan['structure'] for c in sec['clips']]
    issues = validate_speech(all_clips, speech_data)
    fixes = fix_speech_cuts(plan, speech_data)
    return issues, fixes

Fix strategy:

Truncation type	Detection condition	Fix method
Beginning truncation	clip.start > speech.start + margin	clip.start → speech.start - 0.1s
End truncation	clip.end < speech.end - margin	clip.end → speech.end + 0.2s

Boundary tolerance (margin=0.3s): Whisper/FunASR timestamps have ±0.2-0.3s error. After fixing, a second validation must be run.

Stage 5: ffmpeg automatic execution

Goal: Convert the editing plan into ffmpeg commands and execute them.

Rendering strategy: clip segment by segment → concatenate

# 单个片段裁剪+编码
ffmpeg -y -ss 00:00:02.400 -i footage/INPUT.mp4 -t 00:00:23.400 \
  -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,setsar=1" \
  -r 30 -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p \
  -c:a aac -b:a 128k -ar 44100 -ac 2 \
  -movflags +faststart segments/seg_001.mp4

# 拼接成品
ffmpeg -y -f concat -safe 0 -i concat.txt -c copy -movflags +faststart output.mp4

Speed change

# 2倍速（视频+音频同步）
ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" -filter:a "atempo=2.0" out.mp4

Opening highlight montage

Select the most visually striking shots from each segment and place them at the very beginning of the video with fast-cut splicing.

Shot selection principles:

Select 1 highlight frame from each main segment
Each shot 0.5-1 second, fast-cut rhythm
6-10 shots in total, total duration 5-8 seconds
Exclude vertical-screen materials
No audio (-an), ultimately covered by BGM

import subprocess, os

highlights = [
    ('footage/clip_a.mp4',   1.5, 2.2),
    ('footage/clip_b.mp4',   2.0, 2.7),
    # ... 每个0.7秒左右
]

for i, (f, start, end) in enumerate(highlights, 1):
    dur = end - start
    cmd = f'ffmpeg -y -ss {start} -i "{f}" -t {dur} ' \
          f'-vf "scale=1920:1080:force_original_aspect_ratio=decrease,' \
          f'pad=1920:1080:(ow-iw)/2:(oh-ih)/2" ' \
          f'-c:v libx264 -preset ultrafast -crf 23 -pix_fmt yuv420p ' \
          f'-an highlights/hl_{i}.mp4'
    subprocess.run(cmd, shell=True)

# concat
with open('highlights/concat.txt', 'w') as f:
    for i in range(1, len(highlights)+1):
        f.write(f"file 'hl_{i}.mp4'\n")

subprocess.run('ffmpeg -y -f concat -safe 0 -i highlights/concat.txt '
               '-c copy highlights/montage.mp4', shell=True)

Segment title overlay

Use Pillow to generate a transparent RGBA PNG, and overlay it onto the video frame with ffmpeg.Do not use black-background title cards。

from PIL import Image, ImageDraw, ImageFont, ImageFilter

W, H = 1920, 1080
title = "段落标题"

img = Image.new("RGBA", (W, H), (0, 0, 0, 0))
draw = ImageDraw.Draw(img)

# 字体：macOS用冬青黑体，Linux用Noto Sans CJK
# font = ImageFont.truetype("/System/Library/Fonts/Hiragino Sans GB.ttc", 60)  # macOS
# font = ImageFont.truetype("/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc", 60)  # Linux
font = ImageFont.truetype("YOUR_FONT_PATH", 60)

bbox = draw.textbbox((0, 0), title, font=font)
tw, th = bbox[2] - bbox[0], bbox[3] - bbox[1]
x, y = (W - tw) // 2, (H - th) // 2

# 柔和阴影
shadow = Image.new("RGBA", (W, H), (0, 0, 0, 0))
sd = ImageDraw.Draw(shadow)
sd.text((x+2, y+2), title, fill=(0, 0, 0, 180), font=font)
shadow = shadow.filter(ImageFilter.GaussianBlur(radius=6))

img = Image.alpha_composite(img, shadow)
draw = ImageDraw.Draw(img)
draw.text((x, y), title, fill=(255, 255, 255, 240), font=font)
img.save("title_overlay.png")

Overlay onto the video:

# 标题显示前3秒
ffmpeg -y -i section.mp4 -i title.png \
  -filter_complex "overlay=0:0:enable='between(t,0,3)'" \
  -c:v libx264 -preset medium -crf 18 -pix_fmt yuv420p \
  -c:a aac -b:a 128k -movflags +faststart \
  section_titled.mp4

⚠ Key overlay Pitfall:

❌ Do not add shortest=1 — PNG has only 1 frame, which will cause the video stream to be truncated immediately
✅ Directly use overlay=0:0:enable='between(t,0,3)' simplest and most reliable
✅ First validate the overlay effect with a short clip (5-8s)

Segment concatenation (recommended solution)

✅ Correct solution: encode segment by segment → concat copy

echo "file 'final_sec_01.mp4'" > concat.txt
echo "file 'final_sec_02.mp4'" >> concat.txt
ffmpeg -y -f concat -safe 0 -i concat.txt -c copy -movflags +faststart output.mp4

❌ Do not use chained xfade merging: Chained xfade will cause cumulative frame loss, and the latter half will be all black.

Transitions between segments: Hard cuts within segments are sufficient. If gradients are needed, adding fade at the start and end of each segment is more reliable than xfade.

BGM generation and mixing (optional)

BGM can be added manually, or with AI music generation tools (such as MiniMax music, Suno, etc.).

a) AI-generated BGM

Style mapping reference (commonly used for vlogs):

Vlog style	genre	mood	instruments	tempo
Relaxed daily	indie pop	cheerful, carefree	ukulele, acoustic guitar	moderate
Artsy healing	folk, acoustic	warm, gentle	acoustic guitar, piano, strings	slow
Food exploration	jazz, bossa nova	cozy, playful	piano, upright bass	moderate
Ice/snow/winter	cinematic, ambient	serene, majestic	piano, celesta, strings	slow
Tropical/island	tropical house	sunny, relaxed	steel drums, marimba	upbeat
City exploration	lo-fi hip hop	chill, urban	keys, vinyl crackle	moderate

b) Mix into the video

# BGM 音量 10-15%（有语音的vlog要压低BGM）
ffmpeg -y -i video_no_bgm.mp4 -i bgm.m4a \
  -filter_complex "[0:a]volume=1.0[orig];[1:a]volume=0.12[bgm];[orig][bgm]amix=inputs=2:duration=first:dropout_transition=2[aout]" \
  -map 0:v -map "[aout]" -c:v copy -c:a aac -b:a 192k \
  -movflags +faststart output/final.mp4

BGM volume suggestions:

Lots of speech/narration → 8-12%
Little speech, mainly empty shots → 15-25%
Pure empty-shot montage segments → 30-40%

Speed change

ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" -filter:a "atempo=2.0" out.mp4

Pitfalls

CapCut unavailable — v10.4+ encrypts draft_info.json, don't waste time
Vision API endpoints must be distinguished — Some platforms use different endpoints for vision models and text models, confirm you're using the right one
Whisper model selection — base is too poor for Chinese, large is too slow, medium is the best value
Don't skip reference research — An AI without a "good vlog standard" will cut out a流水账
ffmpeg encoding compatibility — When concatenating, pay attention to source material encoding consistency; unify each clip's encoding first, then concat
Vision API rate limiting — Rate limiting is easy during peak hours, add retry and sleep
Do preprocessing before the LLM — Filler words/repetition/shake detection handled in stage 3.5
Preprocessing is a suggestion, not hard trimming — The LLM retains override authority; some filler words may have narrative value
Speech truncation must be automatically validated — The plan output by the LLM may cut in the middle of speech
Whisper hallucination — Pure music/ambient sound segments will cause Whisper to fabricate text, filter with a volume threshold (-40dB)
Vertical mixed with horizontal — Must be detected during the material analysis stage; vertical (9:16) mixed into horizontal is jarring, recommend removing
Don't add shortest=1 to overlay — PNG has only 1 frame, shortest=1 will truncate the video stream
Title card style — White text + soft shadow overlaid on the video frame,Don't use black-background title cards
ffmpeg drawtext unavailable — macOS Homebrew ffmpeg isn't compiled with freetype by default, use Pillow to generate PNG overlay instead
HEVC warnings can be ignored — "Error constructing the frame RPS" is an HEVC decoding warning, doesn't affect output
Non-monotonic DTS — Common timestamp warning during concat; playback is unaffected
Empty shots shouldn't be too long — Pure visuals without speech >6s is too long, recommend compressing to 5-8s
Chained xfade merging is unreliable — Chained xfade causes cumulative frame loss, the latter half goes all black, use concat instead
Validate with short clips first — Every time you change the rendering approach, first validate the effect with a 5-8 second short clip
Must verify frame count after rendering — ffmpeg -i output.mp4 -f null - Check the actual frame count
Don't make BGM too loud — For segments with speech, keep BGM at 8-12%, listen before deciding
BGM transitions between multiple segments — Keep similar key and tempo (bpm difference <20)
Must listen after mixing — Original volume varies greatly across different materials
Don't repeatedly install dependencies — Detect first then install whisper/Pillow etc., don't create a new venv every time
FunASR must have torchaudio installed — pip install funasr modelscope isn't enough, must pip install funasr modelscope torchaudio, otherwise import reports No module named 'torchaudio'
FunASR timestamps are character-level — Unlike whisper which directly gives sentence-level segments, FunASR returns [start_ms, end_ms] for each character, which needs to be aggregated into sentences by punctuation before feeding into the subsequent speech truncation validation (stage 4.5)
FunASR hallucinates less than whisper — With pure ambient sound/music, FunASR usually returns empty text, unlike Whisper which fabricates content. However, a volume threshold (-40dB) is still needed for secondary confirmation

Encoding and concatenation (newly added in the video tutorial hands-on section)

⚠️ After atempo speed-up, audio and video durations are inconsistent — After atempo processing, the audio is 0.4-0.8 seconds shorter than the video per segment. With 8 segments, the cumulative difference is several seconds, and in the latter half the audio and video are completely out of sync.After each segment is edited, the audio-video duration difference must be verified immediately to be <0.05s, and if exceeded, use apad to pad it out:

# 验证
ffprobe -show_entries stream=duration -select_streams v -of csv=p=0 seg.mp4  # 视频时长
ffprobe -show_entries stream=duration -select_streams a -of csv=p=0 seg.mp4  # 音频时长
# 补齐
ffmpeg -y -i seg.mp4 -af "apad" -c:v copy -c:a aac -shortest seg_fixed.mp4

⚠️ concat copy audio timestamps are discontinuous — After multi-segment concat copy, the audio crackles / CapCut audio-video offset occurs.Correct approach: extract audio separately as wav + pad with apad + concatenate with the concat filter, concatenate video separately with concat copy, and finally merge them：

# 每段提取音频（apad补齐到视频时长）
ffmpeg -y -i seg.mp4 -vn -af "apad" -ar 48000 -ac 2 -c:a pcm_s16le -t {视频时长} seg.wav
# concat filter拼接音频（不是demuxer）
ffmpeg -y -i a.wav -i b.wav -filter_complex "[0:a][1:a]concat=n=2:v=0:a=1[aout]" -map "[aout]" -c:a aac audio.m4a
# 合并
ffmpeg -y -i video.mp4 -i audio.m4a -c:v copy -c:a copy output.mp4

CapCut-compatible encoding — Fixing audio-video desync when importing into CapCut:-r 30 -vsync cfr(fixed frame rate) + -x264-params "bframes=0"(no B-frames) + -ar 48000 -ac 2(unified audio)
The crf value should be low for screen-recording content — Screen recordings have dense text, and crf 22 may not be enough bitrate and will look blurry. For screen recordings use-crf 18or-b:v 3M. For general vlog footage, crf 22 is fine
Cumulative quality loss from multiple re-encodings — Editing → concatenation → adding background → overlaying progress bar, each encoding loses quality.Try to get it done in a single filter chain, reducing the number of encodings
Background ffmpeg process leakage — When killing a background task, it may kill only the shell and not the ffmpeg child process. After each kill,ps aux | grep ffmpegverify, and for leftovers usekill -9 PID
The first thing to do before starting is to confirm the output resolution — Keep the resolution unified throughout the whole workflow; do not change it midway. Use 1080p for screen-recording content (dense text is sensitive to resolution), and use the original resolution or 1080p for vlog footage
⚠️ When overlaying video, the audio is overwritten/lost — When using filter_complex overlay to overlay videos such as progress bars/watermarks, ffmpeg may automatically select the audio of the overlay source (silent) instead of the audio of the original video.You must use-mapto explicitly specify the audio source：

# ❌ 不指定map，音频可能来自overlay源（静音）
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex "overlay=0:0" output.mp4

# ✅ 明确map：视频取filter输出，音频取原视频
ffmpeg -i main.mp4 -i overlay.mp4 \
  -filter_complex "[0:v][1:v]overlay=0:0[vout]" \
  -map "[vout]" -map 0:a \
  -c:v libx264 -c:a copy output.mp4

If it still does not work (audio codec issue),doing it step by step is more reliable：

# Step1: 只处理视频（-an去掉音频）
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex "overlay" -map "[vout]" -an video_only.mp4
# Step2: 把原始音频合回来
ffmpeg -i video_only.mp4 -i main.mp4 -map 0:v -map 1:a -c:v copy -c:a copy final.mp4

⚠️ -shortest will truncate audio — When the video and audio durations are not exactly the same,-shortestwill drop the last few seconds of audio.Prefer using-t {精确时长}instead, or make sure the audio and video are equal in length and then do not add -shortest

End-to-end example

Project file structure

my-vlog-project/
├── footage/                    # 原始素材（用户提供）
│   ├── clip_001.MOV
│   └── ...
│
├── analysis/                   # 阶段3输出
│   └── clip_analysis.json
│
├── edit_plan.json              # 阶段4输出
├── edit_plan_fixed.json        # 阶段4.5输出
│
├── output/
│   ├── dashboard.html          # Dashboard网页
│   ├── thumbnails/             # 素材缩略图
│   ├── qc_frames/              # 成品QC帧
│   ├── segments/               # 裁剪后的片段
│   ├── highlights/             # 片头高光蒙太奇
│   ├── titles/                 # 段落标题PNG
│   ├── sections/               # 带标题的段落视频
│   ├── bgm/                    # BGM文件
│   ├── video_no_bgm.mp4        # 无BGM的完整视频
│   └── final.mp4               # 🎬 成品视频
│
└── reference/                  # 阶段2输出（可选）

Intermediate artifact examples

See examples/ the example files under the directory.

Execution workflow quick reference

素材文件夹 footage/
     │
     ▼
 阶段1: 素材盘点 → ffprobe批量扫描 → 清单
     │
     ▼
 阶段2: 参考研究 → 下载优质vlog → 分析节奏（可选）
     │
     ▼
 阶段3: 素材三维分析 → whisper/FunASR+音量+视觉 → clip_analysis.json
     │
     ▼
 阶段3.5: 精修预处理 → 口令/重复/晃动/纯画面标注
     │
     ▼
 阶段4: LLM叙事编排 → edit_plan.json
     │
     ▼
 阶段4+: Dashboard可视化 → 浏览器预览确认
     │
     ▼
 阶段4.5: 语音截断校验 → 自动修复
     │
     ▼
 阶段5: ffmpeg渲染 → 裁剪→标题→拼接→BGM → final.mp4 🎬

Verify

Material inventory completed, manifest generated
Reference research completed, patterns extracted
Three-dimensional material analysis completed (visual + audio + volume)
Material fine-editing preprocessing completed
LLM editing plan generated
Speech truncation check passed
Audio-video duration difference per segment <0.05s
ffmpeg executed successfully, final video output
Final audio is complete (not truncated by -shortest)
Drag to the latter half of the video to check audio-video sync
Final video quality manual confirmation

Author: nyx Research Institute · GitHub · Bilibili · Xiaohongshu @nyx Research Institute License: MIT

About this skill

Trigger Conditions

Prerequisites

1. System Tools

2. Python Dependencies (check first, skip if already present)

3. Visual understanding model (API)

Not needed (pitfalls encountered)

Workflow (7 stages)

Stage 1: Material inventory

Stage 2: Reference research (optional but strongly recommended)

Stage 3: Three-dimensional material analysis

Stage 3.5: Material refinement preprocessing

a) Recording command detection

b) Repeated speech detection

c) Start shake / ending drag detection

d) Pure visual duration limit

e) Output format

Stage 4: LLM narrative arrangement

edit_plan JSON Schema

Stage 4+: Storyboard visualization + Dashboard (browser preview)

Stage 4.5: Speech truncation validation

Automatic fix logic

Stage 5: ffmpeg automatic execution

Rendering strategy: clip segment by segment → concatenate

Speed change

Opening highlight montage

Segment title overlay

Segment concatenation (recommended solution)

BGM generation and mixing (optional)

a) AI-generated BGM

b) Mix into the video

Speed change

Pitfalls

Encoding and concatenation (newly added in the video tutorial hands-on section)

End-to-end example

Project file structure

Intermediate artifact examples

Execution workflow quick reference

Verify