SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscinematic-video-gen

cinematic-video-gen

Design & creation

View repository

Updated 10-13 16:18

About this skill

Cinematic Video Generation — Character Liveliness Enhancement Engine

Problem Diagnosis

Common problems of AI video generation (doubao-seedance / kling / sora / veo):

  • ❌ Characters look like wax figures, expressions are blank
  • ❌ Movements are mechanical, lacking natural smoothness
  • ❌ Hands are stiff, limbs are uncoordinated
  • ❌ Eyes are empty, lacking emotional expression
  • ❌ The same character is inconsistent across different clips

Core Methodology: Five-Dimensional Liveliness Framework

Dimension 1: Expression Vocabulary

Emotion English Instruction Chinese Mood
Sorrowful "eyes glistening with unshed tears, lips slightly trembling, a bittersweet smile forming" Tears in eyes, lips slightly trembling, bitter smile
Melancholic "gazing into the distance with a melancholic longing, wind tousling her hair, lost in thought" Gazing into the distance with sorrow, wind brushing through hair
Tender "a warm gentle smile spreading across her face, eyes crinkling at the corners, radiating kindness" Warm smile, eyes curved like crescents
Resolute "jaw set with quiet determination, eyes blazing with inner strength, unwavering gaze" Chin slightly raised, gaze firm
Surprised "eyes widening with wonder, mouth forming a soft 'oh', face illuminated with sudden joy" Eyes light up, mouth slightly open
Relieved "shoulders dropping with relief, a peaceful exhale visible, serene contentment" Shoulders relax, smile of relief

Usage Rules: The prompt for each clip must contain at least one expression instruction.

Dimension 2: Body Movement Vocabulary

Body Part Micro-Movement Macro-Movement
Hands "fingers gracefully tracing the air" / "hands gently clasping petals" "arms sweeping wide open" / "reaching toward the light"
Head "head tilting slightly" / "chin lifting gently" "turning to look over her shoulder" / "head thrown back in laughter"
Body "shoulders swaying subtly to the rhythm" / "body swaying gently" "spinning slowly with dress flowing" / "leaning into the wind"
Feet "shifting weight from one foot to another" "walking gracefully through the petals" / "twirling on tiptoe"

Usage Rules: Each clip should have at least 1 micro-movement + 1 macro-movement (for large scenes).

Dimension 3: Camera-Subject Interaction

Camera Movement Character Coordination Effect
slow push-in Character slowly turns toward the camera Intimacy, emotional closeness
tracking shot Character strolls, camera follows Natural and smooth, narrative feel
crane up Character looks up, camera rises Grandeur, spiritual elevation
orbit 360° Character turns gently in place, camera orbits Sense of space, dreamlike feel
handheld close-up Close-up of character's micro-expression Documentary feel, emotional impact
pull back reveal Character remains still, camera pulls away Loneliness, sense of insignificance

Usage Rules: The prompt must specify the camera movement method + the character's natural reaction to that camera movement.

Dimension 4: Character Consistency Anchor

Each MV/video project needs a character card, embedded in all clip prompts:

[CHARACTER_ANCHOR]
Name: Chloe (克洛伊)
Appearance: A young woman in her early 20s, long flowing black hair with subtle cherry blossom pink tips, 
  wearing an ethereal white dress with delicate floral embroidery, slender figure, porcelain skin
Signature features: Always has a small cherry blossom hairpin on the left side, 
  eyes are deep brown with a hint of melancholy
Build: Petite, graceful, dancer-like posture
[/CHARACTER_ANCHOR]

Usage Rules：

  • Insert the character anchor at the beginning of each clip prompt (a condensed version is sufficient)
  • Key features (hair color, clothing, accessories) must be repeated in every clip
  • Tone can change, appearance cannot change

Dimension 5: Temporal Coherence

Action continuity between adjacent clips:

Transition Type Instruction
Continuous Action "continuing her graceful turn from the previous moment"
Emotional Shift "her expression shifts from wonder to quiet contentment"
Positional Movement "having moved from the center to the edge of the frame"
Passage of Time "a moment later, petals have settled around her feet"

Prompt Construction Template

Standard Template (required for every clip)

[角色锚点: Chloe, long black hair with pink tips, white floral dress, cherry blossom hairpin]

[运镜] tracking shot following [角色] as she [动作] through [场景].

[主体动作] She [大动作], her [身体部位] [微动作]. 

[表情] [表情指令库中的具体描述].

[场景/氛围] [场景细节], [光影], [色调].

[动态连贯] [与上一个clip的衔接].

 cinematic quality, 4K, shallow depth of field, natural motion blur, film grain

Example: Fixing Scene 8-1 (Rising from Darkness)

Before (stiff version):

"Slowly rising from deep darkness, a dim silhouette of a young woman emerges..."

After (lively version):

"Chloe, a young woman with long flowing black hair tipped in soft pink, wearing an ethereal white floral dress, her cherry blossom hairpin catching the first rays of light. Slow crane-up shot: she lifts her chin slowly, eyes opening with quiet wonder as warm golden light begins to illuminate her face from below. Her fingers uncurl gracefully at her sides, shoulders relaxing as if releasing a deep breath she's been holding. Her lips part slightly, a soft expression of dawning hope crossing her features. Darkness gently dissolves around her like morning mist. Cinematic chiaroscuro, warm amber and gold palette, 4K, shallow depth of field, natural motion blur."

Model-Specific Optimization

doubao-seedance-1-5-pro

  • ✅ Supports first-frame/last-frame reference images → use the frame with the strongest character consistency as the first frame
  • ⚠️ Action descriptions should not be too complex; one core action + one expression is enough
  • 💡 Keyword weights:graceful > smooth > natural > fluid

sora-2

  • ✅ Supports longer prompts → can describe action sequences in detail
  • ⚠️ Weak character consistency → must fully describe appearance every time
  • 💡 Camera movement instructions work well; use specific camera movements more often

kling series

  • ✅ Strong character action generation → action descriptions can be bold
  • ⚠️ Weak facial expression detail → compensate with lighting and angles
  • 💡 close-up + soft lighting = sense of facial expression

Quality Check Checklist

Check item by item after generation:

  • Does the character's facial expression change? (not a puppet face)
  • Is there natural body movement? (not standing still)
  • Does the camera movement coordinate with the character? (not pure camera movement)
  • Is there action continuity with the previous clip?
  • Is the character's appearance consistent?
  • Does the overall atmosphere fit the musical emotion?

Script

scripts/generate_cinematic_prompt.sh

Input: scene description + character card + context → Output: optimized prompt

scripts/quality_check.py

Input: generated mp4 → Output: quality score (motion amount/expression change/continuity)

Collaboration with Other Skills

cinematic-video-gen (prompt增强 + 口型歌词)
  → qingyun-api / video_generate (实际生成)
  → video-lyrics-subtitle (字幕生成 + 烧录)
  → lyrics-video-sync (歌词卡点匹配)
  → ffmpeg-video-editor (后期合成)
  → av-sync-workflow (音视频节拍同步)

🎬 Complete MV Generation Workflow (End-to-End)

The following is a complete pipeline for an MV from scratch to final cut, with all steps using the existing skill toolchain.

Input Material Preparation

All materials are stored in a standardized directory structure:

~/clawd/projects/MediaClaw/output/<project-name>/
├── lyrics/lyrics.txt              # 歌词原文
├── audio/song_final.mp3           # 最终版音频
├── script/mv_script.json          # MV脚本（场景+镜头列表）
├── frames/                        # 参考帧图片
│   ├── 01_first.png               # 每个场景的首帧
│   ├── 01_last.png                # 每个场景的尾帧
│   └── ...
└── script/chloe-card.json         # 角色卡

Step 1: Lyric Timeline Generation

Tool: video-lyrics-subtitle

python3 ~/clawd/skills/video-lyrics-subtitle/scripts/generate_srt.py \
  lyrics/lyrics.txt \
  --audio audio/song_final.mp3 \
  -o subtitles/lyrics.srt

Output: precise start and end times for each lyric line (SRT format)

Step 1.5: Phase Audio/Subtitle Segment Extraction

⚠️ Critical step! Ensure lyrics/audio/video are synchronized. Skipping this step will cause: audio playing from the beginning, subtitle timeline misalignment, and complete video-audio desynchronization.

Based on edit_plan.json the time range of each Phase, extract the corresponding segments from the complete material:

python3 ~/clawd/projects/MediaClaw/skills/cinematic-video-gen/scripts/extract_phase.py \
  --audio audio/song_final.mp3 \
  --srt subtitles/lyrics.srt \
  --start 208.0 \
  --end 237.0 \
  --phase 1 \
  --outdir .

Output:

  • audio/phaseN_audio.m4a — segmented audio (precisely corresponding to the video time range)
  • subtitles/phaseN_lyrics.srt — segmented subtitles (timeline starts from 00:00:00,000 start)
  • lyrics/phaseN_lyrics.txt — lyric text fragments

SRT timeline reset logic:

  • Iterate through all SRT entries, keep entries that overlap with the Phase time range
  • Crop the start and end times of partially overlapping entries to within the Phase range
  • Subtract the Phase start offset from all times (e.g., -208.0s), so that it starts from 00:00:00,000 start

Step 2: Scene × Lyric Mapping

Tool: lyrics-video-sync

Align the scenes/shots in the MV script with the lyric timeline to determine:

  • Which lyric line each clip corresponds to
  • Division of instrumental sections and interlude sections
  • Time range of each clip

Output:edit_plan.json(lyrics + time + scene for each clip)

Step 3: Prompt assembly (six-dimension enhancement + lip sync)

Tool: cinematic-video-gen (this skill)

For each clip, automatically read:

  1. Character anchor — extract fixed appearance description from the character card
  2. First-frame reference image — frames/XX_first.png
  3. Camera movement + action + expression — matched from MV script scene description + word library
  4. Lyric lip sync — read the corresponding lyric line from edit_plan.json
  5. Scene atmosphere — lighting, color tone, background
  6. Dynamic continuity — connection instruction with the previous clip

Output: complete prompt for each clip

Step 4: Video generation (first-frame reference)

Tool: qingyun-api (doubao-seedance)

python3 ~/clawd/skills/qingyun-api/scripts/qingyun-video-doubao-local.py \
  --prompt "<Step 3 生成的完整prompt>" \
  --first-frame frames/01_first.png \
  --model doubao-seedance-1-5-pro-251215 \
  --output video/clips/01_scene1.mp4

Key:--first-frame Ensure character consistency (base64 local file)

Each clip is about 5 seconds, generated sequentially by scene.

Step 5: Subtitle file generation

Tool: video-lyrics-subtitle

# SRT → ASS（KTV 高亮效果）
python3 ~/clawd/skills/video-lyrics-subtitle/scripts/generate_ass.py \
  subtitles/lyrics.srt \
  --style ktv \
  -o subtitles/lyrics.ass

Step 6: Compilation and synthesis (⚠️ use segmented assets!)

Tool: ffmpeg-video-editor + video-lyrics-subtitle

🔴 Must use the segmented assets extracted in Step 1.5, not the complete audio/subtitles!

  • ✅ Use audio/phaseN_audio.m4a(segmented audio)
  • ❌ Do not use audio/song_final.mp3(complete audio)
  • ✅ Use subtitles/phaseN_lyrics.srt(segmented subtitles, starting from 00:00)
  • ❌ Do not use subtitles/lyrics.srt(complete subtitles)
# 6a. FFmpeg xfade 拼接所有 clips + 分段音频 → 无字幕版
~/tools/ffmpeg -i clip1.mp4 -i clip2.mp4 ... \
  -i audio/phase1_audio.m4a \
  -filter_complex "xfade chain..." \
  -map "[vout]" -map N:a \
  video/phase1_raw.mp4

# 6b. 生成分段 ASS 字幕（从分段 SRT 转换）
python3 ~/clawd/skills/video-lyrics-subtitle/scripts/generate_ass.py \
  subtitles/phase1_lyrics.srt \
  --style ktv \
  -o subtitles/phase1_lyrics.ass

# 6c. 用 skill 脚本烧录字幕 → 最终成片
bash ~/clawd/skills/video-lyrics-subtitle/scripts/burn_subtitles.sh \
  video/phase1_raw.mp4 \
  subtitles/phase1_lyrics.ass \
  video/phase1_final.mp4

Full workflow quick reference chart

lyrics.txt + song.mp3
       ↓
  [Step 1] generate_srt.py → lyrics.srt (歌词时间轴)
       ↓
  [Step 1.5] extract_phase.py → phaseN_audio.m4a + phaseN_lyrics.srt + phaseN_lyrics.txt
       ↓
  [Step 2] lyrics-video-sync → edit_plan.json (场景×歌词映射)
       ↓
  [Step 3] cinematic-video-gen → 6维prompt (含口型歌词指令)
       ↓
  [Step 4] qingyun-video-doubao-local.py → clips/*.mp4 (首帧参考生成)
       ↓
  [Step 5] generate_ass.py → phaseN_lyrics.ass (从分段SRT转换)
       ↓
  [Step 6a] FFmpeg xfade → phaseN_raw.mp4 (拼接 + 分段音频)
       ↓
  [Step 6b] burn_subtitles.sh → phaseN_final.mp4 (烧录分段字幕)
       ↓
  🎬 最终成片 (歌词/音频/视频三者严格同步)

Post-generation quality check

Check immediately after each clip is generated:

  • Is the character's appearance consistent with the reference frame?
  • Is the expression changing (not a puppet face)?
  • Is there natural body movement?
  • Is the mouth moving while singing?
  • Is the action continuous with the previous clip?
  • Does the overall atmosphere fit the music?

If not up to standard → adjust the prompt and regenerate (maximum 3 retries for the same clip)

Model fallback order

1. doubao-seedance-1-5-pro-251215  ← 首选，支持首帧参考
2. kling (xingjiabiapi)            ← 备选，动作生成强
3. veo3.1 (Google AI Studio)       ← 备选，质量高但渠道不稳定

🔴 Dimension 6: Lip-Sync Lyrics instruction

Problem

During AI video generation, the character sings but the mouth does not move and does not match the lyrics, looking like lip-syncing.

Solution: Embed lyric segments into the prompt

The prompt for each clip must include the corresponding time period'slyric text + lip-sync description：

[角色锚点 + 运镜 + 动作 + 表情]

She is singing the lyrics: "桜が散る季節に" (在樱花飘落的季节)
Her mouth forms each syllable naturally, lips moving gracefully with the melody.
She sings with emotion, her voice carrying the weight of the words.

[场景/氛围 + 质量后缀]

Lip-sync description word library

Singing state English instruction
Normal singing "singing softly, lips forming each syllable with gentle precision"
High-note burst "belting out a powerful note, mouth open wide, jaw dropped slightly"
Breathy soft singing "singing in a breathy whisper, lips barely parting, intimate and close"
Pause for breath "a brief pause, catching her breath, lips closing softly before the next phrase"
Emotional outburst "voice cracking with emotion, singing through tears, raw and vulnerable"
Chanting/humming "humming softly with closed lips, a gentle melody vibrating through her"

Lyrics→timeline matching rules

The time range of each clip must be in lyrics.json to find the corresponding lyric line, then:

  1. Has lyrics → write into the prompt She is singing: "[原文]" ([中文翻译])
  2. Instrumental section → write into the prompt She moves gracefully to the music, no singing in this moment
  3. Breath/interlude → write into the prompt Between phrases, she takes a breath, a moment of quiet before the next note

Prompt template (full version)

[Character: Chloe, long black hair with pink tips, white floral dress, cherry blossom hairpin]

[运镜] shot of [Chloe] as she [动作] through [场景].

[主体动作] She [大动作], her [身体部位] [微动作].

[表情] [表情指令].

[歌唱] She is singing: "[对应歌词原文]" ([中文翻译]).
Her lips form each syllable naturally, [口型状态描述].
She pours emotion into every word, [情感力度].

[场景/氛围] [场景细节], [光影], [色调].

[动态连贯] [与上一个clip的衔接].

cinematic quality, 4K, shallow depth of field, natural motion blur, film grain

Example: Scene 9-1 with lyrics version

[Character: Chloe, long black hair with pink tips, white floral dress, cherry blossom hairpin]

Slow dissolve shot: Chloe's silhouette gradually transforms into floating cherry blossom petals.
She is singing the final line: "桜が散る季節に、また逢いましょう" (在樱花飘落的季节，我们再相见)
Her lips softly form each syllable as her body dissolves, singing with bittersweet acceptance.
A gentle smile plays on her face even as she fades, eyes glistening with unshed tears.
The last words linger in the air as petals carry her voice away.
Warm golden light, soft pink and white palette, dreamy atmosphere.
Transitioning from the previous aerial dawn shot into this ethereal dissolution.
cinematic quality, 4K, shallow depth of field, natural motion blur, film grain

Automated Pipeline

lyrics-video-sync (歌词时间轴)
  → cinematic-video-gen (生成带歌词的prompt)
  → qingyun-video-doubao-local.py (首帧参考 + prompt生成视频)
  → ffmpeg-video-editor (合成 + 音频混合)

When lyrics-video-sync outputs edit_plan.json, cinematic-video-gen automatically reads the lyrics line corresponding to each clip, generates the complete prompt (including lip-sync instructions), and then sends it to the video generation API.

Created: 2026-04-10 Updated: 2026-04-10 — Added Dimension 6: Lip-sync lyrics instructions