SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsgenerate-video

generate-video

Design & creation

View repository

Updated 10-15 17:17

About this skill

Generate Video

Use the Veo 3.1 API to create a video for each scene/clip, using the storyboard image as the starting frame.

Content Mode Support

The system supports two content modes, and the video specifications are automatically adjusted according to the mode:

Dimension Storytelling + Visuals Mode (Default) Series Animation Mode
Aspect Ratio 9:16 Portrait 16:9 Landscape
Default Duration 4 seconds 8 seconds
Data Structure segments array scenes array
Prompt Construction build_segment_prompt() build_scene_prompt()
Video Prompt Only character dialogue (if any), no narration Dialogue, narration, sound effects

The aspect ratio is set via API parameters and is not included in the prompt. Duration only supports 4s / 6s / 8s options.

Generation Modes

Concurrency Note: The default concurrency comes from the environment variable VIDEO_MAX_WORKERS(default 2), and can also be overridden with --max-workers .

1. Standard Mode (Recommended)

Generate a video independently for each scene, then concatenate them using ffmpeg:

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --episode 1

Features:

  • Each scene uses the storyboard image as the starting frame to ensure character consistency
  • Scenes are concatenated using ffmpeg
  • Supports resuming from breakpoints
  • Unsatisfactory scenes can be regenerated individually

2. Resume from Breakpoints

If generation is interrupted, you can continue from the last checkpoint:

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --episode 1 --resume

3. Single Scene Mode

Generate a video for a single scene (for testing or regeneration):

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --scene E1S1

4. Batch Independent Mode

Independently generate all pending scenes:

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --all

5. Batch Custom Selection Mode

Generate videos for multiple specified scenes:

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --scenes E1S01,E1S05,E1S10

Combined with resume from breakpoints:

python .claude/skills/generate-video/scripts/generate_video.py \
    my_project script.json --scenes E1S01,E1S05,E1S10 --resume

Features:

  • Specify multiple scene IDs separated by commas
  • Supports resuming from breakpoints

Veo 3.1 Technical Reference

Feature Description
Image-to-Video Use the storyboard image as the starting frame
Single Clip/Scene Duration Storytelling mode defaults to 4 seconds, series animation mode defaults to 8 seconds
Duration Options Only supports 4s / 6s / 8s
Resolution 1080p (Default)
Aspect Ratio Storytelling mode 9:16 Portrait, series animation mode 16:9 Landscape

About the extend Feature

Veo 3.1's extend featureis only suitable for extending a single clip/scene(for example, to let one action continue), and is not suitable for connecting different shots.

If a certain clip/scene needs a longer duration, you can use extend:

  • Each extension adds a fixed +7 seconds
  • Can be extended up to 148 seconds
  • The starting frame cannot be changed during extension

⚠️ Important Resolution Restriction：

  • The extend feature only supports videos at 720p resolution
  • Videos at 1080p or higher resolution cannot use extend
  • To extend, the initial video must be generated at 720p resolution

⚠️ Not recommended for multi-shot sequences: use ffmpeg concatenation between different scenes instead of extend.

Workflow

  1. Load project and script

    • If no project name is specified, ask the user
    • From projects/{项目名}/scripts/ load the script
    • Confirm all scenes have storyboard_image

  2. Generate video

    • Iterate over all scenes of the specified episode
    • Each scene uses the storyboard image as the starting frame to generate a video independently
    • Save checkpoint to support resuming from a breakpoint

  3. Update script

    • Update video_clip path
    • Update scene status to completed

Video Prompt Generation

Prompts are automatically constructed by the functions in generate_video.py , selecting a different builder based on the content mode:

  • Storytelling mode: build_segment_prompt() - Does not include novel_text(narration), only includes character dialogue (if any)
  • Episodic animation mode: build_scene_prompt() - Includes dialogue, narration, sound effects

Prompt Structure

According to Veo official guidance and the latest data structure, the prompt is divided into two parts:

image_prompt (used for storyboard image generation)

{
  "scene": "[场景环境描述：包含地点、氛围、关键视觉元素的叙述性描述]",
  "composition": {
    "shot_type": "[镜头类型：Long Shot / Medium Shot / Close-up / Extreme Close-up 等]",
    "lighting": "[光线描述：光源方向、色温、阴影特征]",
    "ambiance": "[氛围描述：色调、情绪、环境效果]"
  }
}

video_prompt (used for video generation)

{
  "action": "[动作描述：主体在场景时长内的具体动作、姿态、表情变化]",
  "camera_motion": "[摄像机运动：Static / Pan Left / Pan Right / Tilt Up / Tilt Down / Zoom In / Zoom Out / Tracking Shot]",
  "ambiance_audio": "[环境音效：仅描述 diegetic sound（画内音），不包含背景音乐]",
  "dialogue": [
    {"speaker": "[角色名]", "line": "[对话内容]"}
  ],
  "narration": "[可选：旁白或内心独白，仅剧集动画模式使用]"
}

Episodic animation mode example

Script scene data:

{
  "scene_id": "E1S01",
  "image_prompt": {
    "scene": "[场景环境描述文本]",
    "composition": {
      "shot_type": "[镜头类型]",
      "lighting": "[光线描述]",
      "ambiance": "[氛围描述]"
    }
  },
  "video_prompt": {
    "action": "[动作描述文本]",
    "camera_motion": "[摄像机运动类型]",
    "ambiance_audio": "[环境音效描述]",
    "dialogue": [
      {"speaker": "[角色名]", "line": "[对话内容]"}
    ],
    "narration": "[旁白文本，可选]"
  }
}

Storytelling mode example

{
  "segment_id": "E1S01",
  "novel_text": "[小说原文，用于后期人工配音]",
  "image_prompt": {
    "scene": "[场景环境描述文本]",
    "composition": {
      "shot_type": "[镜头类型]",
      "lighting": "[光线描述]",
      "ambiance": "[氛围描述]"
    }
  },
  "video_prompt": {
    "action": "[动作描述文本]",
    "camera_motion": "[摄像机运动类型]",
    "ambiance_audio": "[环境音效描述]",
    "dialogue": [
      {"speaker": "[角色名]", "line": "[仅当 novel_text 包含角色对话时填写]"}
    ]
  }
}

Key points of storytelling mode：novel_text Does not participate in video generation and is dubbed manually in post-production.dialogue Only includes character dialogue from the original text.

Dialogue Format

Dialogue format recommended by Veo 3:Speaker（manner）说道："text"

emotion Convert to manner
happy happily
sad sadly
angry angrily
surprised in surprise
scared in fear
neutral calmly
determined firmly
cold coldly
proud smugly
anxious anxiously

Sound Effect Description

Sound effects blend naturally into the scene, without using tags:

  • 1 sound effect:背景中传来{效果}。
  • 2 sound effects:可以听到{效果1}和{效果2}。
  • Multiple sound effects:环境音：{效果1}、{效果2}，以及{效果3}。

Camera Movement Description

Camera movement is converted into natural language:

  • dolly in → 镜头缓缓推进。
  • pan left → 镜头向左平移。
  • track → 镜头跟随移动。
  • handheld → 手持镜头轻微晃动。

Negative Prompt

Automatically exclude unwanted elements via API parameters:

background music, BGM, soundtrack, musical accompaniment

There is no need to declare "no background music" in the video prompt; this is handled by the negative_prompt parameter.

Audio Specifications

The video audio includes the following content:

  • Character dialogue: automatically generated via the dialogue field
  • Ambient sound: wind, rain, city noise, natural sounds, etc.
  • Scene sound effects: footsteps, door sounds, object collision sounds, etc.

BGM is automatically excluded via negative_prompt . To add background music, use /compose-video for post-processing.

API Usage

Video Generation

generate_video() The method supports the video generation feature:

from lib.gemini_client import GeminiClient

client = GeminiClient()

# 获取内容模式对应的画面比例
# 说书模式: 9:16, 剧集动画模式: 16:9
video_aspect_ratio = get_aspect_ratio(project_data, 'video')

# 获取默认时长
# 说书模式: 4s, 剧集动画模式: 8s
default_duration = 4 if content_mode == 'narration' else 8

# 生成视频（返回三元组）
path, video_ref, video_uri = client.generate_video(
    prompt=video_prompt,
    start_image="projects/{项目名}/storyboards/scene_E1S1.png",
    aspect_ratio=video_aspect_ratio,
    duration_seconds=str(default_duration),
    output_path="projects/{项目名}/videos/scene_E1S1.mp4"
)

Video extension (only when longer than 8 seconds is needed)

⚠️ Important: The extension feature only supports 720p resolution videos. To use the extension feature, the initial video must first be generated at 720p.

Use the generate_video() method to extend the video via the video parameter:

# 先生成初始视频（如需延长，必须使用 720p）
path, video_ref, video_uri = client.generate_video(
    prompt=video_prompt,
    start_image="projects/{项目名}/storyboards/scene_E1S1.png",
    aspect_ratio="16:9",
    duration_seconds="8",
    resolution="720p",  # ⚠️ 延长功能要求 720p
    output_path="projects/{项目名}/videos/scene_E1S1.mp4"
)

# 延长同一个场景（+7秒）
# ⚠️ 注意：仅 720p 视频可以延长，1080p 无法延长
path2, video_ref2, video_uri2 = client.generate_video(
    prompt="继续当前动作...",  # 延续同一场景的 prompt
    video=video_ref,  # 传入上一次返回的 video_ref
    output_path="projects/{项目名}/videos/scene_E1S1_extended.mp4"
)

# 也可以使用本地视频文件延长
path3, video_ref3, video_uri3 = client.generate_video(
    prompt="继续当前动作...",
    video="projects/{项目名}/videos/scene_E1S1.mp4",  # 本地文件路径
    output_path="projects/{项目名}/videos/scene_E1S1_extended2.mp4"
)

# 或使用保存的 URI 恢复并延长（跨会话）
path4, video_ref4, video_uri4 = client.generate_video(
    prompt="继续当前动作...",
    video=video_uri,  # 使用之前保存的 URI 字符串
    output_path="projects/{项目名}/videos/scene_E1S1_extended3.mp4"
)

Types supported by the video parameter

Type Example Description
Video object video_ref The return value from the previous call, used directly
GCS URI gs://bucket/video.mp4 Wrapped as a Video object
API URI https://... Wrapped as a Video object
Local file ./output.mp4 Read the file contents

Pre-generation checks

  • All scenes have approved storyboard images
  • Dialogue text length is appropriate
  • Action descriptions are clear and simple