video-generator

Video Generation

Workflow Overview

1. Phase 1: Initial → Gather requirements, STOP for user confirmation
2. Phase 2: Global Definitions → Define style, characters, voices, BGM (text only, no images)
3. Phase 3: Clip Planning → Segment into clips, plan each clip, determine reference image needs
4. Phase 4: Reference Images → Generate reference images (MANDATORY before Phase 5)
5. Phase 5: Execution → Generate keyframes, videos, audio

---

Critical Rules (MUST Follow)

Before starting, memorize these non-negotiable rules:

1. [PHASE 1 STOP] MUST ask questions to gather information. DO NOT assume or guess missing details—always ask the user. Never proceed without explicit user confirmation.

2. [DETAILED VIDEO PROMPT] Video prompts must include detailed transition_description (2-4 sentences). One-line prompts are insufficient.

1. [KEYFRAME DIFFERENCE] Last keyframe must show interpolatable change from first keyframe: subject position/pose, subject state (open/close, appear/disappear), or composition change. Subtle-only changes (lighting, background) while subject stays static cause unnatural video motion.

2. [PHASE 4 MANDATORY] MUST generate reference images before keyframes. Never skip Phase 4.

3. [ASPECT RATIO] ALL keyframes must use 16:9 or 9:16, and must be upright (not rotated). Never generate 1:1 or other ratios.

4. [NO TTS FOR ON-SCREEN] Never use TTS for on-screen dialogue or singing. Video model generates audio with lip sync.

5. [NARRATION CLIP BY CLIP] Generate off-screen narration separately for each clip, not all at once.

6. [AUDIO MIXING] When combining audio tracks (video audio, narration, BGM), preserve ALL tracks—overlay, never replace. Narration must be clearly audible and maintain consistent volume across all clips.

---

Image Generation Tools

Tool	Use When
generate_image	Create new images (with or without references)
generate_image_variation	Edit existing images

---

Phase 1: Initial

Gather Information

Field	Description
Purpose	Goal and target audience
Narrative arc	Story structure and key points
Duration	Total length in seconds
Aspect ratio	16:9 or 9:16 only
Visual style	Sub-genre aesthetic (e.g., "Makoto Shinkai anime", "Pixar 3D")
Reference materials	Reference videos, images, brand guidelines
Language	For dialogue and narration
Recurring elements	Characters/objects with appearance descriptions
Dialogue/singing needs	On-screen character audio
Narration needs	Off-screen narrator (gender, tone, pace)

Five-Dimension Expert Framework

Use these perspectives to guide your questions:

Dimension	Expert Role	Key Questions
Strategy & Audience	Creative Director	Who is this for? What's the goal? What action should viewers take?
Narrative & Structure	Screenwriter	What's the story? Key moments? Emotional arc?
Visual Style	Director + Art Director	What look and feel? Reference videos/images? Color mood?
Shot Execution	Cinematographer	Any specific shots in mind? Product hero shots needed?
Sound Design	Sound Designer	Voiceover? Music mood? Dialogue? Sound effects?

Ask questions across all dimensions. Prioritize based on user's initial description.

[MANDATORY STOP - DO NOT PROCEED WITHOUT USER CONFIRMATION] Summarize gathered information and wait for user confirmation before Phase 2.

---

Phase 2: Global Definitions (Text Only)

Visual Style Specification

Define these 4 dimensions (applied to primary reference images in Phase 4):

Dimension	Example Values
Sub-genre	Makoto Shinkai anime, Pixar 3D, cyberpunk noir
Rendering + Line	2D hand-drawn with thick outlines, 3D cel-shading
Color + Lighting	High saturation neon, soft diffused natural light
Detail density	Minimalist, highly detailed backgrounds

Example specification:

Sub-genre: Cyberpunk anime
Rendering + Line: 2D digital painting, thin glowing outlines
Color + Lighting: High saturation neon (pink, cyan, purple), dark backgrounds, rim lighting
Detail density: Highly detailed backgrounds, moderate character detail

Recurring Elements

For each character/object:

Field	Description
unique_identifier	Name for reference
appearance	Text description for prompts
outfit_description	Clothing/accessories (characters)
language	Spoken/sung language (if applicable)
mechanical_properties	Physical behavior (if applicable)

Voice Profiles

• On-screen: From character definitions (dialogue/singing)
• Off-screen narrator: name, gender, tone, pace, language

BGM Source Decision

Scenario	BGM Source
Music video / diegetic music (visible source)	Embedded (in video prompt)
Background mood music	Separate (Phase 5 BGM Preparation)
No music	None

If Separate, define: genre, instruments, tempo

---

Phase 3: Clip Planning

Segmentation Rules

• Clips: 4, 6, or 8 seconds only
• Each clip: one action, one scene

Per-Clip Specification

Field	Values
narrative_purpose	establish / develop / climax / resolve / transition / supplementary (product shot, detail, reaction, insert, B-roll, POV)
pacing	slow / moderate / fast
scene	Environment description
content_action	Subject + action + trajectory
transition_description	[REQUIRED] Detailed transition process. Must include: subject appearance, movement trajectory, state changes, existence statements. 2-4 sentences minimum.
duration	4 / 6 / 8
camera_movement	static / pan / tilt / dolly / zoom / crane / arc / handheld
first_keyframe_framing	Shot size + angle + composition
first_keyframe_visible_content	What's visible
last_keyframe_framing	Shot size + angle + composition
last_keyframe_visible_content	What's visible
last_keyframe_edit_from_first	yes / no (see decision table below)
inter_clip_boundary	continuous / scene_cut
first_keyframe_reuse	yes / no
last_keyframe_required	yes / no
on_screen_dialogue	"Name: text" or "Name: [lyrics] (style)" or None
sound_effects	Sources or None
bgm_source	embedded / separate / none
bgm_cue	If embedded: style, BPM, instruments. If separate: emotion, intensity
narration_cue	Narrator text or None

Field Dependencies

• inter_clip_boundary = continuous → next clip's first_keyframe_reuse = yes
• first_keyframe_reuse = yes → previous clip must have last_keyframe_required = yes

Keyframe Difference Requirement

When planning last_keyframe_visible_content, ensure interpolatable change from first_keyframe_visible_content:

• Subject position/pose change (movement, rotation, action)
• Subject state change (open/close, appear/disappear, expression)
• Composition change from camera movement (zoom, pan result)

[WARNING] Avoid last keyframes with only lighting or background changes while subject remains static—this causes unnatural video motion.

Decision: last_keyframe_edit_from_first

Camera Movement	First & Last Keyframe Overlap?	Set to
static, small pan/tilt, zoom	Yes (same scene area)	yes
large pan, dolly, tracking, crane, arc	No (different area)	no

transition_description Requirements

This field directly becomes part of the video prompt. The more detailed, the better.

Must include:

1. Subject appearance: Key visual features that must remain consistent throughout
2. Movement trajectory: How subject/camera moves through space and time
3. State changes: How objects/environment change over the duration
4. Existence statements: What is present throughout (prevents pop-in/pop-out)

Length guideline: 2-4 sentences minimum. One-line descriptions are insufficient.

transition_description Examples

Insufficient	Sufficient
"Open box revealing jar"	"The frosted glass jar with gold lid is inside the box from the start, hidden by the closed cream-colored lid. Elegant hands with manicured nails lift the lid upward smoothly. As the lid rises, the jar gradually comes into view - first the gold cap edge, then the full jar nestled in champagne velvet."
"Person walks left to right"	"Woman in white dress with brown hair starts at left edge of frame, walks steadily rightward at moderate pace, maintaining upright posture, reaches right edge by end of clip."
"Light turns on"	"Room starts in complete darkness. Light gradually increases from the ceiling fixture at center, warm yellow glow spreading outward across the wooden furniture until fully illuminated."

Physical Consistency Check

Movement	Constraint
Pan/Tilt/Zoom	Camera fixed, content within rotational/zoom range
Dolly/Tracking/Crane	Content physically traversable within duration
Arc	Subject centered in both keyframes, environment allows orbit
Handheld	Similar to Dolly but allows irregularity
Combined	Must satisfy ALL involved movement constraints

Common Mistakes:

Mistake	Correction
"Pan from corridor entrance to middle"	Use "dolly forward"
First: room A, Last: room B	Split into two clips
6-second clip covering 100 meters	Extend duration or reduce distance

[MANDATORY] Reference Image Requirements

After all clips planned, list required reference images:

Element	Clips Using It	Required Images
(name)	Clip X (MS), Clip Y (CU)	Full body, Face close-up

[WARNING] Only generate what clips actually need. Do NOT generate all angles by default.

---

[MANDATORY] Phase 4: Reference Image Generation

MANDATORY. Do not skip to Phase 5.

Generation Order

Step 1: Primary reference (visual anchor)

• Tool: generate_image (no references)
• Prompt MUST include: Full Visual Style Specification from Phase 2 + element description
• White background
• Ends with "no text, no watermarks, no logos, no labels, no annotations"

Step 2: Additional angles/shots

• Tool: generate_image with primary reference as reference
• Prompt: New angle/shot only (style inherited from reference)
• White background
• Ends with "no text, no watermarks, no logos, no labels, no annotations"

[WARNING] Never generate additional refs without using primary ref as reference.

---

Phase 5: Execution

Global Rules

[CRITICAL] ALL keyframes: aspect ratio from Phase 1 (16:9 or 9:16). Never 1:1.

First Keyframe

first_keyframe_reuse = yes → Use previous clip's last keyframe (no generation)
first_keyframe_reuse = no  → Generate new keyframe

If generating first keyframe:

• Tool: generate_image
• References: Appropriate Phase 4 images
• Aspect ratio: 16:9 or 9:16
• Prompt includes:
  • Visual style (sub-genre + key characteristics, brief)
  • Scene environment
  • Framing (shot size + angle + lens)
  • Visible content
  • Subject appearance + outfit
• Prompt ends with: "no text, no watermarks, no logos, no annotations"

Last Keyframe

last_keyframe_required = no  → Skip
last_keyframe_required = yes:
  last_keyframe_edit_from_first = yes → Edit mode
  last_keyframe_edit_from_first = no  → Generate mode

If EDIT mode:

• Tool: generate_image_variation
• References: [first_keyframe, Phase 4 refs...]
• Prompt: "Edit this image: [changes only]"
• Do NOT repeat unchanged elements

If GENERATE mode:

• Tool: generate_image
• References: [first_keyframe (scene ref), Phase 4 refs...]
• Aspect ratio: 16:9 or 9:16
• Prompt includes:
  • Visual style (brief)
  • Last keyframe framing + visible content
  • Subject appearance and end state
  • "Same location/environment as reference"
• Prompt ends with: "no text, no watermarks, no logos, no annotations"

Consistency Checklist (Easily Overlooked)

When generating last keyframe, verify:

• Interpolatable change: Clear difference in subject position/pose, state, or composition (not just lighting/background)
• Same lighting direction and shadows as first keyframe
• Same color temperature (warm/cool)
• Same depth of field
• Same outfit, facial features, body proportions
• Environment details consistent

Video Generation

Video prompt should be detailed. Even with keyframes, video models may drift during generation.

Prompt includes:

• Visual style (brief)
• Pacing (slow / moderate / fast)
• transition_description from Phase 3 (detailed, 2-4 sentences)
• Subject appearance (key features for consistency)
• Scene environment (brief)
• Audio (see below)

Audio in prompt:

Type	Include
On-screen dialogue	"Name says: text" with tone, language
On-screen singing	"Name sings: [lyrics]" with style, language
Sound effects	Source + quality
Embedded BGM	Style, BPM, instruments, mood

Prompt ending by bgm_source:

• embedded → (no ending, music described in prompt body)
• separate/none → End with "No background music."

Example (music video with embedded BGM):

Hatsune Miku center stage, singing in Japanese with sweet electronic voice: 
"ラララ、光の中で踊り出す", energetic J-pop at 140 BPM with synthesizer, 
crowd cheering, concert atmosphere

[CRITICAL] Never use TTS for on-screen dialogue/singing. Video model generates audio with lip sync.

BGM Sourcing (if bgm_source = separate)

Method: Search and download from royalty-free music libraries (e.g., Pixabay, YouTube Audio Library).

[CRITICAL] Generating music with Python or any other tools is strictly prohibited. You must only use pre-existing, royalty-free tracks.

Match the downloaded music to the style defined in Phase 2.

Narration Generation (if narration exists)

[WARNING] Generate clip by clip, not all at once.

• TTS for off-screen narrator only
• Same voice profile across all clips
• Verify audio duration fits clip duration

Audio Summary

Type	Method	Output
On-screen dialogue/singing	Video model	Embedded
Sound effects	Video model	Embedded
Embedded BGM	Video model	Embedded
Separate BGM	Search only	Separate track
Narration	TTS (clip by clip)	Separate track

Audio Mixing (Final Assembly)

When combining multiple audio sources:

Track	Source
Video audio	Embedded in video clips (dialogue, sound effects, embedded BGM)
Narration	TTS generated (off-screen narrator)
Separate BGM	Searched from royalty-free source

[CRITICAL] Mixing rules:

• Preserve ALL audio tracks—overlay, never replace one with another
• Narration must be clearly audible—not drowned out by other tracks
• Narration volume must be consistent across all clips