video-style-analysis

Perfect video reproduction (agent analysis + TTS dubbing + audio-video synthesis)

Task Objective

This Skill is used to: perfectly reproduce the complete content of the original video, including regenerating the visuals (maintaining high similarity) and cloning the original video's dubbing style (timbre, intonation, rhythm), and synthesizing the final video.
Capabilities include: agent-based video analysis (visual + audio), style prompt generation, TTS dubbing generation, audio-video synthesis.
Trigger conditions: use when the user needs to perfectly reproduce video content, clone the original video's dubbing style, and maintain the original storyline and visual style.

Prerequisites

Dependency note: ensure the following Python packages are installed
```
opencv-python
numpy
moviepy
Pillow
openai-whisper
```

Core Concept

⭐ Multi-layer video semantic analysis: first make a rough judgment, then unravel step by step

This Skill should not compress the video into a single prompt right from the start.

Regardless of whether ASR is available, alayer-by-layer convergenceanalysis process must first be completed: first determine what content mechanism the video belongs to, then determine what is happening in the frame, then extract evidence that can help with further determination, and only finally generate reproduction constraints.

Mandatory order:

Media baseline: resolution, landscape/portrait orientation, duration, frame rate, whether there is an audio track, whether there is OCR/subtitles
Rough content classification: narrative / talking-head / documentary / tutorial / performance / training / product showcase / ambient film
Scene semantic skeleton: location type, spatial structure, number of people, subject organization, lighting and environment
Subject and behavior analysis: who is in the frame, what they are doing, whether the action is synchronized or adversarial, whether it is training or performance
Discriminative evidence extraction: clothing, shoes, props, equipment, text, logos, screen content
Subcategory convergence and reproduction constraints: continue converging "looks like sports" into "training / performance / competition / rehearsal", etc., then write the prompt and negative constraints

Principles：

It is not allowed to jump directly from "multiple people + indoor" to "multi-person event scene"
It is not allowed to stop directly at "looks like dancing"; you must continue checking movements, formation, clothing, shoes, text, and props
First write thefact layer, then write theinterpretation layer, and only finally write thegeneration layer
Each layer must answer:我看到了什么、这说明什么、还缺什么证据

6-layer, 16-dimensional semantic analysis (general)

Layer 0: Media baseline

媒体规格: resolution, aspect ratio, landscape/portrait orientation, duration, frame rate, clarity
音频角色: no human voice / narration / dialogue / ambient sound / music-dominant / mixed

Layer 1: Content mechanism judgment

内容类型: narrative, documentary, tutorial, stage, training, advertisement, travel, product, pure ambient
叙事驱动力: ASR-driven, visually driven, subtitle-driven, music-driven, or mixed-driven

Layer 2: Scene semantic skeleton

地点与空间: indoor/outdoor, building type, space use, depth of field and positioning relationships
时间与光线: Day/night, natural light/artificial light, warm/cool, hard light/diffused
主体规模: single person, two people, small group, large group portrait
主体组织关系: dialogue, collaboration, confrontation, formation, spectating, performance, teaching

Layer 3: Subject and Behavior

核心行为: walk, run, jump, practice, perform, speak, do, display, interact, operate
动作结构: static, repetitive action, continuous action chain, explosive action, synchronized action
外观线索: age group, gender tendency, clothing category, degree of uniformity, hairstyle, makeup and styling

Layer 4: Discriminative Evidence

道具/器材/鞋服细节: the specific objects or clothing details that best help subclass determination
文字与符号线索: subtitles, signage, screen UI, wall text, printed text on clothing, logos

Layer 5: Generation Constraints

构图与摄影: shot size, camera position, composition, camera movement, rhythm
风格与氛围: color tone, texture, realism, period feel, cultural style, emotional climate
复刻硬约束: what must be preserved, what must not be wrong, what may be approximate

Example (abstract method, not a hardcoded dance template)：

Coarse classification: indoor multi-person synchronized activity
Behavior judgment: repeated foot lifts, formation training, consistent rhythm
Evidence extraction: uniform clothing, split-toe dance shoes, wooden floor, wall text
Subclass convergence: group basic dance step training / folk dance rehearsal, rather than generalizing into "multi-person exercise"

⭐ ASR priority + tiered fault tolerance mechanism

Key insight: ASR speech recognition is not just "speech-to-text", but also akey step in understanding video narrative。

Comparison：

Method	Image only	Image + ASR
Number of characters	May misjudge (e.g., count 7)	Accurate (e.g., actually 2)
Protagonist identification	Uncertain	Known immediately
Narrative line	Needs guessing	Directly stated in text
Character relationships	Unclear	Clear

Therefore: When ASR is effective, ASR resultsguidecharacter identification and narrative understanding; but it does not replace visual semantic analysis, and even less can it skip "coarse classification → evidence extraction → subclass convergence".

Execution method: For the ASR step, if the main session environment cannot install Whisper, a sub-agent session will automatically be started to execute it.

⚠️ Tiered fault tolerance mechanism

Problem: Not all videos have audio, and not all audio can yield an effective narrative line.

Solution: A three-level fault tolerance mechanism, with automatic degradation handling.

Level 1: ASR success + clear narrative line (best) ✅

Trigger conditions：

The video has an audio track
ASR successfully extracts text
The text contains clear character information and narrative line

Handling method：

Follow the standard process: ASR → narrative line analysis → character identification → visual verification
Output high-quality narrative analysis

Example: Complete plot-type short video (containing clear character relationships and narrative line)

Level 2: ASR success but narrative line unclear (second best) ⚠️

Trigger Conditions：

The video has an audio track
ASR successfully extracts text
The text lacks clear character information or narrative lines (e.g., ambient sound, music, fragmented dialogue)

Processing Method：

ASR-assisted mode: Use ASR timestamps as scene boundaries
Text assistance: Extract keywords as style hints
Fall back to visual analysis: Mainly rely on keyframe visual analysis
Character recognition: Pure visual inference + ASR keyword assistance

Example Scenarios：

Travel documentary: ASR may only contain ambient sound, but timestamps can be used for scene segmentation
Food video: ASR may only contain cooking sounds, but the scene rhythm can be extracted

Level 3: No ASR or ASR failure (fallback) 🛡️

Trigger Conditions：

The video has no audio track
ASR fails completely (audio corruption, unsupported format, etc.)
Visual-only video

Processing Method：

Pure visual analysis mode
Scene segmentation: Completely relies on keyframe visual differences
Character recognition: Use character_detection.jsonfrom Step 1.5 (InsightFace face embedding clustering; even with outfit changes, the same character can be matched)
Layer-by-layer visual semantic analysis: Execute in the order of "coarse content classification → subject behavior → evidence extraction → subclass determination → constraint generation"
Narrative line inference: Infer based on scene changes and action sequences, rather than directly guessing the theme

Risk Warning：

The number of characters may be misjudged (e.g., the previous 7 vs. the actual 2)
The narrative line may be inaccurate
The user needs to confirm the character recognition results

Example Scenarios：

Silent videos, animated videos, pure visual display videos

Automatic Fallback Process

def analyze_with_fallback(video_path):
    """
    带容错的视频分析流程
    """
    # 尝试提取音频
    audio = extract_audio(video_path)
    
    if audio is None:
        # Level 3: 无音频
        return pure_visual_analysis(video_path)
    
    # 尝试ASR
    asr_result = perform_asr(audio)
    
    if asr_result is None or not asr_result["full_text"]:
        # Level 3: ASR失败
        return pure_visual_analysis(video_path)
    
    # 尝试叙事线分析
    narrative = analyze_narrative(asr_result["full_text"])
    
    if narrative["clarity_score"] < 0.5:
        # Level 2: 叙事线不清晰
        return hybrid_analysis(video_path, asr_result, narrative)
    
    # Level 1: 完整流程
    return full_analysis_with_narrative(video_path, asr_result, narrative)

Output Identifier：

Each analysis result must include the analysis_level field ("level_1" / "level_2" / "level_3")
Level 2/3 need to add the confidence_warning field to prompt the user for verification

Operation Steps

Phase 1: Narrative Understanding (ASR First)

Step 1: Intelligent Keyframe Extraction

Execution Method: Call Python script

python scripts/smart_keyframe_extractor.py \
  --video_path <原视频路径> \
  --output_dir output/keyframes/ \
  --min_fps 2.0

Core Functions：

Scene change detection: Identify scene boundaries (inter-frame pixel difference threshold 15%)
Action density analysis: Extract more from action-dense areas and moderately from static areas
Uniform extraction fallback: At least 2 frames per second to ensure important shots are not missed

Output：

output/keyframes/*.jpg - Keyframe images
output/keyframes/extraction_result.json - Extraction results (scene list, keyframe list, video information)

Step 1.5: Character Detection and Cross-Frame Clustering

Execution Method: Call Python script (InsightFace)

python scripts/character_detector.py \
  --keyframes_dir output/keyframes/ \
  --output_path output/analysis/character_detection.json \
  --extraction_result output/keyframes/extraction_result.json \
  --similarity_threshold 0.4

Core Functions：

Face detection: Use InsightFace (buffalo_l) to detect faces and their positions in each frame
Feature extraction: Extract 512-dimensional face embedding vectors (ArcFace)
Cross-frame clustering: Cosine similarity greedy clustering (default threshold 0.4), even with outfit changes, the same character can be matched by facial features
Character profiles: Generate the list of appearance frames for each character, face/body region bboxes, and representative face crops
ASR integration(optional): After step 3.5 is complete, you can rerun with --narrative_path to associate visual characters with text character names

Why this step is needed：

In pure image analysis, the same character after an outfit change will be misjudged as a different person
When ASR fails, without character detection you can only rely on Claude looking at the images to guess
Face embeddings remain stable across outfit changes and scene changes, making them the most reliable way to match characters across frames

Output：

output/analysis/character_detection.json - Character detection and clustering results
output/analysis/characters/char_N_face.jpg - Representative face crops for each character

Correct understanding of the person count field：

unique_characters = Cross-frame face cluster count, which indicates how many faces that "may be different people" were clustered at the underlying detection level,and cannot be directly taken as the number of people on site
frame_visible_people = once for each keyframe extracted in step 1 单帧可见人数估计
visible_people_stats.stable_visible_people_estimate = the default used by the replication pipeline stable visible person count estimate
scene_visible_people_stats = the person count statistics for each scene, which should be used first when subsequently judging subject scale, group portrait strategy, and prompt person-count relationships

Output example：

{
  "total_keyframes_analyzed": 10,
  "total_faces_detected": 15,
  "unique_characters": 2,
  "visible_people_stats": {
    "stable_visible_people_estimate": 2,
    "min_visible_people": 1,
    "max_visible_people": 2
  },
  "scene_visible_people_stats": [
    {
      "scene_id": 0,
      "time_range": "0.0-5.0s",
      "stable_visible_people_estimate": 2
    }
  ],
  "characters": [
    {
      "character_id": 0,
      "label": "char_0",
      "matched_name": null,
      "appearance_count": 7,
      "keyframe_indices": [0, 2, 3, 5, 6, 8, 9],
      "scene_ids": [0, 1, 2, 3],
      "face_bboxes": {"frame_0000.jpg": [120, 50, 220, 180]},
      "body_bboxes": {"frame_0000.jpg": [80, 50, 260, 440]},
      "representative_face_path": "output/analysis/characters/char_0_face.jpg",
      "avg_det_score": 0.92
    }
  ]
}

Step 2: Color and motion analysis

Execution method: call a Python script

python scripts/video_analyzer.py \
  --video_path <原视频路径> \
  --output_path output/analysis/color_analysis.json

Analysis content：

Technical specifications: resolution, frame rate, duration, encoding format
Color distribution: RGB mean, HSV features, brightness, contrast
Hue identification: warm_red_orange, warm_yellow, green, cyan, blue, purple, etc.
Motion features: static, slow_movement, normal_movement, dynamic_fast, etc.

Output：

output/analysis/color_analysis.json - Color analysis results

Step 3: ASR speech recognition ⭐ Core step

Execution method: call a Python script (automatic audio extraction + speech recognition)

python scripts/asr_transcriber.py \
  --video_path <原视频路径> \
  --output_path output/analysis/asr_result.json \
  --model_size base \
  --language zh \
  --word_level

Automatic execution flow：

Audio extraction: use ffmpeg to extract the audio track from the video (16kHz, mono, WAV format)
Speech recognition: use the Whisper model to convert audio to text
Timestamp extraction: generate paragraph-level and word-level timestamps

⚠️ Important note：

If the main session environment cannot install the Whisper library, it willautomatically start a sub-agent session for execution
The sub-agent session has an independent environment, can install dependencies, and execute the complete flow
After the ASR results are returned, the main session continues with the subsequent steps

Analysis content：

Speech-to-text: extract the complete narration text
Timestamp extraction: paragraph-level and word-level timestamps (for audio-visual alignment)
Narrative content: characters, identities, relationships, and narrative themes contained in the text

Output：

output/audio/extracted_audio.wav - Extracted audio file
output/analysis/asr_result.json - ASR recognition results

Output example：

{
  "video_path": "原视频.mp4",
  "language": "zh",
  "full_text": "大家好，今天要分享的是...",
  "duration": 45.2,
  "segments": [
    {
      "id": 0,
      "start": 0.0,
      "end": 3.5,
      "text": "大家好，今天要分享的是",
      "words": [
        {"word": "大家好", "start": 0.0, "end": 0.8, "confidence": 0.95}
      ]
    }
  ],
  "voice_style": {
    "speed": {"chars_per_second": 3.2, "level": "中等"},
    "pause": {"count": 5, "frequency": 1.1},
    "estimated_tone": "温柔",
    "estimated_emotion": "温暖"
  }
}

Step 3.5: Narrative Line Analysis + Rough Content Classification ⭐ Key Step

Execution method: Review the draft + Agent supplement

Draft has been automatically generated：output/analysis/narrative_analysis.json by draft_generator.py automatically generated at the end of Phase 1, containing the person names extracted from the ASR text and the narrative arc skeleton divided into three equal time segments. Claude only needs to read the ASR text and then fill in all [TODO] placeholders.

Core Concept: First determine "what type of content" this video is, then use the ASR text to identify characters and narrative lines to guide subsequent image verification.

Analyze Content：

3.5.1 Rough Content Classification (Must Be Done First)

Determine whether the video belongs to: Drama / Talking-head / Documentary / Tutorial / Performance / Training / Product Display / Pure Atmosphere
Determine the dominant mode of understanding for the current video: ASR-driven / Visual-driven / Subtitle-driven / Music-driven
If the ASR text is very weak, treat it only as a supplementary signal; do not forcibly fabricate a complete plot

3.5.2 Narrative Theme Identification

Identify the story type from the text (romance/inspirational/documentary/advertisement, etc.)
Identify the narrative perspective (first-person "I"/third-person)
Identify the emotional tone (warm/tense/humorous/healing, etc.)

3.5.3 Character Identification (from Text)

Protagonist Identification: Who is the protagonist explicitly mentioned in the text?
Character Relationships: What is the relationship between the characters? (couple/friends/family, etc.)
Character Identity: What is the character's true identity? (wealthy person/ordinary person/entrepreneur, etc.)

3.5.4 Narrative Arc

Introduction: The beginning of the story (0-? seconds)
Development: The development of the story (?-? seconds)
Turn: The turning point of the story (?-? seconds)
Conclusion: The ending of the story (?-? seconds)

Output：

output/analysis/narrative_analysis.json - Narrative line analysis result

Example Output(Neutral placeholder example, only demonstrating the structure):

{
  "narrative_theme": "都市人物关系变化故事",
  "narrator": "第一人称主角（女性）",
  "characters_from_text": [
    {
      "name": "角色甲",
      "role": "主要对手戏角色，表面身份普通",
      "identity": "隐藏真实背景"
    },
    {
      "name": "叙述者（我）",
      "role": "第一人称主角",
      "identity": "故事推动者"
    }
  ],
  "narrative_arc": {
    "act_1": {"name": "建立关系", "time_range": "0-15s"},
    "act_2": {"name": "冲突发展", "time_range": "15-35s"},
    "act_3": {"name": "结果揭示", "time_range": "35-49.5s"}
  }
}

Important: After this step is completed, you should first refine the semantic_analysis.jsonin Step 3.6, then proceed to the character visual verification in Step 4.

Step 3.6: 6-Prism Semantic Analysis Skeleton (Layer-by-Layer Convergence)

Execution method: Review the draft + Agent completes behavior, evidence, and subclass judgments

Draft has been automatically generated：output/analysis/semantic_analysis.json has automatically written the media baseline and generated a 6-prism skeleton for each scene. Claude should prioritize refining this file before proceeding to Steps 4-6.

Core Concept: This step is not simply about writing an analysis report, but about producing areplication master brain. First use a fixed questioning framework to see the video clearly, then write the analysis conclusions directly as unified generation constraints that subsequent steps must inherit.

6-Prism Fixed Questioning Framework：

narrative_prism: What is this segment talking about? What does this shot accomplish only? What content type does it currently belong to? Allowed unknown
subject_prism: Who is the subject? What is the group size? What is the relationship between the subjects? Who must appear?
action_prism: What are the subjects doing? Is the action static, repetitive, synchronized, or a continuous action chain?
scene_prism: What is the space? What is the lighting like? What clothing, shoes, props, text, and environmental evidence are there?
camera_prism: How should this shot be filmed? What are the shot size, angle, camera movement, and the shot's focal point?
constraint_prism: Which facts must be preserved? Which can be approximated? Which must never be generated incorrectly?

This step must also separately structure 4 categories of high-value judgments：

Ontology / subclass judgment: You cannot just write broad categories like "training / performance / event"; you must continue converging to a finer level; write into narrative_prism.subtype_judgment
Behavior judgment: You cannot just write "is moving / is dancing"; you must state what the subject is actually doing and on what basis this is judged; write into action_prism.behavior_judgment
Evidence chain: You cannot just jot down shoes, text, and props in a scattered way; you must organize them into "direct evidence / supporting evidence / counterexample hypotheses / unresolved points"; write into scene_prism.evidence_chain
Negative constraints: They cannot just be scattered in the description; the highest-risk facts most likely to be generated incorrectly must be listed separately; write into constraint_prism.negative_constraints

Content that must be completed：

Media baseline: Confirm resolution, aspect ratio, duration, audio mode, and dominant mode of understanding
6-prism completion: Every scene must complete narrative_prism / subject_prism / action_prism / scene_prism / camera_prism / constraint_prism
Unified generation contract / generation_contract: At step 3.6, lock the final video format and reference-image strategy for the entire project first. Later steps 8 and 9 can only inherit; they cannot each decide on their own
Unified style template / style_consistency: It is not an independent brain, but an expanded description of generation_contract . Later steps 8 and 9 must directly inherit it, and are not allowed to improvise separately
Open-world judgment: If the specific category is uncertain, label_status = unknownis allowed, but skipping behavior, evidence, and constraints is not allowed
Evidence-driven replication: Even if it cannot be named precisely, clothing, shoes, props, text, environment, and action organization must be written clearly
Negative constraint organization: Write the points of "must not be generated incorrectly" into constraint_prism
Do not only see without penetrating: Seeing shoes, text, formations, and actions is not enough; they must be turned into structured fields that subsequent prompts will consume

generation_contract + style_consistency at least must answer these points：

What form is the original video itself:live_action / stylized_live_action / anime
What form will this project ultimately be uniformly generated into: only one may be chosen
If model capability is insufficient, is whole-project uniform downgrading allowed; if allowed, it can only be downgraded for the whole project together, not locally in a chaotic way
What form must the character reference image be: directly the same form, or, in source_visual_mode when explicitly live-action, uniformly use pseudo_realistic_human_illustration this kind of realistic character illustration bridging solution
What form must the scene video be: it must be consistent with target_visual_mode and cannot be led astray by the reference-image medium
What are the background rules and composition rules for character reference images
For multi-person scenes, which visible_characters must be given identity anchors, and which core characters also need full_body_outfit
How should clothing material and color language be unified
Which consistencies must be maintained, and which deviations are explicitly forbidden
Which keywords must appear, and which keywords absolutely must not be mixed up

Principles：

The generation form is not decided temporarily at step 8 or step 9, but is a project-level conclusion locked in first at step 3.6
generation_contract and style_consistency together form the main replication brain
Every subsequent character image generation and every scene image generation must carry the same generation_contract + style_consistency
Between characters, only differences such as "hairstyle / face shape / gray pants vs black pants / standing position temperament" may be changed; the overall form and the project's unified drawing style must not be changed
For multi-person lineups, if visible_characters and full-body clothing anchors are not supplemented, step 9 is very likely to produce the entire lineup with the same face and the protagonist changing clothes; this must be avoided in step 6 first

Output：

output/analysis/semantic_analysis.json - 6-prism semantic analysis skeleton

Important：

If semantic_analysis.json has not yet been fully supplemented, do not rush to write scene_prompts.json
What is fixed is the problem structure, not the classification answer
final_label can be unknown
crowd mode only affects the reference image strategy; it does not equal the semantic conclusion of "on-site event"

Step 4: Deep visual analysis (subject behavior + evidence extraction)

Execution method: review the first draft + the agent verifies with read_image

The first draft has been automatically generated：output/analysis/coherence_analysis.json Contains the character skeleton mapped from character_detection.json. Claude needs to use read_image to view keyframes and representative character faces, fill in appearance descriptions, and supplement "what the subject is doing, why it is judged this way, and what other evidence supports it".

⚠️ Core requirements：

Step 3.5 narrative line analysis must be completed first
Step 3.6 semantic skeleton must be completed first
The read_image tool must be used to check keyframe content
Character identification must be based on the narrative line, rather than guessing from scratch
Prioritize using the character_detection.jsonfrom step 1.5: this file contains cross-frame character clustering, the number of people visible in a single frame, and scene population statistics.For population judgment, first look at visible_people_stats / scene_visible_people_stats, do not directly take unique_characters as the on-site population

Step 4.1: Assign keyframes based on the narrative line and scene

# 读取叙事线分析
with open("output/analysis/narrative_analysis.json") as f:
    narrative = json.load(f)

# 读取ASR时间戳
with open("output/analysis/asr_result.json") as f:
    asr_data = json.load(f)
    segments = asr_data["segments"]

# 根据ASR时间戳和叙事弧线，分配关键帧到对应角色
for segment in segments:
    start_time = segment["start"]
    end_time = segment["end"]
    text = segment["text"]
    
    # 根据文本内容判断这段属于哪个角色
    # 例如："我第一次见到角色甲" → 角色甲
    #       "那时我才明白真相" → 叙述者（我）
    
    # 找到对应时间的关键帧
    keyframes_in_range = [kf for kf in keyframes 
                          if start_time <= kf["time"] <= end_time]

Step 4.2: Use read_image to judge subject behavior

Besides seeing "who it is", you must also answer:

Is the main subject a single person, two people, a small group, or a large group portrait
What is the subject organization mode: dialogue / watching / lining up / collaboration / confrontation / performance / teaching
What is the current main behavior: walking, speaking, practicing, performing, displaying, operating, interacting
What is the action structure: static, repetitive action, synchronized action, continuous action chain, explosive action

Step 4.3: Use read_image to extract discriminative evidence

The following evidence must be actively checked, rather than only writing "looks like":

Clothing: whether it is uniform, whether it has uniform characteristics, color layers, functionality
Shoes: sneakers, leather shoes, dance shoes, work shoes, bare feet, etc.
Props/equipment: rackets, musical instruments, kitchenware, tools, training equipment, vehicles, etc.
Text and signs: wall text, screen UI, signage, clothing prints, subtitles
Environmental details: floor material, spatial characteristics such as stage/classroom/workshop/court/kitchen

Step 4.4: Use read_image to verify character appearance

# 选择要检查的帧（每个角色的代表性帧）
frames_to_check = []

# 对于每个角色，选择其出现的关键帧
for char in narrative["characters_from_text"]:
    char_frames = get_keyframes_for_character(char, segments, keyframes)
    frames_to_check.extend(char_frames[:5])  # 每个角色最多5帧

# 用read_image检查
result = read_image(frames_to_check)

# 从read_image结果提取角色外貌特征
# 注意：要和叙事线中的角色对应

Step 4.5: Character definition table (based on narrative line + visual verification)

Must be based on the narrative line + read_image results

Example character definition table (neutral placeholder example)：

Character	Text recognition (narrative line)	Keyframe verification	Age	Appearance	Clothing	Appearance Scene
Character A	"The first time I met Character A"	frame_005,010,015	25-35 years old	East Asian male, short black hair, well-defined facial features	Light-colored shirt → dark-colored jacket	1-5
Narrator	"Only then did I understand the truth"	frame_050,070,080	25-30 years old	East Asian female, short black hair	White top → formal dress	5-7

Comparison: Wrong Approach vs Correct Approach

Wrong Approach	Correct Approach
Recognize 7 characters from the image	Recognize 2 protagonists from the ASR text
Analyze each scene independently	Use ASR timestamps to assign scenes to characters
Guess the narrative line	The text directly states the narrative line
Do not know who the protagonist is	The text clearly states the protagonist

Output：

output/analysis/coherence_analysis.json - Scene analysis + character definition

Step 5: Audio-visual correlation analysis + subclass determination (agent)

Execution method: Review the first draft + agent supplements semantic scoring

The first draft has been automatically generated：output/analysis/audio_visual_correlation.json Includes the automatic mapping of ASR text and scene times. Claude needs to supplement semantic matching, emotional consistency scoring, and complete the final convergence judgment "from coarse classification to subclass".

Analysis Content：

Timeline Mapping: Establish the temporal correspondence between audio text and video shots
Semantic Matching Degree: Analyze the relevance between the narration text and the visual content
Emotional Consistency: Evaluate the consistency between speech emotion and visual atmosphere
Subclass Determination: For example, further converge "sports" into "training / competition / performance", and further converge "multi-person activity" into "formation training / classroom practice / stage performance / on-site activity"
Negative Constraint Organization: Clarify which content must not be generated incorrectly, such as the number of people relationship, formation, clothing type, key props, shoe type, text information

Output：

output/analysis/audio_visual_correlation.json - Audio-visual correlation analysis results

Stage 2: Intelligent Multi-dimensional Prompt Generation

Step 6: Generate visual style prompts

Execution method: Review the first draft + agent supplements scene descriptions and character appearances

The first draft has been automatically generated：output/prompts/scene_prompts.json Includes the character list mapped from character_detection, the scene skeleton derived from scene data (scene_id/duration/main_character already filled in), the visual_style_prompt derived from color analysis, and realistic photo style prompt prefix. Claude needs to truly implement the "content type, behavior, evidence, subclass determination, negative constraints" obtained in steps 3.5-5 into the prompt, rather than just adding a few adjectives.

Data Sources：

Color data:output/analysis/color_analysis.json
Character definition:output/analysis/coherence_analysis.json
Content coarse classification / subclass determination: conclusions from Step 3.5, Step 3.6, and Step 5

Before writing the prompt, you must first clearly write 4 things：

What type of video content is this
What the subject is doing in the current scene
Which evidence determines this judgment
Which visual facts must never be generated incorrectly

Must also inherit the replica main brain from step 3.6：

scene_prompts.json Must carry both generation_contract and style_consistency
When step 8 generates character reference images, first read generation_contract.character_reference_mode, then apply style_consistency
When step 9 generates scene videos, it must also first read the same generation_contract.scene_generation_mode, then apply style_consistency
Multi-person scenes must add scenes[] in visible_characters, and cannot only write main_character
Each scene must carry semantic_anchor, directly attach step 3.6'scontent type / subclass judgment / behavior summary / key evidence / negative constraintsto the generation layer
The core character's character_ref_prompts cannot only have bust identity images; at least the main character and key costume characters must add full_body_outfit
If style inconsistency appears, prioritize going back to fix step 3.6; do not only apply temporary patches on a single image or a single video segment

Suggested Prompt composition order：

Unified prefix consistent with target_visual_mode + original video size ratio
Scene and spatial skeleton
Subject scale and organizational relationships
Semantic anchors (content type + subclass judgment + behavior summary)
Discriminative evidence (costume / shoes / props / text / environment)
Lighting and color tone
Negative constraints (avoid generating training as a performance, avoid generating uniforms incorrectly, avoid generating group portraits as scattered passersby)

Example output(neutral placeholder example, only demonstrating structure):

视觉风格提示词：
电影感，warm_yellow色调，缓慢推拉镜头，流畅转场

场景1提示词：
[按 target_visual_mode 填写统一前缀], 9:16 vertical, 540x960, 校园入口外景，日间自然光，角色甲（东亚男性，25岁，黑色短发，浅色衬衫）

场景4提示词：
[按 target_visual_mode 填写统一前缀], 9:16 vertical, 540x960, 室内餐桌场景，暖色灯光，角色乙（东亚男性，35岁，短发，米白衬衫）

Output：

output/prompts/visual_style_prompt.txt - Visual style prompt
output/prompts/scene_prompts.json - Prompt for each scene

⚠️ scene_prompts.json must strictly follow the JSON schema below; the scripts in steps 8-9 depend on these fields：

{
  "global_style": "电影感，warm_yellow色调，缓慢推拉镜头",
  "generation_contract": {
    "source_visual_mode": "live_action",
    "target_visual_mode": "live_action",
    "mode_lock_reason": "原视频是真人，最终视频仍统一保持真人出镜；但平台对真人参考图有限制，因此步骤8统一使用写实人物插画参考图桥接",
    "fallback_policy": "如果模型能力不足，只允许整项目统一降级，不允许局部混用",
    "character_reference_mode": "pseudo_realistic_human_illustration",
    "scene_generation_mode": "live_action",
    "required_keywords": ["live action", "realistic photo style", "真人出镜"],
    "forbidden_keywords": ["anime", "illustration", "角色设定图", "live action 与动漫词混用"],
    "consistency_rules": [
      "步骤8和步骤9都必须服从同一份 generation_contract",
      "scene_generation_mode 必须与 target_visual_mode 一致",
      "如果 character_reference_mode 与 target_visual_mode 不同，只允许整项目统一采用写实人物插画桥接方案，且 source_visual_mode 必须是真人"
    ]
  },
  "style_consistency": {
    "style_family": "统一项目风格名称",
    "character_render_mode": "角色参考图必须完全服从 character_reference_mode；真人项目走桥接时，也必须是写实人物插画参考图，并保持真人比例、真实皮肤明暗和布料褶皱",
    "scene_render_mode": "场景视频必须完全服从 scene_generation_mode，不能被参考图介质带偏",
    "lighting_rule": "统一光线逻辑",
    "palette_rule": "统一主色倾向和服装配色",
    "background_rule": "角色参考图必须单人、纯净浅色背景、不带第二个人",
    "framing_rule": "角色参考图默认平视胸像；核心角色还要补全身服装锚点图",
    "costume_rule": "服装材质、颜色、版型沿用原视频",
    "character_prompt_block": "步骤8每次生角色图都必须先读取 character_reference_mode；只有 source_visual_mode 为真人时，才能走写实人物插画桥接；多人场景至少给核心角色补 identity_portrait + full_body_outfit",
    "scene_prompt_block": "步骤9每次生场景都必须先读取 scene_generation_mode；输入参考图只锁定身份、发型、服装和鞋子，最终画面仍必须服从 scene_generation_mode；多人场景必须用 main_character + visible_characters 的多张参考图防止同脸",
    "must_keep": ["scene_generation_mode 与 target_visual_mode 一致", "多人场景的人脸和发型必须分开", "核心角色的裤型和鞋型必须稳定"],
    "negative_constraints": ["角色参考图不要画成漫画脸、二次元大眼或赛璐璐阴影", "不要只喂一张参考图去生成多人场景", "不要局部换风格"]
  },
  "video_generation": {
    "source_resolution": "540x960",
    "source_aspect_ratio": "9:16",
    "target_resolution": "540x960",
    "provider_ratio": "9:16",
    "provider_resolution": "720p",
    "prompt_prefix": "[按 target_visual_mode 填写统一前缀], 9:16 vertical, 540x960"
  },
  "characters": [
    {
      "id": "char_0",
      "name": "角色甲",
      "gender": "男性",
      "age": "25-35岁",
      "appearance": "东亚男性，黑色短发，五官立体",
      "clothing": "浅色衬衫→深色外套"
    }
  ],
  "character_ref_prompts": [
    {
      "character_id": "char_0",
      "reference_type": "identity_portrait",
      "prompt": "先读取 character_reference_mode，再按唯一目标形态写身份参考图提示词，锁定脸型和发型"
    },
    {
      "character_id": "char_0",
      "reference_type": "full_body_outfit",
      "prompt": "先读取 character_reference_mode，再按唯一目标形态写全身服装锚点提示词，锁定长裤和鞋子"
    }
  ],
  "scenes": [
    {
      "scene_id": 1,
      "prompt": "[按 target_visual_mode 填写统一前缀], 9:16 vertical, 540x960, 校园入口外景，日间自然光，char_0（东亚男性，25岁，黑色短发，浅色衬衫），自然站立",
      "duration": 5.2,
      "main_character": "char_0",
      "visible_characters": ["char_0", "char_1", "char_2"],
      "semantic_anchor": {
        "content_type": "训练 / 表演 / 仪式 / 课堂 / 比赛 / 商品展示 / 剧情 / unknown",
        "subtype_judgment": "更细的子类判断，如：女子舞踊基础练习 / 课堂示范 / 队列排练",
        "behavior_summary": "一句话说明主体到底在做什么，不能只写抽象词",
        "evidence_to_preserve": ["必须在生成中保留的证据，如鞋型 / 字样 / 道具 / 队形"],
        "negative_constraints": ["最容易生成错的点，如不要把训练生成成演出 / 不要把分趾鞋生成成运动鞋"]
      },
      "time_range": "0-5.2s"
    }
  ]
}

Field descriptions：

video_generation: size and ratio information automatically inherited from the original video; step 9 must prioritize following this configuration
generation_contract: the unified generation form locked in step 3.6; both step 8 and step 9 must consume it first and cannot bypass it
style_consistency: project-level unified style template, which is the expanded description of generation_contract , and must be mandatorily consumed by step 8 and step 9
characters: character definition list (id, gender, age, appearance, clothing), used by step 8 to automatically generate character reference images
character_ref_prompts: prompt for the character reference image; supports reference_type=identity_portrait/full_body_outfit
scenes: scene list (scene_id, prompt, duration, main_character, visible_characters), used by step 9 to generate videos
semantic_anchor: scene-level semantic anchors, mandatorily penetrating step 3.6's ontology / subclass / behavior / evidence / negative constraints to the generation layer
main_character: value must correspond to characters[].id, used to look up character reference images
visible_characters: the character anchors in multi-person scenes that are actually to be fed to step 9; if missing, same-face issues are extremely likely

Large on-site mode (crowd mode)：

When character_detection.json in visible_people_stats.stable_visible_people_estimate > 10 , it is by default judged aslarge group portrait scene
Note: crowd mode is only a reference image generation strategy, not the final semantic conclusion; because there are many people, do not hastily write the content as "event site"
Even if it enters crowd mode, it is still necessary to continue judging whether it is actually training, performance, competition, classroom, ceremony, assembly, or another multi-person organized activity
unique_characters is still retained in the analysis results as underlying clustering reference; however, the number of characters, formation relationships, and scene scale must prioritize using 单帧可见人数 and 稳定可见人数估计
In crowd mode, by default, models are not built individually for every passerby, but a small number of identity anchors should still be retained for the foreground subjects and key costume characters.
Step 9 mainly uses scene prompts + visible_characters to replicate the multi-person on-site relationships: foreground subjects, background crowd, positional hierarchy, and activity atmosphere, rather than generating a separate reference image for each passerby.
Only if the user explicitly requests per-character modeling should you switch from crowd mode back to per-character reference image mode.

Step 7: Generate TTS replication guidance

Execution method: Execute only when ASR has valid text

Pre-execution check: If output/analysis/asr_result.json 's full_text is empty, or segments is empty, it means the video has no valid narration text that can be replicated. In this case,skip Step 7 and Step 10, enter pure visual replication mode, and no longer generate or consume tts_guide.json。

The first draft has been automatically generated：output/prompts/tts_guide.json reference_text (complete narration text), duration_target, and the TTS parameters mapped from voice_style have been automatically filled in from the ASR results. Only when the ASR text is non-empty does Claude need to review these fields.

Data source：

ASR results:output/analysis/asr_result.json

TTS replication parameters：

{
  "tts_parameters": {
    "gender": "女性",
    "tone": "温柔亲切",
    "speed": "中等",
    "emotion": "温暖鼓励",
    "rhythm": "平稳节奏"
  },
  "reference_text": "大家好，今天要分享的是...",
  "duration_target": 45.2
}

Output：

output/prompts/tts_guide.json - TTS replication guidance (generated and consumed only when ASR has valid text)

⚠️ tts_guide.json must strictly comply with the following JSON schema; the script in Step 10 depends on these fields：

{
  "tts_parameters": {
    "gender": "女性",
    "tone": "温柔亲切",
    "speed": "中等",
    "emotion": "温暖鼓励",
    "rhythm": "平稳节奏"
  },
  "reference_text": "大家好，今天要分享的是...",
  "duration_target": 45.2
}

Field descriptions：

tts_parameters.gender: Gender (male/female), used for voice mapping
tts_parameters.tone: Tone (gentle/ impassioned/ calm), used for voice mapping
tts_parameters.speed: Speech rate (slow/ relatively slow/ medium/ relatively fast/ fast), used for rate adjustment
tts_parameters.emotion: Emotion (warm/ serious/ relaxed), used for voice mapping
reference_text: Complete narration text, used for TTS input

Phase 3: Precise style replication and integrated audio-visual generation

Step 8: Generate unified-form character reference images

Execution method: Call Python script (ApiMart Gemini image generation)

python scripts/image_generator.py \
  --prompts_json output/prompts/scene_prompts.json \
  --output_dir output/角色参考图/ \
  --config config/api_config.yaml

The script reads scene_prompts.json in generation_contract、style_consistency and the character_ref_prompts / characters list. First confirm scene_generation_mode == target_visual_mode, then generate character reference images according to character_reference_mode . Only when source_visual_mode is explicitly a real person is Step 8 allowed to uniformly switch to pseudo_realistic_human_illustration as the realistic human illustration bridging reference image. Anime source videos must continue to use the anime character reference image template.

Mandatory rules：

If generation_contract has not been locked yet, Step 8 must directly report an error and stop
If the target form of the character reference image is inconsistent with the scene target form of Step 9, the Step 8 output cannot continue to be used for Step 9
The refs_manifest.json generated for the current project must record this generation_contractfor Step 9 to verify
If a real-person project cannot directly feed real-person reference images, the bridging image must be arealistic human illustration reference image: non-photographic, real human proportions, realistic skin light and shadow, realistic fabric folds
Anime source videos cannot use realistic human illustration bridging and must continue to use the anime character reference image template
As long as the bridging image showscomic face / anime big eyes / cel shading / cartoon line art, it must be rejected and regenerated, and cannot flow into Step 9

Character reference image strategy：

Standard mode（stable_visible_people_estimate <= 10): Select the main subject to generate reference images based on the estimated stable visible headcount
Every key character entering the generation pipeline should have at least identity_portrait
For the main character and key costume characters, you should also add full_body_outfit, specifically to lock in pants, shoes, and costume fit
Large-scale on-site mode / crowd mode（stable_visible_people_estimate > 10): By default, do not "model every single person individually," but still retain a small number of front-row subject anchors; you cannot rely on just 1 protagonist image to carry the entire scene
The purpose of crowd mode is not to reduce the number of subjects to recreate, but to avoid incorrectly reducing a large group scene video into "29 independent characters modeled one by one," which would destroy the sense of a real on-site event
In crowd mode, multi-person information should be retained in scene_prompts.json in the scene description, rather than relying on a large number of single-character reference images

Configuration requirements：

config/api_config.yaml configure models.image(default provider: apimart)
API key via config/api_keys.yaml or environment variable APIMART_API_KEY / VOLCENGINE_API_KEY settings

Output：

output/角色参考图/*.jpg - Unified form reference image for each character
output/角色参考图/refs_manifest.json - Reference image manifest

Step 9: Generate video clips

Execution method: Call a Python script (Seedance 2.0 API)

python scripts/video_generator.py \
  --prompts_json output/prompts/scene_prompts.json \
  --refs_dir output/角色参考图/ \
  --output_dir output/videos/ \
  --config config/api_config.yaml \
  --parallel 1

The script does the following for each scene:

Find the reference images for the scene's main character + visible_characters (from refs_manifest.json)
First verify whether refs_manifest.json and the current generation_contract are consistent; if not, stop immediately
Build a prompt with @图片N invocation statements
Call the Seedance 2.0 API to submit the task → poll → download the video
Before submission, each scene prompt must automatically overlay the same generation_contract and style_consistency
Supports up to 9 reference images (character + composition + style, etc.)

Mandatory requirements for multi-person scenes：

main_character can only solve "who the protagonist is," but cannot solve "the whole group should not have the same face"
visible_characters must list the character anchors that truly need to be fed into Step 9
If a multi-person scene is fed only one protagonist reference image, it is very easy to end up with "everyone growing into the same face"
If the main character does not have full_body_outfit, it is very easy to get "pants, shoes, and lower-body fit drift"

Mandatory parameters(priority determined by scene_prompts.json.video_generation ,config as fallback):

ratio: inherit the original video aspect ratio (e.g., 9:16 / 16:9 / 1:1)
resolution: API generation tier (currently usually 720p or 1080p), then normalize to the exact original video size after generation
duration: 4-15 seconds (consistent with the original video scene duration, automatically clamped)
max_reference_images: 9

Size rules：

The final output size must match the original video(for example, if the original video is 540x960, then the generated scene video must also be saved as 540x960）
If the video generation API can only work at tiered resolutions, it is allowed to first generate at the nearest tier, then locally uniformly scale/pad to the original video size

Resume from breakpoint: existing scene_*.mp4 files are automatically skipped.

Output：

output/videos/scene_*.mp4 - Video clip for each scene
output/videos/videos_manifest.json - Video manifest

Step 10: Generate TTS voiceover

Execution method: Only execute when Step 7 has confirmed that valid narration text exists

Skip condition: If asr_result.json has no available text (full_text == ""), then the video is considered to haveno TTS replication requirement. Directly skip step 10 and enter pure visual replication mode.

python scripts/tts_generator.py \
  --text "<解说文本>" \
  --output_path output/audio/tts_narration.wav \
  --tone "温柔" \
  --speed "中等" \
  --emotion "温暖" \
  --service edge_tts

Supported services：

edge_tts(default, free): Microsoft Edge TTS, supports multiple Chinese voices
custom: custom TTS API (specified via --custom_api_url )
sine: sine wave fallback (for debugging only)

Voice mapping（tone + emotion → voice）：

Tone+Emotion	edge_tts voice
Warm/Female	zh-CN-XiaoxiaoNeural
Calm/Neutral	zh-CN-XiaoyiNeural
Passionate/Male	zh-CN-YunxiNeural

You can directly specify the voice name via --voice to override the automatic mapping.

Output：

output/audio/tts_narration.wav - TTS dubbed audio
output/audio/tts_narration_tts_result.json - TTS parameter record

Step 11: Audio-video compositing

Execution method: divided into two modes based on whether TTS exists

Mode A: has TTS text → execute 11a + 11b

# 11a 拼接场景视频
python scripts/scene_concat.py \
  --video_dir output/videos/ \
  --output_path output/videos/merged_video.mp4 \
  --order_json output/prompts/scene_prompts.json

# 11b 合成音视频（使用ASR时间戳精确对齐）
python scripts/audio_video_mixer.py \
  --video_path output/videos/merged_video.mp4 \
  --audio_path output/audio/tts_narration.wav \
  --output_path output/复刻视频.mp4 \
  --asr_json output/analysis/asr_result.json

Mode B: no TTS text → execute only 11a (pure visual replication)

Default rule: when ASR is empty, the final deliverable is defined as theno-replication-dubbing version. That is, only concatenate video clips, do not execute audio_video_mixer.py。

# 11a 拼接场景视频（此时 merged_video.mp4 即最终交付物）
python scripts/scene_concat.py \
  --video_dir output/videos/ \
  --output_path output/videos/merged_video.mp4 \
  --order_json output/prompts/scene_prompts.json

Current Skill's default definition: in pure visual replication mode, the final output is merged_video.mp4。
does not retain the original video audio track by default; "retaining the original audio track" is an additional requirement that the user must explicitly request, and it should be implemented through a separate audio strategy, rather than reusing the TTS replacement pipeline.

# 11a 拼接场景视频
python scripts/scene_concat.py \
  --video_dir output/videos/ \
  --output_path output/videos/merged_video.mp4 \
  --order_json output/prompts/scene_prompts.json

# 11b 合成音视频（使用ASR时间戳精确对齐）
python scripts/audio_video_mixer.py \
  --video_path output/videos/merged_video.mp4 \
  --audio_path output/audio/tts_narration.wav \
  --output_path output/复刻视频.mp4 \
  --asr_json output/analysis/asr_result.json

Output：

output/复刻视频.mp4 - Final video (when TTS text exists)
output/videos/merged_video.mp4 - Final video of pure visual replication (when no TTS text exists)

📋 Complete step summary

Stage	Step	Content	Execution method	Output
Stage 1	Step 1	Intelligent keyframe extraction	Python script	keyframes/
Stage 1	Step 1.5	Character detection and cross-frame clustering	Python script (InsightFace)	character_detection.json
Stage 1	Step 2	Color and motion analysis	Python script	color_analysis.json
Stage 1	Step 3	ASR speech recognition ⭐	Python script	asr_result.json
Stage 2	Step 3.5	Narrative line analysis ⭐	Claude analysis	narrative_analysis.json
Stage 2	Step 3.6	General semantic analysis skeleton ⭐	Claude analysis	semantic_analysis.json
Stage 2	Step 4	Deep visual analysis (based on narrative line) ⭐	Claude+Read tool	coherence_analysis.json
Stage 2	Step 5	Audio-visual correlation analysis	Claude analysis	audio_visual_correlation.json
Stage 2	Step 6	Generate visual prompts	Claude generation	scene_prompts.json
Stage 2	Step 7	Generate TTS guidance (optional)	Claude generation	tts_guide.json
Stage 3	Step 8	Generate character reference images	Python script (ApiMart Gemini image generation)	Character reference images/
Stage 3	Step 9	Generate video clips	Python script (Seedance 2.0 API)	videos/scene_*.mp4
Stage 3	Step 10	Generate TTS voiceover (optional)	Python script (edge_tts)	tts_narration.wav
Stage 3	Step 11	Audio-video compositing / visual-only delivery	Python script	replica_video.mp4 / merged_video.mp4

Complete data flow chain

原视频.mp4
    │
    ├─→ [步骤1] smart_keyframe_extractor.py → keyframes/*.jpg
    ├─→ [步骤1.5] character_detector.py → character_detection.json + characters/*.jpg
    ├─→ [步骤2] video_analyzer.py → color_analysis.json
    └─→ [步骤3] asr_transcriber.py
            ├─→ extracted_audio.wav
            └─→ asr_result.json
                    │
                    ↓
            [步骤3.5] Claude分析 → narrative_analysis.json
                    │
                    ↓
            [步骤3.6] Claude分析 → semantic_analysis.json
                    │
                    ↓
    [步骤4] Claude Read(keyframes) + 叙事线 → coherence_analysis.json

    ↓

[步骤5] Claude分析 → audio_visual_correlation.json

    ↓

[步骤6-7] Claude生成提示词
    ├─→ scene_prompts.json（视觉风格+场景提示词+角色参考图提示词）
    └─→ tts_guide.json（TTS参数+文本）

    ↓

[步骤8] image_generator.py（ApiMart Gemini 图片生成）→ 角色参考图/*.jpg

    ↓

[步骤9] video_generator.py（Seedance 2.0 API + 角色参考图）→ videos/scene_*.mp4

    ↓

[步骤10] tts_generator.py（edge_tts，可选）→ tts_narration.wav

    ↓

[步骤11a] scene_concat.py → merged_video.mp4
    ├─→ 若无有效 ASR / 无 TTS 需求：merged_video.mp4 即最终输出
    └─→ 若有有效 ASR / 需要 TTS：进入步骤11b

[步骤11b] audio_video_mixer.py → 复刻视频.mp4

Notes

1. ASR takes priority, but visual-only degradation is allowed ⭐ Most important

Try ASR first, then determine whether to enter the text-driven workflow
When valid ASR text is available: text understanding > image understanding; ASR results guide the narrative line, character identification, and audio-visual alignment
When no valid ASR text is available: immediately switch to visual-only replication mode, focusing on keyframes, character detection, and scene narrative, and no longer force the TTS pipeline
Skipping ASR detection is prohibited; however, empty ASR does not mean the workflow failed, but rather that the video is not suitable for TTS replication

2. Scheduling mode

This Skill is the scheduling core: after Claude reads SKILL.md, it executes the entire workflow step by step
Steps 1-3: Claude calls Python scripts (perfect_replication_workflow.py or individual scripts)
Steps 3.5-7 (including step 3.6): Claude executes directly (analyzing ASR text, completing the semantic skeleton, viewing keyframes, generating prompts)
Steps 8-9: Claude calls Python scripts (image_generator.py、video_generator.py, generated via API)
Steps 10-11: Claude calls Python scripts; among them, step 10 and step 11b are executed only when valid ASR text exists
API configuration：config/api_config.yaml defines the provider,config/api_keys.yaml stores the key (or uses environment variables)

3. Character identification workflow

Step 1.5 automatically detects and clusters faces (character_detection.json）
Identify characters from ASR text (name, identity, relationships)
Complete semantic_analysis.json the subject scale, behavior, evidence, and subclass determination in
Assign keyframes to characters from ASR timestamps
Use read_image to verify character appearance
Generate the final character definition table

4. Character ID naming convention ⭐ Most important

The character label output by step 1.5 (e.g., char_0, char_1) isthe unified character ID across the entire pipeline
When step 6 generates scene_prompts.json ,characters[].id、scenes[].main_character、scenes[].visible_characters[] the same ID must be used
When step 8 generates reference images, the default filename for the identity image is {character_id}.jpg, and the full-body clothing anchor image is {character_id}__full_body_outfit.jpg
When step 9 looks up reference images, it reads both main_character and visible_characters
If the IDs are inconsistent, step 9 will not find the reference image, and the quality of the generated video will drop severely
If crowd mode is entered, character IDs that do not have individually generated reference images should still be retained in the analysis results; character records must not be deleted just because reference image generation was skipped
But scene_prompts.json the characters list that actually enters the generation pipeline should converge according to 稳定可见人数估计 , and unique_characters must not be copied verbatim as the number of people

5. Step dependencies

Steps 1-3 can be executed in parallel (or call perfect_replication_workflow.py to complete them all at once, including step 1.5)
Step 3.5 must be completed before or at the same time as step 3.6(the narrative line provides text semantics)
Step 3.6 must be completed before step 4 and step 6(behavior, evidence, and subclass determination precede character refinement and prompt writing)
Step 4 must use the Read tool to view the keyframe images
When generating prompts in step 6, you must reference the color analysis from step 2, the narrative line from step 3.5, and the semantic skeleton from step 3.6
The JSON generated in step 6 must pass Schema validation（scripts/schema_validator.py), the entry points for steps 8-9 will automatically validate
If the ASR text is empty, steps 7, 10, and 11b are skipped by default
Steps 8-9 depend on scene_prompts.json; steps 10-11b depend on TTS-related files only when a valid tts_guide.json exists
Steps 8-9 require that the API key has been configured (config/api_keys.yaml or the environment variable VOLCENGINE_API_KEY）

6. JSON template and Schema validation

Each JSON output in steps 3.5-7 (including step 3.6) has a standard template, located at assets/schema_templates/
When generating JSON, refer to the field definitions in the template to ensure no required fields are omitted

scripts/schema_validator.py can run independently to validate any intermediate file:

python scripts/schema_validator.py --file output/analysis/semantic_analysis.json --type semantic

Step 8 (image_generator.py) and step 9 (video_generator.py) entry points have integrated automatic validation and will report clear errors when fields are missing

7. Other notes

Character definitions must be consistent: the same character uses exactly the same description in all scenes
Video dimensions must follow the original video: they must not be hardcoded at the Skill level as 16:9 / 1280x720
When asr_result.json in full_text is empty, you should switch topure visual replication mode, and no longer force TTS generation
In pure visual replication mode, the default final output is output/videos/merged_video.mp4
When visible_people_stats.stable_visible_people_estimate > 10 , enter crowd modeby default: retain all character detection results, but step 8 does not model every character individually, retaining only a small number of foreground subjects and key clothing anchors
"Retain the original video audio track" is not the current default pipeline; only when the user explicitly requests it should it be handled as a separate audio strategy
Whisper dependency: step 3 requires installing the openai-whisper package (or using a sub-agent)

8. Unified generation form prompt ⭐ Mandatory requirement

All scene prompts and character reference image prompts must first comply with generation_contract。

There are only 3 allowed final video target forms：

live_action
stylized_live_action
anime

Allowed bridging reference image forms：

pseudo_realistic_human_illustration

Compatibility note: in old projects safe_stylized_human is regarded as a historical name; newly generated files should no longer write this value.

Rules：

A project can only choose 1 type
scene_generation_mode must match target_visual_mode exactly
character_reference_mode is usually consistent with target_visual_mode ; if a live-action project is restricted by the platform and cannot directly use live-action reference images, the entire project can be uniformly changed to pseudo_realistic_human_illustration
pseudo_realistic_human_illustration means: realistic human illustration reference image, not manga, not anime, not a cartoon face
If source_visual_mode = anime, then character_reference_mode must not use pseudo_realistic_human_illustration
If the model capability is insufficient, onlyuniformly downgrade the entire project
It is forbidden for “one segment's character reference image to be a bridge safety image while another segment suddenly switches to a different bridging method” — this kind of local chaotic switching
It is forbidden for “a multi-person scene to be fed only one protagonist image, resulting in the entire team having the same face” — this kind of structural error

Example prefix：

live_action

realistic photo style, live action, 9:16 vertical, 540x960,

stylized_live_action

stylized live action, realistic human proportions, 9:16 vertical, 540x960,

anime

anime style, clean cel shading, 9:16 vertical, 540x960,

When generating prompts in step 6, you must check：

Whether the current project prefix matches target_visual_mode consistently
Whether the character reference image prompt and the scene prompt belong to the same form
Whether the aspect ratio and resolution are consistent with the original video
Whether cross-form conflicting words appear (such as live action and 插画角色设定图 mixed together)

Resource index

Core scripts

Smart keyframe extraction: scripts/smart_keyframe_extractor.py
Character detection and clustering: scripts/character_detector.py (step 1.5, InsightFace face embedding clustering)
Video color analysis: scripts/video_analyzer.py
ASR speech recognition: scripts/asr_transcriber.py
API client: scripts/api_client.py (credential loading, image/video API wrapper)
Character reference image generation: scripts/image_generator.py (step 8, ApiMart Gemini image generation)
Scene video generation: scripts/video_generator.py (step 9, Seedance 2.0 API)
TTS generation: scripts/tts_generator.py (step 10, edge_tts preferred + switchable)
Video concatenation: scripts/scene_concat.py (step 11a, ffmpeg)
Audio-video compositing: scripts/audio_video_mixer.py (step 11b)
JSON Schema validation: scripts/schema_validator.py (validates intermediate JSON for steps 3.5-7 including 3.6)
Draft generator: scripts/draft_generator.py (automatically generates a JSON draft for steps 3.5-7 including 3.6 from the outputs of steps 1-3)
Complete workflow: scripts/perfect_replication_workflow.py (complete pipeline for steps 1-11)

JSON templates (standard output format for steps 3.5-7 including 3.6)

Narrative line analysis: assets/schema_templates/narrative_analysis_template.json
Semantic analysis skeleton: assets/schema_templates/semantic_analysis_template.json
Deep visual analysis: assets/schema_templates/coherence_analysis_template.json
Audio-visual correlation analysis: assets/schema_templates/audio_visual_correlation_template.json
Scene prompts: assets/schema_templates/scene_prompts_template.json
TTS replication guide: assets/schema_templates/tts_guide_template.json

Configuration files

API configuration: config/api_config.yaml
API key example: config/api_keys.yaml.example (copy to api_keys.yaml and fill in the key)

Reference materials

ASR output format: references/asr_output_format.md
Prompt templates: references/prompt_templates.md
Style category: references/style_categories.md

About this skill

Task Objective

Prerequisites

Core Concept

⭐ Multi-layer video semantic analysis: first make a rough judgment, then unravel step by step

6-layer, 16-dimensional semantic analysis (general)

Layer 0: Media baseline

Layer 1: Content mechanism judgment

Layer 2: Scene semantic skeleton

Layer 3: Subject and Behavior

Layer 4: Discriminative Evidence

Layer 5: Generation Constraints

⭐ ASR priority + tiered fault tolerance mechanism

⚠️ Tiered fault tolerance mechanism

Level 1: ASR success + clear narrative line (best) ✅

Level 2: ASR success but narrative line unclear (second best) ⚠️

Level 3: No ASR or ASR failure (fallback) 🛡️

Automatic Fallback Process

Operation Steps

Phase 1: Narrative Understanding (ASR First)

Step 1: Intelligent Keyframe Extraction

Step 1.5: Character Detection and Cross-Frame Clustering

Step 2: Color and motion analysis

Step 3: ASR speech recognition ⭐ Core step

Step 3.5: Narrative Line Analysis + Rough Content Classification ⭐ Key Step

3.5.1 Rough Content Classification (Must Be Done First)

3.5.2 Narrative Theme Identification

3.5.3 Character Identification (from Text)

3.5.4 Narrative Arc

Step 3.6: 6-Prism Semantic Analysis Skeleton (Layer-by-Layer Convergence)

Step 4: Deep visual analysis (subject behavior + evidence extraction)

Step 4.1: Assign keyframes based on the narrative line and scene

Step 4.2: Use read_image to judge subject behavior

Step 4.3: Use read_image to extract discriminative evidence

Step 4.4: Use read_image to verify character appearance

Step 4.5: Character definition table (based on narrative line + visual verification)

Step 5: Audio-visual correlation analysis + subclass determination (agent)

Stage 2: Intelligent Multi-dimensional Prompt Generation

Step 6: Generate visual style prompts

Step 7: Generate TTS replication guidance

Phase 3: Precise style replication and integrated audio-visual generation

Step 8: Generate unified-form character reference images

Step 9: Generate video clips

Step 10: Generate TTS voiceover

Step 11: Audio-video compositing

Mode A: has TTS text → execute 11a + 11b

Mode B: no TTS text → execute only 11a (pure visual replication)

📋 Complete step summary

Complete data flow chain

Notes

1. ASR takes priority, but visual-only degradation is allowed ⭐ Most important

2. Scheduling mode

3. Character identification workflow

4. Character ID naming convention ⭐ Most important

5. Step dependencies

6. JSON template and Schema validation

7. Other notes

8. Unified generation form prompt ⭐ Mandatory requirement

Resource index

Core scripts

JSON templates (standard output format for steps 3.5-7 including 3.6)

Configuration files

Reference materials