paper-cutout-remotion

Paper Cutout Remotion (paper cutout style layered animation)

Turn a storyboard script into a paper cutout style layered Remotion video. Core methodology:It is not about generating one pretty image and adding camera movement, but about breaking the frame into four separate layers: background/back row/main subject/foreground, and having each layer move at a different rhythm, using front-to-back occlusion to create a sense of depth and dimensionality.

This method is not only suitable for historical subjects. As long as the frame can be broken into "background, protagonist, supporting characters, foreground"—historical figures, educational popular science, business stories, character relationships, urban development—it can all be produced with the same workflow.

Toolchain: apiz CLI (image generation) + Python/Pillow/NumPy (green screen keying and layer separation) + MiniMax (apiz speak voiceover, direct API fallback) + Remotion (React controls layers/animation/subtitles/rendering) + FFmpeg (acceptance checks).

Supporting references：

• Component API quick reference:references/components.md
• Four-layer separation methodology:references/layering.md(the soul of the paper cutout style, required reading)
• apiz prompt recipe:references/prompt-recipes.md

New project workflow (10 steps)

Execute in the following order, and only move to the next step after each step is complete. If the order is messed up, it will lead to wrong asset proportions, failed keying, and unclear hierarchy.

1. Determine the shots (do not draw images directly)

Define the shots first, do not draw images first. The same character has different sizes, positions, and orientations in a wide shot and a close-up. If you generate assets randomly first, you will inevitably have to redo them later.

For each shot, first answer:Who is the largest? Where are the supporting characters? Where is the horizon?

• Wide shot: convey scale (e.g. "Prosperous Tang Chang'an", the emperor is largest and centered, the ministers are smallest)
• Close-up shot: express emotion/action (e.g. "Ten Thousand Nations Pay Tribute", the central figure is largest, the kneeling front row is next)

2. Break each shot into four layers

See references/layering.md. Each shot should be broken into at least:

• Background layer(landscape/palace/paper texture, no characters)
• Back row layer(distant ministers/attendants, tertiary)
• Main subject layer(protagonist, primary, largest)
• Foreground layer(nearby characters/decorations, secondary, occlusion creates depth)

3. Scaffolding

mkdir "<VIDEO_WORKSPACE>/<项目名>"
cp -R "./skills/paper-cutout-remotion/templates/remotion-project/" \
      "<VIDEO_WORKSPACE>/<项目名>/"
cd "<VIDEO_WORKSPACE>/<项目名>" && npm install

⚠️ Use native npm install, not rtk npm install(rtk will translate it into npm run install and cause an error).

Create the asset directory at the project root:

mkdir -p public/assets/plates public/assets/layers public/assets/source
mkdir -p public/audio/narration public/audio/sfx

4. Generate character-free background plates (gen_plates.py)

Background plates should retain only the environment (landscape/palace/paper texture/torn paper edges/decorations).Absolutely do not draw the main characters into the background—if you draw them in, they can no longer fly in independently, sway, or be resized.

# 单张
python scripts/gen_plates.py "唐代长安宫殿全景，水墨远山，宣纸纹理，撕纸边缘" \
    public/assets/plates/01-tang-wide-bg.png

# 批量（plates_spec.yaml）
python scripts/gen_plates.py --batch plates_spec.yaml public/assets/plates/

The script will automatically append the "no characters" constraint. For the prompt recipe, see references/prompt-recipes.md。

5. Generate character asset sheets (gen_characters.py)

Generate characters on a pure green background (to facilitate later chroma key cutouts). The prompt must enforce: orientation, full body, white paper-cut outline, pure green background, no text or watermark. The script will automatically append these constraints.

# 先写 assets_spec.yaml（角色清单，见 scripts/gen_characters.py 注释）
# dry-run 检查提示词
python scripts/gen_characters.py assets_spec.yaml public/assets/source/ --dry-run
# 实际生成
python scripts/gen_characters.py assets_spec.yaml public/assets/source/

6. Green-screen cutout + split into independent PNGs (split_sheet_green.py)

Cut out asset sheets with a green background into transparent PNGs; multi-person asset sheets are split into independent files by grid.

# 单人物（只抠图不拆）
python scripts/split_sheet_green.py public/assets/source/emperor.png \
    public/assets/layers emperor 1

# 6 人群臣素材表（自动推算网格）
python scripts/split_sheet_green.py public/assets/source/ministers.png \
    public/assets/layers minister 6
# 显式 3 列 2 行
python scripts/split_sheet_green.py public/assets/source/ministers.png \
    public/assets/layers minister 6 --cols 3 --rows 2

After cutting out, check *-full-alpha.png(full cutout result) to confirm the green screen is clean and there is no green residue on the edges. If there is green residue, increase split_sheet_green.py 's EDGE_FEATHER, or reinforce the prompt with "pure flat solid green, no gradient".

7. Static layout (layout first, then add animation)

Do not add animation as soon as you get the assets. First, in scenes.tsx , place all charactersin a static layout, and check:

• Is the protagonist the largest and most eye-catching?
• Are any supporting characters blocking the protagonist's face, hands, or key props?
• Are all characters' feet on the same reasonable ground plane?
• Do the orientations match the scene relationships?

Only modify src/scenes.tsx(scene component) and src/Root.tsx(SCENES table).Do not modify theme.ts / ui.tsx / cutout.tsx(style lock file).

8. Set zIndex occlusion + roleMotion staggered animation

Assign zIndex according to the four-layer model (background 0 / back row 1-2 / main subject 3-4 / foreground 5-6 / text 10+), and set PaperActor and role for each delay：

// 主角先出（delay 4），配角补充（delay 16-22），后排填充（delay 30-40）
<PaperActor src="assets/layers/emperor.png" x={960} y={920} width={650}
            role="primary" delay={4} zIndex={3} seed="s2emp" />
<PaperActor src="assets/layers/maid-left.png" x={620} y={900} width={240}
            role="secondary" delay={18} zIndex={5} seed="s2maid1" />
<PaperActor src="assets/layers/minister.png" x={1500} y={860} width={165}
            role="tertiary" delay={34} zIndex={1} seed="s2min1" />

Apply <BackgroundPan> to the background for a 1-3% slow push-in. For detailed layer specifications, see references/layering.md。

9. Voiceover + subtitles + sound effects (gen_tts.py)

After the visuals are complete, generate narration by scene and use the real duration to cut the shots.

# 写 narration.yaml（见 scripts/gen_tts.py 注释）
python scripts/gen_tts.py narration.yaml --out-dir public/audio/narration

Output s01.mp3 s02.mp3 ... + timeline.json(including frames_playback）。

• Prefer apiz speak (unified authentication, built-in download)
• On failure, fall back to direct connection to api.minimaxi.com/v1/t2a_v2(from .env read minimaxi=KEY）

Use timeline.json's frames_playback to write back the frames of the SCENES table in Root.tsx, and attach <Audio src={staticFile('audio/narration/sNN.mp3')} />。

Sound effect layering (according to the original requirements):

• Narration: one mp3 per segment, directly determining the shot duration
• BGM: looped playback, volume below the voice (<Audio src={staticFile('audio/bgm.wav')} loop volume={0.3} />）
• Entrance sound effects: protagonist sfx/impact.wav, secondary sfx/whoosh.wav, back row sfx/tick.wav

10. Preview + acceptance + render

npm run dev          # Remotion Studio 逐帧预览
npx tsc --noEmit     # 类型检查（长渲染前必做）
# 抽帧检查排版（每场接近结尾的静帧）
npx remotion still Episode --frame=<场景末帧-30> out/check-N.png
npm run render       # 渲染 MP4
ffprobe -v error -show_streams -show_format out/episode.mp4   # 验收

Style DNA (immutable)

Item	Value
Canvas	1920×1080 @ 30fps, rice-white rice-paper background #f3ead6
Four-layer model	Background (z0) → back row (z1-2) → main subject (z3-4) → foreground (z5-6) → text (z10+)
roleMotion three tiers	primary(78/55/0.86) / secondary(58/38/0.90) / tertiary(38/22/0.95)
White border + shadow	drop-shadow(4向 #f5eedc) + drop-shadow(0 18px 9px rgba(20,15,12,.32))(PAPER_FILTER, all four items are indispensable)
Ink color	#1f1a14(warm ink black, titles/strokes)
Accent colors	Cinnabar red #b8322a / blue #2f5d8a / orange #c8732a / green #3f7a3a / gold #b8893a
Fonts	Title/body Song typeface stack Source Han Serif SC; subtitle Hei typeface stack (small-text readability)
Assets	Real PNGs (apiz-generated + green-screen cutout),notdrawn purely with code
Image generation	apiz CLI, default fal-ai/nano-banana-pro(best suited for paper-cutout/illustration style)
Voiceover	MiniMax，apiz speak（speech-2.8-hd)/ direct fallback (speech-02-hd), default female-shaonv

Asset directory conventions

public/
├── assets/
│   ├── plates/        # 背景底板（无人物，gen_plates 生成）
│   │   ├── 01-tang-wide-bg.png
│   │   └── 02-tang-close-bg.png
│   ├── source/        # 原始绿背景素材表（gen_characters 生成，抠图前）
│   │   └── ministers.png
│   └── layers/        # 抠好的透明角色 PNG（split_sheet 产出，scenes.tsx 引用这里）
│       ├── emperor-1.png
│       └── minister-1.png ...
├── audio/
│   ├── narration/     # 旁白 mp3 + timeline.json（gen_tts 产出）
│   ├── sfx/           # 入场音效 impact/whoosh/tick.wav
│   └── bgm.wav        # 背景音乐
└── captions/          # （可选）SRT 字幕

Acceptance checklist (must pass before rendering)

• Clear hierarchy: the protagonist is the largest (about 4 times the width of the back row), supporting characters are not evenly distributed
• No occlusion or continuity errors: supporting characters do not block the protagonist's face/hands/key props; orientation matches the scene relationships
• Feet on the same ground: all characters' y coordinates fall on the same reasonable ground line
• Staggered entrances: characters enter in the order primary→secondary→tertiary, not appearing at the same time
• White edge + shadow: every PaperActor has PAPER_FILTER (paper-cut feel + three-dimensional feel)
• Audio-visual sync: sound effects align with character entrance frames; subtitles follow the narration sentence by sentence
• Single-layer subtitles: one layer at the bottom, no repetition, does not block characters' feet
• FFmpeg frame extraction: render a still frame near the end of each scene, check the items above
• Clean green screen: no green residue on edges after keying (check *-full-alpha.png）

Common pitfalls

• rtk npm install will fail, use native npm install。
• Drawing first, then setting up the shotwill definitely require rework—character proportions/orientation/hierarchy will all be wrong. Be sure to set up the shot first, then generate assets.
• Drawing characters into the backgroundruins it—the background base plate must have no characters, otherwise they cannot fly in independently.
• Impure green screen(with gradients/shadows) → green residue in keying. Strengthen the prompt with "pure flat solid green, no gradient, no shadow on background", or increase EDGE_FEATHER。
• apiz errors are single-line Error: not JSON, when returncode≠0 lib_apiz._run_apiz directly throws stderr.
• Inconsistent model names: apiz speak uses speech-2.8-hd, direct minimaxi API uses speech-02-hd. The two scripts already use the correct names respectively.
• Insufficient apiz balance(may actually be very low) → gen_tts automatically falls back to direct connection; if gen_plates/gen_characters fail, you need to top up, or switch to another drawing model.
• apiz generate --json 's image URL path needs calibration the first time：lib_apiz.extract_image_url() multi-path fallback, on first failure check .last_generate.json add the actual field to the front of the fallback list. See details at the end of references/prompt-recipes.md .
• Repeated seeds will synchronize the floating—each PaperActor's seed carries a scene prefix + number (s2emp、s2maid1）。
• Foreground zIndex must be greater than the subject—only when the foreground occludes the subject does it create depth, not the other way around. See references/layering.md。
• Character's head going out of frame (the most easily made layout mistake): a PaperActor's y is the feet, character display height = width × (图片高/宽比). Tall narrow characters (height/width often 2.3~2.6) can easily have their heads exceed the 1080 canvas. Before setting coordinates, calculate:width × 高宽比 ≤ y - 60(leave 60px headroom). For example, height/width=2.4, y=1030, then width ≤ (1030-60)/2.4 ≈ 404. When unsure, use Python to measure the real size:Image.open(path).getbbox()。
• Single-character keying naming：split_sheet_green.py for count=1 outputs <prefix>-1.png(with -1 suffix). When referencing in scenes.tsx, either use the -1 suffix, or rename to remove it after keying. Multi-character asset sheets are naturally <prefix>-1.png ~ <prefix>-N.png。
• Chrome Headless Shell download slow/fails(Accessing storage.googleapis.com from within China is unstable; 113MB easily times out): the template's remotion.config.ts is already configured to auto-detect the local Chrome (C:/Program Files/Google/Chrome/Application/chrome.exe), skipping the download. If you switch machines and the Chrome path differs, change this configuration.
• Audio mixing fails (audio-mixing directory missing): the root cause is concurrent rendering racing over the Windows temp directory. The template already sets Config.setConcurrency(2) to avoid this. ⚠️ Never run multiple remotion renderat the same time (foreground + background concurrency triggers this issue). Run only one rendering task at a time. For a reference success case see demo-wx-article-3(same configuration, renders correctly with audio).<Audio> components must explicitly pass volume={1}。
• Do not stop at the no-voiceover estimated-timing version—after real TTS generation you must write back a version of frames using timeline.json.
• Narration is cut off (scene switches before it finishes): the root cause is using the wrong frame-count field.timeline.json has two fields:
  • frames_source = ceil(音频秒×30)+15: the number of frames actually needed by the original-speed audio (including 0.5s ending buffer)
  • frames_playback = ceil(frames_source/1.2): the compressed frame count after 1.2x speedup Key: if the audio isplayed at original speed(without adding playbackRate={1.2}), the SCENES table's frames must use frames_source, not frames_playback. Otherwise each scene is 1~2 seconds shorter than the audio, and the narration gets faded out by FadeScene and switched before it finishes. Only when the audio has <Audio ... playbackRate={1.2} /> attached should you use frames_playback. Scenes that need a slow, lingering feel, such as classical poetry, are recommended to use original speed.
• apiz URL already calibrated (measured)：apiz generate --json after success, the image URL is in result.images[0].url，lib_apiz.extract_image_url() already hits this path, no need to manually calibrate again. The image is on a public CDN (cdn-hk.51sux.com), and urllib can download it directly.

Scope of application

This method applies as long as the visuals can be split into "background, protagonist, supporting characters, foreground":

• Historical figures / dynasty narratives (Tang/Song/Ming…)
• Knowledge popularization (figures + concept diagrams)
• Business stories (founder/product/user tiers)
• Character relationships (family trees/organizations/social networks)
• Urban development (landmarks + figures + scenes)

What truly gives paper-cutout animation depth is not generating a few more images, but giving each image a clear narrative position.