srt-whiteboard-animation

SRT Whiteboard Animation (mask choreography + stream drawing)

Convert SRT subtitles into a whiteboard hand-drawn animation:ChoreographyReuse zoned mask reveal (reveal zone by zone in narrative order, zones not yet started are fully hidden, overlap handled with protectedRegions protection);Drawing methodSwitch to streaming strokes—each zone, within its own allowed mask, has the pen tip continuously glide along the skeleton/grid to lay down ink (start with ink → add color), all zones share one persistent canvas, and zones already drawn remain on the canvas. All user-facing explanations, storyboards, configuration, and interface text must be in Chinese.

Unlike frame-by-frame jumps or rectangular wipes: this skill's strokes arecontinuously flowing; unlike whole-image stream: this skill draws sequentially bysubtitle narrative zones, allowing control over the entrance order and timing of each element.

Default implementation parameters

Item	Default requirement
Paper background	Generated images use a warm beige aged-paper color (recommended #F5EBD7); when rendering, sample inward from the four corners of the original image to tint the background; pure white is forbidden.
Drawing method	Stream continuous strokes per zone: start with ink(lay down line art) → add color color(restore original colors); weights ink:color = 2:1。
Stroke path	--ink-path grid(grid, default, stable) or skeleton(skeleton tracing, fits illustrations with clear line art better).
Coloring style	--color-fill contour-wipe(contour scan, default) or brush(brush along trajectory).
Undrawn zones	A zone's allowed mask = rectangle region minus "subsequent zones + protectedRegions"; zones not yet started are fully hidden.
Duration source	Each image's sceneDurationMs comes from the time span of that scene's subtitles (recommended 25–35 seconds per scene).
Edit boxes	The preview console shows all numbered edit boxes by default; edit boxes are not part of the animation frame content.

Unified image-generation visual specification (mandatory)

Source images for all scenes must follow the same visual language; before generation, write the following requirements fully into the image-generation prompt, and check them one by one after generation:

• Style and composition: Minimalist hand-drawn illustration, pure sketch style, restrained Notion-like doodle aesthetic. Focus on conceptual expression, not realism; simple composition, clean background, lots of white space, calm and clear overall emotion, consistent lines/figures/colors within the series.
• Colors and materials: Beige paper background #F5EBD7, dark gray sketch lines; only red, orange, and blue may be used as small conceptual accents. No other accent colors, high-saturation color schemes, or complex textures.
• Figures and objects: Objects are expressed with simple outlines, a few lines, and white space, emphasizing relationships/changes/core concepts rather than real proportions, materials, and details.
• Absolutely forbidden: Any text, words, letters, numbers, fonts, or labels in the scene source image; realistic feel, photographic detail, 3D effects, painterly texture; complex scenes, dense backgrounds, intricate decorations, and high-saturation images.
• Exception for drawing hands: If the user explicitly states that the text on the pen barrel is their logo and requests it be kept, the drawing-hand.png logo on the pen barrel may be kept; it does not belong to the scene source image text and does not need to be removed or redrawn. Without explicit user instruction, still treat the image as having no text.

Confirmation gates (mandatory)

In the default workflow,after each step is completed you must stop and wait for explicit user confirmation, before starting the next step. Before confirmation, you must not generate the next step's images, annotations, previews, videos, or merged files; "no reply", "previous blanket authorization", or "the user did not object" must not be treated as confirmation. If the user requests changes to a previous step, redo only that step and wait for confirmation again after completion.

The only coupled action is:After the annotation JSON is created, you must immediately and automatically open the preview console and load the directory containing that JSON; this is part of the Step 3 delivery and does not require separate confirmation for "opening the preview console". If the browser's File System Access API requires a user gesture, use the browser interface to select this already-determined directory; do not ask the user for extra confirmation or change to having the user open the preview console themselves because of this.

Workflow

1. Read subtitles and produce a strategy (do not generate images). Use scripts/parse_srt.py to parse the SRT into subtitle segments and propose suggested storyboards at 25–35 seconds per scene. Based on this, output the illustration strategy: each scene's scene number, core expression, visual subject, corresponding subtitle interval, and sceneDurationMs. Each scene expresses only one core idea.Stop after completion and wait for the user to confirm the strategy.
2. Generate line art. Only after the user confirms the strategy, generate 16:9 line art on a warm beige old-paper background scene by scene according to the "Unified Image Generation Visual Specification", with background #F5EBD7, keeping ample whitespace between subjects for easy automatic splitting; do not generate text, complex photos, overlapping objects, or elements that conflict with the specification.Stop after completion, show the line art, and wait for user confirmation.
3. First read the subtitles, then look at the image, then annotate and open the preview console. Only after the user confirms the line art, first read the subtitles corresponding to the image, then actually view the image, and obtain the original image's pixel width and height; do not speculate about the image based only on subtitles, nor mechanically sort solely by position in the image. First extract the subtitle narrative events, then map the visible subjects in the image to the events, and arrange the drawing in the semantic order of "scene setup → key person/object → action conflict or change → reaction/result". Then create <图片名>.annotation.json. After creation is complete, immediately open assets/preview.htmlin the default browser, and via the preview console's "Open Folder" loadallin the directory containing the annotation file <名称>.png + <名称>.annotation.json; do not merely provide the file path or ask the user to do it themselves.Stop after the preview console has loaded the directory, and wait for the user to confirm the annotations and preview content.
4. Generate the region preview image. Only after the user confirms the annotations and preview content, use render_annotation_preview.py to produce a numbering/direction check image, verify that the regions match the narrative order, that all regions are within the canvas, and that overlapping subjects are protected with protectedRegions .Stop after completion and wait for the user to confirm the preview image.
5. Adjust and save in the preview console. Only after the user confirms the preview image, adjust in the preview console that is already open and has the corresponding directory loaded: by default (not playing) the full image and region boxes are displayed; the canvas isrectangular proxy: drag the four sides and four corners of a region to change region, and on the right change the name/direction/start(ms)/end(ms)(duration = end − start, read-only) andsubtitles, drag the module listto adjust the order(automatically reorder sequence), and selecting a module automatically highlights the corresponding subtitle; drag the timeline or press play to see the reveal (regions that have not started are not displayed);direction only affects this proxy. When done, click "Save this scene/Save all" to write back to the original .annotation.json(including each region's subtitle, and align sceneDurationMs to the end of the last region + 0.5s).Stop after saving and wait for the user to confirm the final annotations and timing.
6. Render the final video via the command line. Only after the user confirms the final annotations and timing, use render_stream_whiteboard.py to produce a fully cleared MP4 scene by scene, and spot-check three time points: the opening, the middle of any overlapping module, and the ending.Stop when done and wait for the user to confirm the final video.
7. Merge multiple scenes (only applies to multiple scenes). Only after the user confirms all single-scene final videos, use merge_scenes.py to merge them into one in order.Stop when done and wait for the user to confirm the final composite video.

Directory conventions

Create in the user's project:

assets/whiteboard/<项目名>/
  scene-01-<名称>.png
  scene-01-<名称>.annotation.json     # 与 png 同名
  scene-01-<名称>-whiteboard.mp4      # 成片
  scene-01-<名称>-preview.mp4         # 真实片段（预览台生成，低清）

Images and configs must have the same name:foo.png corresponds to foo.annotation.json. The preview console automatically loads the config based on this.

Semantic ordering and pixel-level annotation (must be performed)

1. Reading basis: Before annotating, both the subtitles and the viewed original image must be available. If either is missing, request it first; do not generate annotations.
2. Ordering basis: sequence、startMs and label must reflect the order of events in the subtitles, not merely left-to-right, top-to-bottom, or visual prominence.
3. Coordinate basis: Each module outputs integer pixels in the original image coordinate system x、y、width、height; the origin is the top-left corner, and percentages/ratios/estimated coordinates or omitted dimensions are prohibited.canvas.width/canvas.height must equal the original image pixel dimensions.
4. Module fields: Each element contains sequence、narrativeRole、subtitle、region、reveal、handPath。narrativeRole use Chinese to explain its narrative role in the subtitles;subtitle stores the subtitle text corresponding to that region (from the SRT, for preview console linkage and subsequent use);sequence is consecutive starting from 1.
5. Validation: Before generating the preview, check whether each region is within the canvas, whether it covers the corresponding visible subject, and whether it matches the subtitle event; for overlapping subjects use protectedRegions to protect and then draw the modules.

Timing model (specific to the stream drawing method)

• Total duration of each scene sceneDurationMs comes from that scene's subtitle time span (parse_srt.py 's scenes[].sceneDurationMs）。
• Regions are drawn serially: The stream drawing method is a single pen moving, so regions within the same scene shouldproceed sequentially in time（startMs without overlapping): the next region starts from the previous region's startMs + durationMs(+ optional 100–300ms breathing). If startMs overlap, the renderer still processes them in order, but visually they are no longer concurrent.
• Within the region, ink→color: For each region, the durationMs is split by ink:color = 2:1 into an initial stroke segment and a color-add segment.durationMs is determined by the preview console'sstart/end time(end−start), and can be aligned with the duration of the corresponding subtitle in that region; 150 pixels/second × drawing distance can also be used as an initial estimate.
• Gaze ending: After all regions are drawn, automatically pad to sceneDurationMs, and ensure the ending holds the complete original image for at least 0.5 seconds.
• reveal.direction Under the stream drawing method,does not determine the actual strokes(strokes are automatically generated from the skeleton/grid); it is only for the preview console's rectangular proxy demonstration; it is kept so that the preview console remains usable.

Mask invariants (orchestration layer, must be enforced)

• At time t, a module may only display pixels after its reveal.startMs ≤ t and not exceeding the current drawing progress; any lines/fills/images of a module that has not started must not appear.
• For each region'sallowed mask = rectangle region minus allsubsequent modules' region, then minus this module's reveal.protectedRegions. stream ink placement is restricted to the allowed mask, so subsequent regions will not reveal lines early.
• protectedRegions uses the same original-image integer pixel coordinates as region , for cases where rectangles are too large, subjects overlap, or background lines might leak.
• The renderer has implemented the order of "restrict ink placement to the allowed mask → subsequent regions and protected areas are inherently untouched"; the preview console's rectangular proxy uses the equivalent destination-out subtraction to demonstrate the same orchestration.

Configuration example

{
  "sceneId": "scene-01",
  "canvas": { "width": 1672, "height": 941 },
  "storyBasis": "该幕字幕的事件摘要",
  "sceneDurationMs": 9000,
  "elements": [
    {
      "id": "rockery",
      "label": "假山场景",
      "sequence": 1,
      "narrativeRole": "故事的场景铺垫",
      "subtitle": "猴子山上，一只小猴子坐在假山顶端，手里拿着香蕉。",
      "type": "structure",
      "region": { "x": 20, "y": 120, "width": 540, "height": 780 },
      "reveal": { "direction": "top_to_bottom", "startMs": 300, "durationMs": 2600, "maskPaddingPx": 22, "protectedRegions": [] },
      "handPath": { "start": [290, 130], "end": [290, 890], "easing": "easeInOut" }
    }
  ]
}

direction / handPath Only for the preview console's rectangular proxy; final strokes are automatically generated by stream and require no fine-tuning.

Using scripts

All rendering scripts are run with the skill's built-in .venv interpreter (dependency isolation).

1. Prepare environment(on first use or when dependencies are missing):

   python scripts/prepare_env.py --check   # 探测；成功末行输出 ENV_PY=<路径>，捕获备用
   python scripts/prepare_env.py           # 缺则建 .venv 并装 opencv-python/numpy/av
   

2. Parse subtitles + suggest shots：

   python scripts/parse_srt.py <字幕.srt> --target-sec 30 --min-sec 25 --max-sec 35
   

3. Region numbering preview image：

   python scripts/render_annotation_preview.py <图片> <标注> <预览图输出>
   

4. Preview console (no server needed): Open directly in Chrome / Edge assets/preview.html, click "Open Folder" to select the directory → load all images + same-name annotations → drag to edit → "Save" writes back to the original files. Writing back requires the File System Access API (Chrome/Edge); other browsers fall back to downloading and manually overwriting. Rendering still goes through the command line (step 5 below).
5. Render a single shot：

   <ENV_PY> scripts/render_stream_whiteboard.py <图片> <标注> <输出mp4> assets/drawing-hand.png \
       [--ink-path grid|skeleton] [--color-fill contour-wipe|brush] [--total-ms <毫秒>]
   

   --total-ms When omitted, use the annotation's sceneDurationMs. The last line outputs OUTPUT=<路径>。
6. Merge multiple shots：

   <ENV_PY> scripts/merge_scenes.py --inputs 幕1.mp4 幕2.mp4 幕3.mp4 --output final.mp4
   

Quality check

Confirm before/after rendering:

• The first frame is a clean warm beige old-paper background, with no lines revealed early.
• Have read the corresponding subtitles and actually viewed the original image;canvas matches the original image pixel dimensions, and all region are integer pixel coordinates within the canvas.
• sequence、startMs is consistent with the subtitle event order; the preview image numbers/labels/regions come from the same annotation JSON.
• Check at three time points—the opening, the middle of any overlapping module, and after all modules are complete: undrawn modules are all invisible, overlapping protected areas do not leak, and the final frame shows the complete original image.
• The pen tip stays close to the advancing strokes; clear line-art illustrations can use --ink-path skeleton to make the strokes fit better.
• After all modules end, hold the complete original image for at least 0.5 seconds.
• After merging multiple scenes, the order and duration match the subtitle storyboard.

If you need to modify the effect, first adjust the annotations (region/order/timing) in the preview console (assets/preview.html) and save, then render from the command line; do not repeatedly produce videos without basis.