SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssketch-story-remotion

sketch-story-remotion

Design & creation

View repository

Updated 10-15 23:59

About this skill

Sketch Story Remotion (Beginner Learns AI hand-drawn style)

Turn the series story script (markdown) into a Remotion video with a consistent hand-drawn style. All style assets are code-generated (no image assets); just reuse templates/remotion-project/src/ the four style files in, and any new episode made by any model will be consistent with Episode 1.

Canonical example: the finished project for Episode 1 is in <VIDEO_WORKSPACE>/xiaobai-ai-ep01/, when you run into layout/pacing issues you are unsure about, look at its src/scenes.tsx。

New episode workflow

  1. Read the latest story script(usually under <NOTES_VAULT>/创作/小白学AI第一季/ ). Use the latest markdown specified by the user as the source of truth; do not carry over an old outline.
  2. Make a story checklist: first list the key beats in the original text as a beat checklist, including character actions, causal turning points, props, jokes, punchlines, and the ending hook. Use this table during later rewrites to prevent the story from being compressed into an outline.
  3. Break into scenes: break the story into 812 scenes, with one visual idea per scene (see "Scene grammar" below). If the original text is long, expand to 1216 scenes rather than deleting bridging beats.
  4. Scaffold： 
    mkdir "<VIDEO_WORKSPACE>/xiaobai-ai-epNN"
cp -R "./skills/sketch-story-remotion/templates/remotion-project/" \
      "<VIDEO_WORKSPACE>/xiaobai-ai-epNN/"
cd "<VIDEO_WORKSPACE>/xiaobai-ai-epNN" && npm install
    ⚠️ Use native npm install, do not use rtk npm install(rtk will translate it into npm run install and cause an error).
  5. Write an animated draft: only change the SCENES table in src/scenes.tsx and src/Root.tsx .Do not change theme.ts / sketch.tsx / characters.tsx / ui.tsx(style lock file; if you really need to add new generic elements, add components instead of changing the visual parameters of existing components).
  6. Generate voiceover and subtitles: generate TTS by scene and character, producing per-scene audio, the full-episode SRT, and TTS metadata. See "Multi-voice voiceover and subtitle workflow" below.
  7. Write back the audio-video synced version: use the real TTS durations to change the frames for each scene in Root.tsx , then adjust the appearance frames of in-scene text, frames, arrows, and expressions according to the subtitles/metadata. Do not stop at the estimated-timing version without voiceover.
  8. Verification loop(required):
    • npx tsc --noEmit
    • Render a still frame near the end of each scene and check them one by one:npx remotion still Episode --frame=<场景末帧-30> out/check-N.png, use image reading to check for text overflow, element overlap, and occlusion.
    • After the video is complete, hand the final rendered screenshot of each scene to a Subagent for independent visual review; focus on floating bubbles/blocking people, tails not pointing at the speaker, sweat drops/anger lines too far from the head, text overflow, and accidental occlusion. After revising according to the review results, re-render screenshots of the affected scenes and check again.
    • Spot-check the appearance frames of subtitles/keywords: they must not appear before the line is spoken, and must not still be missing after the line has finished.
    • For process words, list words, sticky-note words, and punchline keywords, dokeyword-level sync spot checks: first list 画面文字 -> 对应口播短语 -> 预计源时间, render at least two frames for each keyword: "0.5~1 second before appearance" and "0.5 second after appearance"; the earlier frame must not reveal that word, and the later frame may only reveal words that have already been spoken.
    • After all pass, npx remotion render Episode out/epNN-demo.mp4。

Story fidelity

  • The source is a story, not an outline. When converting to a video script, similar sentences can be merged, but key story beats must not be deleted in a way that breaks causality.
  • Preserve "action continuity" and "prop continuity." For example, actions like a character pulling out a sticky note, writing down keywords, pointing at the computer, and sending an email are the bridges that let the audience understand the next line.
  • Narration can be shortened, but the emotional progression of the source must be preserved: source of pressure → AI steps in → misunderstanding/worry → Lao Wu corrects course → definition → responsibility boundary → human work → punchline/next episode.
  • If cuts are unavoidable, first list the beats that will be removed, and confirm that they are not connecting segments necessary for understanding what follows.
  • Before TTS, do one comparison pass: check the beat checklist scene by scene, and confirm that no important sentences, actions, turns, or punchlines have been missed.

Style DNA (immutable)

Item Value
Canvas 1920×1080 @ 30fps, off-white paper background #fbfaf5
Ink color #161616(INK), line width 3~4
Accent colors Red #d0402b(titles/warnings/circles and crosses), blue #3a6bc9(annotations/key points of character dialogue), orange #e8862e(arrows/flows), green #3f8a3a(checkmarks/positive)
Font macOS handwriting font stack:'HanziPen SC','Hannotate SC','Wawati SC','Kaiti SC',cursive(in theme.ts 's FONT; rendering machines must have the font installed)
Lines All wireframes are generated with sketch.tsx 's wobbly series (seeded jitter),forbiddento draw graphic elements with straight SVG rect/line or CSS border
Animation Elements are "drawn out": strokeDash progressive stroke + delayed fill; text fades in + slight upward movement;nomodern motion effects such as slide-in, bounce, or rotation
Assets All drawn in code, no bitmap/icon libraries/emoji

Character profiles (unchanged across the series)

Character Component Parameter conventions
Xiaobai (newbie protagonist) StickMan No glasses, size 170~190, commonly used moods: worried / shocked / happy
Lao Wu (veteran employee mentor) StickMan glasses must be on, size one step larger than Xiaobai (200~210), often paired beside with ThermosCup(thermos, placed at hand side x±55, y about 50 above the ground)
Client/passerby StickMan hair={false} bald to distinguish them, when angry armsUp + mood="shocked" + three short red anger lines above the head
AI Xiaozhi (standalone form) CuteAi Cute chibi little computer: antenna + blush + big eyes, use it for standing scenes
AI Xiaozhi (desktop form) AiScreen Monitor on the desk;faceOn face on (same CuteFace as CuteAi), when displaying emails or other content do not show the face, use children to draw the content

Humans are always StickMan, AI is always CuteAi/AiScreen, do not mix them or invent new forms.

Series voice timbres (fixed)

Character/Track Purpose MiniMax voice_id
Narration Narration, punchlines, paraphrased lines for extras such as clients/colleagues moss_audio_2ecaeaac-5e5a-11f1-99fb-96e792fde6a1
Xiaobai Xiaobai's direct dialogue Chinese_playful_streamer_nv1
Old Wu Old Wu's direct dialogue Chinese_casual_instructor_nv1

Narration uses the fixed IDs in the table above; do not use the old temporary narration IDs.

Series default pacing and BGM

  • The final delivery version defaults to processing at 1.2x speed: voiceover, subtitles, scene frames, and visual animation must all land on the same 1.2x timeline. If TTS/post-processing supports speech rate, generate or process the audio at 1.2x; otherwise speed up the audio in Remotion/ffmpeg while synchronously compressing the subtitle timing and visual timeline—you cannot speed up only one of them.
  • The series default BGM uses the audio track of <BGM_FILE> , looping to fill the entire video when the duration is insufficient.
  • BGM is remixed from the no-BGM final cut, defaulting to volume=-34dB, with highpass=f=80、equalizer=f=500:t=q:w=1.5:g=-3, vocal sidechain ducking, and fade in/out at the start and end; the video stream uses copy to avoid re-compressing the image quality.

Scene grammar (layout conventions)

Each scene is an independent component, with internal useCurrentFrame() starting from 0. Common layouts:

  • Opening card: gray series name (y300) → large black title (y400, 78px) → red suspense subtitle (y530, 54px) + hand-drawn underline → protagonist reveal (y910).
  • Dialogue scene：RedTitle top center (y70) → DeskLine ground line (y800~880) → characters stand on the ground line → Bubble speech bubble (left character tail="left" placed top-left, right character tail="right" placed top-right) → Narration bottom narration (y960).
  • Diagram scene: key points are arranged in a vertical column or a horizontal three-cell row using SketchRect+HandText , processes are connected using SketchArrow(orange), emphasis uses SketchHighlightCircle(red circle) or SketchUnderline, and negation uses SketchCross(red X).
  • Comparison scene: a gray vertical divider line in the middle, with one RedTitleon each side (black on the left, red on the right); after listing the items on the left side, draw a large red X over the whole side.
  • Punchline card: pure centered text, one black line + two red lines (62px) + a long underline, with two small figures standing below for contrast.

Rules:

  • Element appearance order = narration order, with one beat every 4070 frames; the frame is drawn first (1825 frames), and the text inside the frame enters 8~10 frames later.
  • Text should not appear all at once on the whole screen like a PPT. Lists, sticky notes, process words, speech bubbles, and punchlines must all appear one by one in the order of the voiceover.
  • The keyword/speech bubble corresponding to a line of dialogue should appear near the start of that line and complete its main animation before the line finishes; do not reveal later punchlines in advance.
  • Speech bubbles must form a clear relationship with the speaker: the tail should point as close as possible to the character's head/body area, and the bubble body must not cover the character; when the character is not in the frame, do not force in a dialogue bubble—first add the character or change it to narration/annotation.
  • Emotion symbols such as sweat drops, anger lines, and surprise lines must be attached near the character's head; do not hard-code positions too far from the head, which looks like they are floating in the air.
  • Reasonable overlap may be retained, for example intentionally letting one element cover another to express pressure, occlusion, or emphasize a relationship; however, text accidentally pressing on a character's face/body, a speech bubble blocking a character, or an annotation unintentionally passing through a character all count as visual errors and must be fixed by adjusting coordinates or leaving a safe zone for the character.
  • Text layers (HandText/Bubble/Narration) are in <Svg> outside; graphics are all in <Svg> .
  • All progress is uniformly prog(frame, start, duration)(with built-in clamp); seeds use a scene prefix to guarantee uniqueness (e.g. s3box1）。
  • One scene, one concept—don't cram in a second one.

Pacing (no-voiceover version)

30fps. Estimate scene duration by "reading the on-screen text slowly once, then adding 12 seconds": pure title card 130 frames; dialogue scene 250350 frames; diagram/comparison scene 350450 frames; closing summary ~500 frames. Between scenes, Root's FadeScene automatically does a 12-frame fade in/out; do not introduce new elements in the last 30 frames of a scene.

Multi-voice voiceover and subtitle workflow

Generate voiceover in segments by scene (for TTS see tts-skill / MiniMax), and you must revise the animation after generating the audio:

  1. Create a TTS script (recommended location: scripts/generate_epNN_tts.py), with the data structure split by scene → utterances, and each utterance containing at least role、text。
  2. Role mapping uses "series voiceover timbres (fixed)": the narrator handles narration, punchlines, and relaying minor-character lines; Xiaobai reads only Xiaobai's lines; Lao Wu reads only Lao Wu's lines.
  3. Generate each utterance's audio separately first, then concatenate into one audio file per scene, placed in public/audio/scenes/scene-01.mp3 …; at the same time keep the segmented audio in work/audio/generated/。
  4. Segmented audio filenames must include a text hash; do not cache using only line numbers, to avoid mistakenly reusing old audio after changing lines.
  5. Generate TTS metadata, recording at least each utterance's scene, role, text, start/end time, and start/end frame; generate SRT from the metadata, write it to work/captions/epNN.srt and copy it to public/captions/epNN.srt。
  6. Use ffprobe or metadata to get each scene audio segment's duration, first compute the source timeline sourceFrames = ceil(时长秒 × 30) + 15, and compress the final delivery timeline by 1.2x to playbackFrames = ceil(sourceFrames / 1.2)。
  7. Add audio in the Sequence; if the audio file is not already the sped-up version, use <Audio src={staticFile('audio/scenes/scene-01.mp3')} playbackRate={1.2} />, and drive the visuals with the same 1.2x mapping.
  8. Write back the in-scene animation anchors according to the metadata: when a line is spoken, the corresponding character, sticky note, box, arrow, or red circle starts being drawn from around that point.
  9. Before rendering, keep remotion.config.ts at default (audio will be automatically mixed into the mp4).
  10. After exporting the no-BGM version, mix in the generic BGM according to the "series default pacing and BGM" to produce the final BGM version.

Audio-visual sync acceptance

After generating the voiceover, you must check whether the audio, subtitles, and animation are aligned:

  • The total video duration should be close to the total audio duration; the audio track must not be noticeably longer or shorter than the video track.
  • No new text appears in the last 30 frames of each scene, to avoid cutting scenes right after it appears.
  • Spot-check the start frames of 24 utterances per scene: subtitles/keywords may be set up 36 frames early, but must not reveal all the later text a dozen or so frames early.
  • Process words within a long utterance cannot be aligned only by the start time of the whole sentence; they must be checked separately by phrase position. For example, for progressive words like "first recognize strokes, then recognize characters, then recognize meaning," you cannot reveal the whole group "strokes/characters/meaning" before saying "first recognize strokes."
  • Spot-check the end of each scene: before the last line is finished, the corresponding text and graphics on screen should already have appeared; you cannot have "the line is finished, but the text hasn't arrived yet."
  • On-screen text appears progressively along with the voiceover, especially for three-word sticky notes, flowcharts, comparison lists, and quote cards; a full page must not be laid out all at once.
  • Render near-final still frames and necessary mid-section still frames, and check for text overflow, speech bubbles pressing on heads, overlapping elements, and subtitles being obscured.
  • The final screenshot of each scene must go through a Subagent visual review; during the review, distinguish between "occlusion with narrative intent" and goofs that mistakenly block characters or press on text. If problems are found, fix the visuals first, then take new screenshots to confirm.
  • The final BGM version must be remixed from the no-BGM final cut; confirm that the BGM has been looped to fill the entire track, that its volume does not overpower the vocals, and that there are fade-ins and fade-outs at the beginning and end.
  • After the final export, use ffprobe to check the video/audio duration of the mp4, and clean up Remotion/headless browser test processes.

Common pitfalls

  • rtk npm install will fail, use native npm install。
  • Text overflowing the hand-drawn box: prioritize reducing text/line breaks (HandText supports \n), and secondarily enlarge the box; do not shrink the font size below 26.
  • HandText defaults to a center anchor (x is the center);align="left" when x is the left edge.
  • Bubble 's x/y is the top-left corner, the width w is fixed, and the height expands with the content; be careful not to press on the character's head.
  • When checking extracted frames, focus on whether the speech bubbles are "floating on top": the tail not pointing at the person, the bubble covering the person, and the speaking character not being present are all layout problems that need to be fixed.
  • Repeated seeds will make the two lines look exactly the same, giving it away.
  • Before long renders/batch still frames, first npx tsc --noEmit; Remotion's runtime error messages are less intuitive than tsc's.
  • On a new machine, the first render will automatically download Chrome Headless Shell (about 90MB), which is normal.
  • Do not treat the estimated frames from the earlier no-voiceover version as the final pacing; after real TTS is generated, a version must be written back.
  • Do not delete story-connecting segments just to pursue 8~12 scenes; the number of scenes can increase, and coherence takes priority.