fframes-video

Making videos with fframes

fframes is a Rust library that renders video from code. A video is a Rust struct with a render_frame(frame) -> Svgr method: for every frame it returns an SVG tree, built with the svgr! macro (SVG markup with {rust expressions}). Scenes split the timeline, timeline! and springs animate values, an audio map places sound, and ffmpeg encodes the result.

Each project gets a command line (from fframes::cli). Use it for everything: render a video, open the real-time preview window, and look at frames, contact sheets and loudness numbers as files you can read. Read references/design.md before designing, references/api.md while writing code and references/audio.md for sound.

When matching a supplied video, first follow the reference-analysis guidance in references/design.md. Its cuts, pacing and motion take precedence over the design defaults below; fast edits and intentional black frames may be part of the requested result.

1. Install

Rust from https://rustup.rs, then the system libraries ffmpeg is built with:

# macOS
brew install pkg-config ffmpeg x264 x265 opus nasm ninja
# Debian / Ubuntu
sudo apt-get install -y yasm nasm ffmpeg libx264-dev libx265-dev libopus-dev libclang-dev clang ninja-build libvpx-dev libasound2-dev
# Arch
sudo pacman -S ninja yasm nasm ffmpeg x264 x265 opus clang

Windows links a prebuilt FFmpeg 9 shared build instead (set FFMPEG_DIR to the unzipped ffmpeg-n9.0-latest-win64-gpl-shared build from BtbN/FFmpeg-Builds, add its bin to PATH, install LLVM with winget install LLVM.LLVM and set LIBCLANG_PATH). Details are in the fframes README.

2. Create a project

curl -fsSL https://raw.githubusercontent.com/dmtrKovalenko/fframes/main/scripts/new-video.sh | bash -s -- my-video --yes
# or install the generator once and use it directly:
cargo install --locked cargo-fframes
cargo fframes new my-video --format landscape --fps 30 --yes

Options: --template single-scene|multi-scene, --format landscape|portrait|square|uhd, --fps, --title, --backend, --dir. The project depends on the fframes release that matches the installed cargo-fframes; --git uses the repository's main branch instead. Always pass --yes so nothing waits for input.

Templates:

• single-scene (default): one scene. It adapts to every format; start here for single-shot clips, title cards and portrait video.
• multi-scene: two scenes. Start here for anything with several scenes; it only accepts landscape and uhd.

The template content is placeholder code that shows the API. Replace its layout, colors, fonts and decorations with a design made for the video the user asked for.

Use the Skia GPU backend (the default). cargo fframes new picks Skia on Metal (macOS) or Vulkan (Linux, Windows). It renders about 10x faster than the CPU backend and gives you the real-time preview window. On macOS and Linux the first build downloads prebuilt Skia and ffmpeg and only compiles the Rust dependencies (about a minute; other targets, or metal and vulkan together, compile Skia from source for ~20 minutes). Start it right away and write the video while it runs:

cd my-video && cargo build --release        # run in the background, the first build is the slow one

--backend cpu needs no Skia but has no preview window; pick it only when there is no GPU.

The project renders as generated:

my-video/
  src/lib.rs        # the video (from the template)
  src/main.rs       # the command line
  media/            # fonts, images and audio compiled into the binary (one starter font)
  README.md         # the commands below

Every command below is cargo run --release -- <command>. Define a short shell function (a variable like R="cargo run --release --" does not split into words in zsh, the macOS shell):

R() { cargo run --release -- "$@"; }

3. The loop

Do not write tests at all when using this skill. Do not create unit, integration, snapshot or visual-regression tests, test modules, or test harnesses. Validate the video with inspect, rendered frames, contact sheets, audio checks and the native preview.

For the GPU backend, offer the native preview before a full render. Do not render the whole video as a routine review step. Once the changes are ready to review:

1. Show a complete command in a copyable shell code block, using the actual project path and package name; do not require the user to have defined the $R helper. For example: cd /absolute/path/to/fframes && cargo run --release -p my-video -- preview 0s.
2. Explicitly offer to run that preview for the user. If they have already authorized opening or refreshing the preview, show the command and continue within that authorization.

After a requested change, set the command's start time to a specific second at or just before the changed section, with enough lead-in to see its entrance or transition. For example, preview 8.5s lets the user review a change around 9 seconds without replaying the beginning. State which section they will see. Render the full video when the user requests the final export; do not require another confirmation if that request is already explicit.

After every change:

1. $R timeline lists the scenes with their frame and second ranges, and every audio track. Check structure and pacing here before looking at pixels.
2. $R inspect checks a frame every 0.25 s plus the first and last frame of every scene. It reports missing images, fonts or glyphs, text cut off by the edge of the canvas, invalid SVG (zero-sized rectangles, bad radii), broken transforms and panics, each with its time and scene. It exits with code 2 on errors. Fix everything it reports. A warning that only appears on a scene's first frames is usually an entrance; check it with a strip.
3. $R strip <scene or range> -n 12 writes a contact sheet (strip.png): evenly spaced frames in one labelled image. Open it with your image tool. It is the fastest way to judge layout, rhythm and motion.
4. $R frame Intro@end,Outro@50% writes full-size PNGs into frames/ for detail checks (typography, alignment, contrast) and prints problems found in those frames.
5. $R onion "Intro@0..Intro@1s" -n 6 blends frames into one image (onion.png) to show the path and spacing of a movement: easing, overshoot, stagger.
6. $R preview Intro opens a real-time window with sound for the user to watch (space play/pause, h/l seek a second, j/k step a frame, q quit). It blocks until closed, so start it in the background when running it for the user. Show the command and offer to launch it as described above; after edits, use a specific second near the change.

When an encoded sample is needed, $R render Intro --draft encodes one scene at half resolution. $R render writes the full video for a requested final export.

Rules:

• Look at the PNGs before saying anything looks good.
• Prefer strip to many frame calls; add --scale 0.5 when composition is all you need.
• Add --json when parsing output: the result goes to stdout, progress to stderr.
• Keep --release: debug builds render many times slower.

Addressing time

120 (frame), 3.2s, 500ms, 1:05.5, 50%, start, end; scenes by struct name, Intro (also matches IntroScene, case-insensitive), #3 (scene index), Intro[1] (second scene of that type); inside a scene Intro@1.2s, Intro@12, Intro@50%, Intro@end. Ranges: a..b (end exclusive), a.., ..b, all, or a scene name for the whole scene. Separate several times with commas.

All commands

command	use it to
timeline	see scenes, durations, audio tracks and their mix settings
inspect [RANGE] [--every 0.1s | --all-frames] [--fail-on warning]	find problems without rendering pixels
strip [RANGE] -n N [--columns 4] [--width 480]	review flow and motion in one image
frame TIMES [-o dir] [--svg]	full-size PNGs (and the laid-out SVG) of chosen frames
onion RANGE -n N	see a movement's trajectory and easing
svg TIME	read a frame's final SVG as text (positions, text, colors)
preview [TIME] [--paused] [--mute]	real-time window with sound, for the user
render [RANGE] [-o out.mp4] [--draft]	encode the video or a part of it (audio cut to match)
snapshot TIMES [--update]	compare frames with approved PNGs, .diff.png marks changes
audio analyze [RANGE] [--waveform w.png]	loudness (LUFS), true peak, clipping, silence, per scene
audio at TIMES	which sounds play at a moment, where in their file and how loud
audio render [RANGE] -o a.wav	the mix as a WAV file

Browser editor

fframes also has a web editor with a timeline and scrubbing, useful when a person wants to tweak a video interactively. It runs the video compiled to WebAssembly and needs Node.js plus wasm-pack; the setup is the editor/ folder of the hello-world example in the fframes repository (https://github.com/dmtrKovalenko/fframes/tree/main/examples/hello-world). Mention it when the user asks for a GUI editor. For watching use preview, and for your own review use the CLI.

4. Writing the video

impl Video for MyVideo<'_> {
    const FPS: usize = 30; const WIDTH: usize = 1920; const HEIGHT: usize = 1080;
    fn duration(&self) -> Duration<'_> { Duration::Auto }            // sum of the scenes
    fn audio(&self) -> AudioMap<'_> { AudioMap::none() }              // see references/audio.md
    fn define_scenes(&self) -> Scenes<'_> { Scenes::from(vec![&self.intro as &dyn Scene, &self.main]) }
    fn render_frame<'a>(&'a self, frame: Frame, ctx: &FFramesContext<'a, '_>) -> Svgr<'a> {
        fframes::svgr!(<svg xmlns="http://www.w3.org/2000/svg" width={Self::WIDTH} height={Self::HEIGHT}>
            <rect width={Self::WIDTH} height={Self::HEIGHT} fill={BACKGROUND} />
            {ctx.render_scenes(&frame)}
        </svg>)
    }
}

• One scene per idea, 2-6 s each. Inside a scene frame.seconds() counts from the scene start.
• Default to frame.animate(fframes::timeline!(...)) directly in svgr! attributes for motion with fixed values. svgr! lifts these animations into cached statics. Define easings inline in the timeline, including Easing::Spring { mass: 1.0, stiffness: 180.0, damping: 20.0 }; named easing constants are optional. Leave the end time off a spring so it settles on its own.
• For dynamic values or calls outside that optimized pattern (including animate_loop), construct timelines once on self and sample them by reference. Use cached AnimationRuntime values for starts that change after construction or custom/subframe clocks. Camera parameters and shader uniforms can use these APIs too; do not duplicate springs or tween interpolation. See references/design.md and references/api.md.
• Markup without {} is cached across frames. Keep decoration literal and put animated values on a wrapping <g transform={..} opacity={..}>.
• render_frame runs for every frame on several threads: no panics, file reads or heavy work in it. Prepare data in the constructor or a OnceLock. Media lookups return Option; fall back to Svgr::empty() instead of expect.
• Put every font file in media/ and refer to it by family name, with a numeric font-weight.
• Measure text rather than guess: frame.text_width, frame.text_fit(.., TextOverflow::Ellipsis), frame.text_break_lines for paragraphs. inspect catches text leaving the canvas, not text leaving its own box, so check boxes in a frame PNG.
• GPU shaders (SkSL or Shadertoy GLSL) run on the Skia backend through fframes::Shader; they take images and synced video frames as input (green-screen keying, color grading). See references/api.md.

5. Making it look good

The short version of references/design.md:

• One idea per scene, large type, margins of 8-10% of the width, two or three colors plus neutrals and one accent for emphasis.
• Elements enter with a spring or ease-out over 300-600 ms and leave faster, with ease-in over 200-300 ms. Related items stagger by 60-120 ms. Give the viewer 1-2 s to read after the motion settles, and never move everything at once.
• Cross-fade scenes (fn overlap(&self) -> Overlap { Overlap::Previous(0.4) }; the overlap is added to the scene's duration(), see references/api.md) or carry an element across the cut. Keep a little motion during holds so they do not look frozen.
• At 1920x1080: titles 96-140 px, body 44-60 px, at most about 8 words per line and 3 lines per card. Portrait 1080x1920 is watched on a phone: same pixel sizes or larger, content inside the middle 80% because platform UI covers the top and bottom.
• Check every scene against the checklist in references/design.md with a strip.

6. Sound

Put audio files in media/ (compiled in, mono) or load a folder at runtime with MediaDirectory for stereo, then place them:

AudioMap::from([
    AudioTrack::new("music.mp3", Second(0.)..Eof).gain_db(-18.).fade_in(1.).fade_out(2.).duck_under_voice(),
    AudioTrack::new("vo.wav", Second(0.6)..Eof).voice(),
    AudioTrack::new("whoosh.wav", Second(3.1)..Eof).gain_db(-8.),
])

Time sound effects from the same constants that drive the animation. Check levels with $R audio analyze --waveform w.png (about -14 LUFS integrated for web video, true peak below -1 dBTP, no unintended silence) and $R audio at 3.1s for what plays at an event. Then let the user listen in $R preview.

7. Finish

1. $R inspect --fail-on warning passes, or the remaining warnings are understood entrances.
2. A strip of every scene looks right and key frames are checked at full size.
3. $R audio analyze shows sensible levels.
4. For GPU work awaiting review, show the preview command and offer to run it at the relevant second. When the user requests the final export, run $R render -o out.mp4, then confirm size, frame count and audio with ffprobe -v error -show_entries stream=codec_type,width,height,nb_frames,duration out.mp4.

For preview review, provide the concrete preview command, its start time and the offer to launch it. After final export, provide the output path and duration.

Troubleshooting

• Build fails in ffmpeg-sys-fframes: a system library from step 1 is missing (nasm, pkg-config, the codec -dev packages).
• Build fails in the Skia bindings (skia-bindings) with bindgen or libclang errors (Skia is only compiled from source when no prebuilt matches, e.g. metal and vulkan together): point LIBCLANG_PATH at a working libclang (on macOS Xcode's: export LIBCLANG_PATH=$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/lib).
• Skia bindings fail to compile on macOS 27 / Xcode 27 with cannot find type `_Traits` in std___hash_table___node_allocator: only happens when Skia is compiled from source (the prebuilt binaries ship their bindings). skia-bindings 0.153.3 lacks the bindgen rule (rust-skia #1335); use one GPU backend so the prebuilt download matches, or patch a copy of skia-bindings-0.153.3 (add "std::__hash_table.*", after "std::__tree.*", in OPAQUE_TYPES in build_support/skia_bindgen.rs) through [patch.crates-io].
• can't find crate for `fframes_media_dir_macro` on macOS 27 while the file exists: the proc macro was linked by an older Rust whose output the macOS 27 loader rejects (a "LINKEDIT string pool" error). Update Rust (rustup update; 1.98 works) and rebuild.
• Text renders in the wrong font or not at all: inspect shows "No match for ... font-family"; add the font file to media/ and use its exact family name.