framefields-render

framefields-render

Setup

Import everything from the framefields npm package. If the project doesn't have framefields set up yet, follow the framefields skill first (install, project scaffold, first render).

Overview

framefields renders headlessly in Node.js on top of native WebGPU (Dawn) and mediabunny WebCodecs hardware encoders. It does not run a browser, Puppeteer, or Chromium.

Because video rendering is resource-intensive and native GPU pipelines run asynchronously in worker pools, framefields workflows enforce a strict verification ladder: inspect inexpensive intermediate artifacts first before committing to a full video render.

The 5-Step Verification Ladder

Always proceed through the ladder in order. Catching a layout or timing bug at step 1 or 2 takes seconds; catching it after a 30-second 4K video render wastes minutes.

Step 1: Static Typecheck & Schema Validation

Before touching the GPU, verify TypeScript types and parse the composition against the strict runtime schema:

import { CompositorProgramSchema } from "framefields";

// 1. Compile spec
const spec = comp.toSpec();

// 2. Validate against strict schema
const parsed = CompositorProgramSchema.safeParse(spec);
if (!parsed.success) {
  console.error("Schema validation failed:", parsed.error.format());
  process.exit(1);
}

// 3. Lint: things that render without error but not as intended
for (const d of comp.lint()) console.log(`${d.severity} ${d.code}: ${d.message}`);

Typecheck the project:

npx tsc --noEmit -p .

Step 2: Single Frame Inspection (`renderFrame`)

Render isolated keyframe points (e.g. entry, peak action, exit) directly to PNG files on disk to inspect alignment, typography, colors, and opacity:

import fs from "node:fs/promises";
import path from "node:path";
import { HeadlessMediaRenderer } from "framefields";

const renderer = new HeadlessMediaRenderer();

// Render frame 45 (1.5s at 30fps)
const pngBuffer = await comp.renderFrame({
  frame: 45,
  renderer,
});

await fs.writeFile("output/frame-45.png", pngBuffer);

Step 3: Motion & Layout Stability Contact Sheets (`renderFrameGrid`)

To verify easing curves, cut points, kinetic sweeps, or text wrapping without encoding a full video, generate a sequential frame grid image:

import fs from "node:fs/promises";
import { renderFrameGrid } from "framefields";

// Samples frames across the intro sweep (frames 0 to 60)
const gridPng = await comp.renderFrameGrid({
  frames: [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60],
  columns: 4,
  cellWidth: 480, // Downscaled thumbnail size for rapid visual scanning
  showLabels: true, // frame number and timestamp under each cell
});

await fs.writeFile("output/intro-grid.png", gridPng);

Contact sheets immediately reveal:

Motion hitches and non-smooth easing deceleration.
Text clipping or wrapping jumps across keyframe boundaries.
Bad layer z-indexing or missing cut transitions.

Step 4: Programmatic Conformance Probes in Vitest

Write automated headless vitest tests that assert pixel colors, transparency, or layout bounding boxes:

import { describe, it, expect } from "vitest";
import { HeadlessMediaRenderer } from "framefields";
import { buildFilm } from "../film.js";

describe("Hero Scene Conformance", () => {
  it("renders non-black title text on frame 30", async () => {
    const comp = await buildFilm();
    const renderer = new HeadlessMediaRenderer();
    const png = await comp.renderFrame({ frame: 30, renderer });

    expect(png.length).toBeGreaterThan(1000);
    // Use skia-canvas or pixel probes to assert color at target coordinates
  });
});

Step 4.5: Show the User Before Exporting (`preview`)

When a person is in the loop, let them watch the composition before you spend time on the final render. startPreview serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code, the project's files, and the soundtrack mixed by the export engine. Point it at the module that builds the composition:

import { startPreview } from "framefields";

const session = await startPreview(
  { entry: new URL("./film.ts", import.meta.url), export: "buildFilm" },
  { title: "Launch film" },
);
console.log(`Preview at ${session.url}`);
await session.closed; // keep the process alive while the user watches

The export is a function returning the Composition (it may be async), or a Composition itself. The browser calls it, so build-time Node code in it must stay simple: node:path, fileURLToPath(import.meta.url) and synchronous fs reads (JSON, text) work through shims; writes are skipped. Assets referenced by absolute path load from the server, limited to the project that holds the entry (its nearest .git or package.json); pass root with one or more paths for assets kept outside it (e.g. a shared fonts folder).
Run it in the background, and read its output. Start the script as a background command with the longest timeout your harness allows, tell the user the URL, and ask for feedback in the chat. The command's output (stderr, prefixed [framefields preview]) says when the page opened, reports each distinct error from the page once (a build error, a frame that threw, a font or renderer that failed to load, no WebGPU), and says why it stopped. Check it after the page opens instead of assuming the film drew. log redirects or silences these lines, and session.closed resolves with the reason: "idle", "replaced" or "closed".
One tab for the whole loop. The port is fixed per project directory. Running the preview again (after you've made changes) takes over from the one already running, and a tab left open reconnects and reloads into the new version by itself, at the moment it was showing. Don't ask the user to reopen anything. When the user closes the tab, the preview stops by itself about 5 s later (a reload doesn't stop it; idleCloseMs changes the delay). If the preview was stopped (tab closed, time limit, or you killed it), just start it again.
Notes pinned on the picture are the user's precise feedback. Paused, the user clicks the frame (or drags an area) and writes what should change; the note keeps the frame, the spot as fractions of the frame (x, y, and w, h for an area) and the text. Each note prints in the output (note 3 at 0:12.50 (frame 750), spot 42%,31%: …) with the path of note-<id>.jpg, the frame with the spot marked, and all of them are in .framefields/preview-notes/notes.json (.framefields ignores itself in git; notesDir moves it, false turns notes off). To act on them: read notes.json, open each open note's picture to see exactly what it points at, find the layer in the code from the moment and the spot, make the change, and run the preview again. The tab comes back with the notes, so the user can check them and tick them off. You can also set "done": true in notes.json yourself. A chat that can't read files gets the same list from the Copy button in the Notes panel.
Point at a moment. #t=12.5 in the URL opens the player at 12.5 s (#frame=750 by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.
Drive it from a script, not with clicks. If you can run JavaScript in the page (an agent browser), use window.framefieldsPreview: await seek(seconds) or await seekFrame(n) resolve once that frame is drawn (take the screenshot then), play() and pause() control playback, notes() lists the notes, and state() returns { state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }. <html data-preview="ready"> marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.
Show it where the user is. startPreview only serves the page and returns the URL. If your app has a built-in browser (Claude Code's browser pane, Codex's), open the URL there. Otherwise give the user the link, or pass open: true to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).
Playback runs in real time, on any device. Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Next to the timecode the page shows the frame rate playback reaches, apart from the film's own fps; state() reports it as playbackFps, and renderMs is how long the last frame took to draw. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.
Audio is the same mix renderVideo muxes (audio: false skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the picture skips ahead to catch up with the sound.

Step 5: Full Video Export (`renderVideo`)

Once static frames and motion grids are verified, encode the final MP4/WebM video:

import path from "node:path";

const result = await comp.renderVideo({
  outputPath: "output/final-video.mp4",
  quality: "high", // "draft" | "medium" | "high" | "lossless"
  codec: "avc",    // "avc" (H.264) | "hevc" (H.265) | "vp9" | "av1"
  audioCodec: "aac",
  concurrency: 4,  // Parallel frame encoding threads
});

// Always invoke cleanup to release temporary frame cache and file handles
await result.cleanup?.();

Pass qa: true to have the engine check the output while it renders (see Built-in Video QA below) and return the findings as result.qa.

Production Render Script Pattern

Production projects structure their src/render.ts to support both quick frame probes and full exports from the CLI:

// src/render.ts
import fs from "node:fs/promises";
import path from "node:path";
import { HeadlessMediaRenderer, startPreview } from "framefields";
import { buildFilm } from "./film.js";

const OUT = path.resolve(import.meta.dirname, "../output");
const [mode = "video", ...args] = process.argv.slice(2);

if (mode === "preview") {
  // The browser runs film.ts itself; no need to build the film here.
  const session = await startPreview({ entry: new URL("./film.ts", import.meta.url), export: "buildFilm" });
  console.log(`Preview at ${session.url}`);
  await session.closed; // serves until its tab closes or the next preview replaces it
  process.exit(0);
}

const comp = await buildFilm();
await fs.mkdir(OUT, { recursive: true });

if (mode === "frames") {
  const renderer = new HeadlessMediaRenderer();
  await fs.mkdir(path.join(OUT, "frames"), { recursive: true });
  for (const f of args.map(Number)) {
    const png = await comp.renderFrame({ frame: f, renderer });
    const file = path.join(OUT, "frames", `f${String(f).padStart(4, "0")}.png`);
    await fs.writeFile(file, png);
    console.log(`Rendered frame: ${file}`);
  }
} else {
  const started = Date.now();
  const result = await comp.renderVideo({
    outputPath: path.join(OUT, "output.mp4"),
    quality: "high",
  });
  await result.cleanup?.();
  console.log(`Exported video: ${result.filePath} in ${((Date.now() - started) / 1000).toFixed(1)}s`);
}

// CRITICAL: Always explicitly exit process to prevent native WebGPU/mediabunny thread hangs
process.exit(0);

CLI Usage:

# Render specific keyframes for instant review:
tsx src/render.ts frames 0 18 36 72

# Live player for the user (run in the background; re-run after changes and the open tab reloads):
tsx src/render.ts preview

# Render full master video:
tsx src/render.ts

Built-in Video QA (`renderVideo({ qa })`)

The engine measures the video while it renders: every frame as it is read back for the encoder, and the audio as it is mixed. No second decode pass, and the numbers describe exactly what was encoded.

import { formatQaReport } from "framefields";

const result = await comp.renderVideo({
  outputPath: "output/final-video.mp4",
  qa: { targetLufs: -14 }, // or `qa: true` for defaults (no loudness target)
});
console.log(formatQaReport(result.qa!));
if (!result.qa!.passed) process.exitCode = 1; // fail CI / the agent loop

result.qa is structured, so branch on it instead of parsing logs:

`issues[].code`	Severity	Meaning
`frames_missing`	error	Fewer frames encoded than the composition has
`black_frames`	warning	≥ `minBlackSec` (0.5 s) of black frames; has `startSec`/`endSec`
`audio_clipping`	warning	Samples at or beyond full scale
`loudness_off_target`	warning	Integrated loudness more than `lufsTolerance` (1 LU) from `targetLufs`
`animation_truncated`	warning	Keyframes past a layer's clip window never play; has `layerId`
`frozen_video`	info	Picture unchanged for ≥ `minFrozenSec` (2 s); fine for held title cards
`audio_silence`	info	≥ `minSilenceSec` (2 s) below `silenceDbfs` (-60)
`audio_silent`	info	The whole track is silent
`invalid_gradient`	error	A `background` / `fill` gradient string that does not parse: it draws nothing
`unsupported_image_format`	error	An image layer receiving a `.json` / `.lottie` vector animation, video, or audio file
`unknown_color`	warning	A color string that is not a color: it draws transparent
`low_contrast_text`	warning	Text below 3:1 contrast against the solid color behind it (suggests a readable one)
`offscreen_layer`	warning	A static absolute layer entirely outside the canvas
`shadow_unsupported_light`	warning	`castShadow` on a point or ambient light
`pbr_without_environment`	info	`roughness` / `metallic` set but no `Layer.environment()` (no physical shading)
`multiple_shadow_lights`	info	Several `castShadow` lights; only the first casts
`static_track`	info	Every keyframe on a track holds the same value

The document checks also run on their own, before any render: comp.lint() returns the same { severity, code, message, layerId } entries. Run it in Step 1, right after typechecking.

passed is false when any error or warning is present. qa.audio.integratedLufs is ITU-R BS.1770 loudness (streaming platforms normalize to about -14 LUFS); qa.audio.peakDbfs is the sample peak.

Inspecting a Finished File with ffmpeg

For files rendered elsewhere, or to compare against a previous render, inspect with ffmpeg/ffprobe:

# 1. Check video and audio stream properties
ffprobe -v error -show_entries stream=codec_name,width,height,duration,r_frame_rate output/final-video.mp4

# 2. Detect unexpected black frames or dead transitions
ffmpeg -i output/final-video.mp4 -vf "blackdetect=d=0.5:pix_th=0.10" -f null -

# 3. Verify audio loudness levels (prevent clipping / silence)
ffmpeg -i output/final-video.mp4 -af "volumedetect" -f null -

Common Rendering Failure Modes

Symptom	Root Cause	Solution
Entire frame renders black	Missing font registration or `NaN` in uniform	Ensure every font used in `Layer.text` is registered via `FontManager.register()` and listed in `comp.fonts`. Verify all signal outputs are finite numbers.
Media layer is invisible / blank	Bad input handle binding or zero opacity	Check that the media path exists and `opacity > 0`. If using an effect or section, ensure `fit: "cover"` and valid container dimensions.
Audio cuts out early	Missing `durationFrames` on audio layer	Explicitly assign `durationFrames: totalFrames` on `Layer.audio()`.
Vision cutout / mask is empty on a single `renderFrame`	Vision reads the previous rendered frame; frame 0 has none	Render two or more consecutive frames with the same `HeadlessMediaRenderer` and keep the last. `renderVideo` handles this automatically.
Ghosted double of the subject in a vision cutout	Frames rendered out of order (frame grid, or jumping from frame 3 to 48), so the cutout uses pixels from another frame	Verify vision layers with sequential frames or the exported MP4, not `renderFrameGrid`.
Script hangs after rendering	Background worker thread pool held open	Always include `process.exit(0)` at the end of render scripts.
Text looks misaligned or truncated	Layout node box smaller than text string	Set explicit wrapping `width` (e.g. `width: 1400`) or use `width: "fill"`.
Colors appear washed out / dark edges	Unpremultiplied alpha blending	The engine operates with premultiplied alpha. Ensure custom shaders use `srcFactor: "one"`.