SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillssync-captions-to-music

sync-captions-to-music

Design & creation

View repository

Updated 10-15 21:54

About this skill

sync-captions-to-music

Purpose

Music-video ads need captions that feel like the song — short bursts, hits on the vocal, mid-frame placement, mixed-weight typography. This atom is the bridge between a rendered song (with timing data) and a burned-in caption pass.

It is a thin composition wrapper. It does NOT re-implement ASS generation or ffmpeg burning — it normalizes word timestamps to the schema the existing add-captions-burn/make_klap_style_subs.py script consumes, then invokes that script with a music-video preset.

If the song was produced by create-music-elevenlabs --with-timestamps, the timestamps come straight from music_metadata.json. If timestamps are missing or unreliable (a known risk for fast rap lines), the atom can fall back to running Whisper on the rendered MP3.

When to use

  • After the song is approved at the MUSIC GENERATE state of molecules/music-video-ad.
  • Re-burning captions after a song revision (re-runs are cheap; the song is the expensive part).
  • Standalone: any time you have a vocal track + lyrics and want fast-paced word-burst captions in the music-video style.

Do NOT use for:

  • VO ads with a deadpan narrator — those want bottom-third Klap karaoke; use add-captions-burn directly.
  • iMessage chat or motion-graphic ads — they have their own caption systems.

Inputs

  • --video <path> — the video to burn into (required). Typically edits/master-final.mp4 BEFORE the captions pass.
  • --lyrics <path> — lyric file with the canonical sung lines (required). Drives spelling and accent-word detection.
    • Prefer lyrics-plain.md — a clean text file with one sung line per row, no analysis prose, no markdown tables/headings/lists, no inline-code backticks.
    • If you pass lyrics-locked.md and a sibling lyrics-plain.md exists, the script auto-prefers the plain twin.
    • Why this matters: lyrics-locked.md often becomes a project README with structure notes, accent-word reference sections, and timing tables. The parser tries to filter those out (skips | tables, # headings, > blockquotes, code-fenced blocks), but inline-code backticks (`word`) used to mark accent words bled into the captions on the DIBS run — burning literal backticks into every accented caption. The current parser strips backticks/bold-asterisks/italic-underscores defensively (since 2026-05-23), but the cleanest contract is: lyrics-plain.md = only the sung text.
  • --timestamps <path> — one of:
    • ElevenLabs music_metadata.json (preferred, free with --with-timestamps)
    • Whisper word-level word-timestamps.json ({"words":[{"text","start","end"}]})
    • Omit and pass --whisper-fallback to transcribe the song's audio track via Groq Whisper.
  • --output <path> — destination video (required).
  • --accent-words <comma-list> — words to render in bold-italic for emphasis. Defaults: brand name, product name, climax-line keywords if present in lyrics frontmatter.
  • --chunk-size <int> — max words per caption chunk (default 3).
  • --placement <mid|low|high> — vertical placement (default mid for music-video, low for VO-style).
  • --font <name> — default New York (system serif on macOS) for music-video; falls back to Georgia.
  • --font-size <int> — default 64 at 1080×1920 (scaled to PlayResY).
  • --style-preset <name> — music-video (default; serif, mid-frame, mixed weights) or klap-karaoke (Inter Bold ALL CAPS, bottom-third, yellow active-word).
  • --preview-strip <path> — optional output PNG showing chunked timeline vs waveform.

Workflow

  1. Resolve timestamps.

    • If --timestamps points at ElevenLabs music_metadata.json, walk the JSON and extract every {text, start, end} tuple. Schema isn't formally documented by ElevenLabs across versions — the script tries common shapes (metadata.timestamps, timestamps, alignment.words, words) and fails loudly if none match.
    • If it points at a Whisper-shaped file with {"words":[...]}, pass through.
    • If --whisper-fallback is set and no usable timestamps were extracted, extract audio from the video, upload to Groq Whisper (reuse the watch skill's whisper.py wrapper), and write word-timestamps.json alongside the output.

  2. Align timestamps to lyrics. ElevenLabs and Whisper both occasionally emit transcription artifacts (mishears, "uh"s, joined contractions). Diff against lyrics-locked.md and:

    • Drop tokens that don't appear in the lyrics.
    • Re-spell tokens to match the locked lyric spelling (so brand names render correctly).
    • Flag any line where word-count drift is > 20% — these need manual review before the burn.

  3. Chunk into bursts. Group words into chunks of 1–chunk-size (default 3) such that:

    • A chunk never crosses a ,, ., ?, !, or end-of-line in the lyrics.
    • A chunk never crosses an inter-word gap > 600 ms (silence break → start a new chunk).
    • Accent words get their own chunk OR are placed last in the chunk so they hit on the strong beat.

  4. Render ASS. Call make_klap_style_subs.py with --style-preset translated into its --font, --size, --marginv knobs. For music-video:

    • Font: New York 64pt, primary white, no yellow karaoke highlight (still words, not karaoke).
    • Accent words: {\b1\i1}WORD{\b0\i0} inline.
    • Placement: MarginV ≈ PlayResY × 0.50 (mid-frame), Alignment: 5 (centered).
    • Display window: chunk_first_word.start − 30 ms → chunk_last_word.end + 80 ms, clipped to never overlap the next chunk.

  5. Burn. make_klap_style_subs.py already handles ffmpeg burning via -vf subtitles=. Keep audio stream copy.

  6. Preview strip (optional). If --preview-strip is set, render a horizontal PNG: x-axis = song timeline, top row = ffmpeg-extracted waveform, bottom row = colored rectangles per chunk with the chunk text. This is the artifact the storyboard embeds for human review.

Output

  • <output> — captioned video (mp4, H.264, audio copied).
  • <output>.ass — the generated subtitle file (kept for re-burn/inspect).
  • chunks.json — [{start, end, text, accent: bool, words: [...]}] — feeds the storyboard.html preview.
  • preview-strip.png (if requested) — waveform + chunk overlay.
  • manifest.json:
{
  "status": "ok",
  "video_in": "...",
  "video_out": "...",
  "lyrics": "...",
  "timestamps_source": "elevenlabs|whisper-passthrough|whisper-fallback",
  "chunks": 38,
  "accent_words": ["Gum", "Gods", "date", "bay"],
  "warnings": []
}
  • verification.md — chunk count, longest chunk duration, longest inter-chunk gap, max snap shift.

Quality checks

  • Every locked lyric word appears in at least one chunk.
  • No chunk shorter than 180 ms (illegible flash) — merge or extend.
  • No chunk longer than 1.4 s (camped on too long for music-video pacing) — split.
  • Chunk start times within ±100 ms of the matching word's vocal onset.
  • No chunk overlap with the next chunk's start.
  • Accent words actually render with \b1\i1 inline tag.

Failure modes

  • ElevenLabs metadata schema unrecognized → fail with the schema seen, hint to use --whisper-fallback.
  • Whisper transcript diverges materially from locked lyrics (> 20% word drift) → emit warnings[] and stop; operator should re-listen and decide whether the song is mumbling or the lyrics file is stale.
  • Font missing → fall back to Georgia; record in warnings.
  • Accent words never appear in the lyrics — likely a frontmatter typo; surface the mismatched word list before burning.
  • Subtitle line wraps to two lines on narrow frames — drop chunk_size to 2.

Composition

  • Reuses atoms/captions/add-captions-burn/scripts/make_klap_style_subs.py for ASS+ffmpeg burn (does not re-implement).
  • Reuses atoms/captions/retime-captions-to-words algorithmically (the same snap-within-tolerance idea), but operates directly on word-level data, so we don't go through SRT.
  • Whisper fallback reuses the transcription path from the watch skill (scripts/whisper.py).

Example

python3 scripts/sync.py \
  --video peloton-ads/video-NN-gum-of-gods/edits/master-final.mp4 \
  --lyrics peloton-ads/video-NN-gum-of-gods/lyrics/lyrics-locked.md \
  --timestamps peloton-ads/video-NN-gum-of-gods/audio/music_metadata.json \
  --output peloton-ads/video-NN-gum-of-gods/edits/master-captioned.mp4 \
  --accent-words "Gum,Gods,date,bay,below" \
  --preview-strip peloton-ads/video-NN-gum-of-gods/captions/preview-strip.png

Quality Checks

Run these against the emitted chunks.json, manifest.json, verification.md, and the burned <output>.mp4 before a music-video ad is considered captioned. These are the binary gates a verifier (human or skills/verifiers/) checks; the "Quality checks" list above is the prose rationale.

  • Coverage. Every locked lyric word from --lyrics appears in at least one chunk in chunks.json. manifest.json.warnings[] is empty of "lyric word(s) never matched" entries (or each is explained).
  • No drift gate tripped. manifest.json.warnings[] contains no WORD-COUNT DRIFT line. Drift > 20% between the timed stream and the locked lyrics is a hard stop — re-listen before burning.
  • Legibility floor. No chunk in chunks.json is shorter than 0.18 s (the chunk_bursts merge-forward already enforces this; verification.md's longest/shortest read confirms it). Sub-180 ms flashes are illegible at music-video pace.
  • Camp ceiling. No chunk longer than ~1.4 s. verification.md reports "longest chunk"; anything above 1.4 s means a burst is parked on screen and should be split (drop --chunk-size or check for a missing punctuation boundary).
  • Onset snap. Each chunk's start sits within ±100 ms of the matching word's vocal onset (chunks are seeded at first_word.start − 30 ms). Eyeball against the --preview-strip waveform: rectangles should hug the vocal transients.
  • No overlap. No chunk's end runs past the next chunk's start (the clip-overlaps pass guarantees this; confirm visually that captions never stack).
  • Accent emphasis present. Every word in --accent-words that exists in the lyrics renders as {\b1\i1}WORD{\b0\i0} in <output>.ass, and manifest.json.accent_words lists exactly the supplied (and lyric-present) accents.
  • Placement + type. For the default music-video preset, captions are mid-frame (Alignment: 5, MarginV ≈ PlayResY × 0.50) in the serif font (New York, falling back to Georgia), not bottom-third Klap karaoke.
  • No literal markdown burned. No caption contains a stray backtick, **, or __ (the DIBS regression). If lyrics-plain.md exists it must be the parse target — [sync] stderr says which file was read.

Failure Modes

How the atom fails, what the failure looks like, and the operator's move. Most surface through manifest.json.warnings[], the [sync] stderr lines, or a non-zero exit.

  • ElevenLabs schema unrecognized. load_timestamps can't find words in any known shape (alignment.words, timestamps, metadata.timestamps, character-level rollup) and raises ValueError listing the top-level keys it saw. Fix: re-run with --whisper-fallback, or point --timestamps at a real music_metadata.json.
  • No timestamps at all. Neither --timestamps nor --whisper-fallback was given → exits no timestamps provided and --whisper-fallback not set. Supply one.
  • Whisper wrapper missing. --whisper-fallback set but the watch skill's whisper.py isn't importable → exits with the import error. Fix: install the watch plugin or pass pre-computed word-timestamps.json.
  • Lyrics/audio diverge (mumble or stale lyrics). Word-count drift > 20% → a WORD-COUNT DRIFT warning lands in manifest.json and the operator must decide whether the vocal is mumbling or lyrics-locked.md is stale; do not ship the burn until resolved.
  • Lyric file is a README, not lyrics. A lyrics-locked.md packed with timing tables / accent-word sections over-counts "lyric words" (the Coinbase 729-word incident). parse_lyrics filters | tables, # headings, > blockquotes, [section] markers, and code fences — but the safe contract is a sibling lyrics-plain.md, which the script auto-prefers and announces on stderr.
  • Backticks/bold burned into captions. Accent words written as `word` in the lyric file used to render literal backticks (the DIBS run). The parser now strips `, **, and __ defensively; if any leak, switch to lyrics-plain.md.
  • Accent word never in lyrics. A typo'd --accent-words (or frontmatter accent_words) entry that doesn't match any lyric word silently won't emphasize. Cross-check manifest.json.accent_words against intent before the burn; a missing entry usually means a frontmatter typo.
  • Font missing on host. New York absent → the burn falls back to Georgia; expected on non-macOS hosts. Note it and confirm the burned frame still reads as a serif, not a sans/Times fallback.
  • Caption wraps to two lines on narrow frames. A 3-word chunk overflows a tall 9:16 frame → re-run with --chunk-size 2.
  • ffmpeg burn fails. A bad input path or codec issue surfaces as a non-zero ffmpeg exit (the subprocess is check=True). Inspect the kept <output>.ass and the ffmpeg stderr; the subtitle file is preserved for re-burn.