retime-captions-to-words

retime-captions-to-words

Purpose

Captions authored by hand (or generated from an LLM transcript) frequently drift ±100–300 ms relative to actual word starts in the VO. This atom snaps each cue's start time to the nearest matching word from a whisper word-timestamps file, within a tolerance.

End times are recomputed to preserve the cue's original duration unless the next cue would overlap, in which case they butt up to the next cue's new start.

Inputs

• --input <path> — source .srt (required)
• --output <path> — destination .srt (required)
• --words <path> — whisper word-level timestamps JSON (required). Expected schema: {"words": [{"text": "...", "start": 1.234, "end": 1.456}, ...]}
• --tolerance <sec> — max snap distance (default 0.10)
• --preserve-duration <bool> — keep original duration after snap (default true)

Algorithm

For each cue:

1. Take the cue's first 1–3 words (normalized: lowercase, strip punctuation).
2. Find candidate matches in the words list whose normalized form starts the same and whose start is within tolerance of the cue's current start.
3. Pick the candidate with the smallest |word.start - cue.start|.
4. If a match is found:
   • Set cue.start = matched word.start.
   • If preserve-duration, shift cue.end by the same delta. Otherwise leave cue.end unchanged.
5. If no match within tolerance, leave the cue unchanged and record in skipped list.
6. Post-pass: ensure no cue overlaps the next cue's new start; clip end times if needed.

Output

• <output> — retimed SRT
• manifest.json:

  {
    "input": "...", "output": "...", "words": "...",
    "tolerance_seconds": 0.10,
    "cues": 42,
    "retimed": 31,
    "skipped": 11,
    "max_shift_seconds": 0.087,
    "clipped_ends": 2,
    "status": "ok"
  }
  

• verification.md — table of every retimed cue (before/after start, delta).

Quality checks

• max_shift_seconds <= tolerance.
• No cue starts after its end.
• No cue overlaps the next cue (cue.end <= next_cue.start).
• Whisper words file is well-formed (validated on load).

Failure modes

• --words file missing or malformed → fail fast with clear error.
• Cue's first word can't be found in the transcript window — leave untouched, record in skipped.
• Whisper transcript has gaps (silence between words longer than the cue) — algorithm still works because we only match the cue's first word.
• VO and caption first words diverge (caption was rewritten for readability) — cue is correctly skipped; operator should re-author that line.
• Repeated words sync to the wrong instance. When ANY downstream script does a word → timestamp lookup against words.json outside this atom's tolerance-window flow (e.g. retiming overlays, SFX, or zoom-punches to a specific word like "every" or "Claude"), default first-match silently grabs the wrong occurrence if that word appears multiple times. Verified failure: 11-walking-felt-goose 2026-05-24 — "every" in "generate every clip" at 14.6s vs. "Every. Single. Frame." at 28.3s; first-match grabbed the early one, climax click-ratchets almost shipped 14s early.

Canonical word-lookup helper

Any script that retimes media to a specific Whisper word MUST use this signature (or equivalent) — keyword-only after forces conscious disambiguation:

def find_word(words: list[dict], target: str, *, after: float = 0.0) -> dict | None:
    """Return the first Whisper word matching `target` (case-insensitive,
    punctuation-stripped) whose `start` is >= `after`.

    `after` is keyword-only — callers must consciously pass either 0.0
    (acknowledged first-match) or a real disambiguating timestamp.
    """
    t = target.lower().strip(".,?!")
    for w in words:
        if w["text"].lower().strip(".,?!") == t and w["start"] >= after:
            return w
    return None

Companion recommendation: print derived timestamps to stderr BEFORE rendering so misalignment shows up in the log, not in the final video.

Example

scripts/retime.py \
  --input peloton/ads/video-03-ironman-comeback/captions/captions.srt \
  --output peloton/ads/video-03-ironman-comeback/captions/captions-retimed.srt \
  --words peloton/ads/video-03-ironman-comeback/audio/word-timestamps.json

Consumed by /polish (caption-timing branch) and auto-refine.