claude-video-caption

claude-video-caption — Transcription and Animated Subtitles

Pre-Flight

Check Whisper is installed: command -v whisper-ctranslate2 || echo "Run /video setup"
Run bash scripts/preflight.sh "$INPUT" "$OUTPUT"
Analyze existing subtitle tracks: ffprobe -v error -select_streams s -show_entries stream=codec_name,codec_type -of json "$INPUT"

One-Command Caption Pipeline

For the most common use case (transcribe + animated captions + burn-in):

bash scripts/caption_pipeline.sh "$INPUT" "$OUTPUT" [language] [style]

language: auto (default), en, es, fr, de, ja, zh, etc. (99 languages supported)
style: default, minimal, bold, neon, shadow

Step-by-Step Manual Pipeline

Step 1: Extract Audio for Whisper

ffmpeg -y -i "$INPUT" -vn -ar 16000 -ac 1 -f wav /tmp/claude_video_audio.wav

Step 2: Transcribe with faster-whisper

# Auto-detect language, word-level timestamps
whisper-ctranslate2 /tmp/claude_video_audio.wav \
  --model large-v3 \
  --output_format json \
  --word_timestamps True \
  --vad_filter True \
  --compute_type int8 \
  --output_dir /tmp/

# For faster transcription (slight accuracy tradeoff)
whisper-ctranslate2 /tmp/claude_video_audio.wav \
  --model large-v3-turbo \
  --output_format json \
  --word_timestamps True \
  --vad_filter True \
  --compute_type float16 \
  --output_dir /tmp/

Model selection:

Model	Size	VRAM	Speed	Accuracy	Use Case
tiny	75M	~1 GB	32x	Low	Quick drafts
base	142M	~1 GB	16x	Fair	Quick subtitles
small	466M	~2 GB	6x	Good	General use
medium	1.5B	~5 GB	2x	Very good	Quality subtitles
large-v3-turbo	809M	~6 GB	8x	Very good	Best speed/accuracy (recommended)
large-v3	1.5B	~10 GB	1x	Best	Professional captioning

Step 3: Generate ASS Subtitles with Karaoke Timing

Convert Whisper JSON output to ASS format with word-by-word highlighting.

Generate this Python script inline and run it:

#!/usr/bin/env python3
"""Convert Whisper JSON word timestamps to ASS karaoke subtitles."""
import json, sys

def words_to_ass(json_path, output_path, words_per_line=3, style="default"):
    with open(json_path) as f:
        data = json.load(f)

    # Collect all words with timestamps
    words = []
    for segment in data.get("segments", []):
        for w in segment.get("words", []):
            words.append({
                "word": w["word"].strip(),
                "start": w["start"],
                "end": w["end"]
            })

    # Style presets (full 23-field ASS V4+ format)
    # Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour,
    #         BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing,
    #         Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
    styles = {
        "default":  "Default,Arial,20,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,2,1,2,10,10,40,1",
        "bold":     "Default,Impact,24,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,-1,0,0,0,100,100,0,0,1,3,2,2,10,10,40,1",
        "minimal":  "Default,Helvetica,18,&H00FFFFFF,&H000000FF,&H80000000,&H80000000,0,0,0,0,100,100,0,0,1,1,0,2,10,10,30,1",
        "neon":     "Default,Arial Black,22,&H0000FFFF,&H00FF00FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,2,0,2,10,10,40,1",
        "shadow":   "Default,Georgia,20,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,0,3,2,10,10,40,1",
    }
    style_line = styles.get(style, styles["default"])

    # ASS header
    ass = f"""[Script Info]
Title: claude-video captions
ScriptType: v4.00+
PlayResX: 1920
PlayResY: 1080

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: {style_line}

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
"""

    def fmt_time(seconds):
        h = int(seconds // 3600)
        m = int((seconds % 3600) // 60)
        s = seconds % 60
        return f"{h}:{m:02d}:{s:05.2f}"

    # Group words into lines
    for i in range(0, len(words), words_per_line):
        chunk = words[i:i + words_per_line]
        if not chunk:
            continue
        start = chunk[0]["start"]
        end = chunk[-1]["end"]

        # Build karaoke text with \kf tags
        text_parts = []
        for w in chunk:
            duration_cs = max(1, int((w["end"] - w["start"]) * 100))
            text_parts.append(f"{{\\kf{duration_cs}}}{w['word']}")

        text = " ".join(text_parts)
        ass += f"Dialogue: 0,{fmt_time(start)},{fmt_time(end)},Default,,0,0,0,,{text}\n"

    with open(output_path, "w") as f:
        f.write(ass)

if __name__ == "__main__":
    words_to_ass(sys.argv[1], sys.argv[2],
                 int(sys.argv[3]) if len(sys.argv) > 3 else 3,
                 sys.argv[4] if len(sys.argv) > 4 else "default")

Step 4: Burn Subtitles into Video

# ASS subtitles (styled, animated)
ffmpeg -n -i "$INPUT" -vf "ass=/tmp/captions.ass" -c:a copy "$OUTPUT"

# SRT subtitles (simple, with basic styling)
ffmpeg -n -i "$INPUT" -vf "subtitles=/tmp/captions.srt:force_style='FontSize=24,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=2'" -c:a copy "$OUTPUT"

Subtitle Extraction

# Extract first subtitle track as SRT
ffmpeg -n -i "$INPUT" -map 0:s:0 subtitles.srt

# Extract as ASS
ffmpeg -n -i "$INPUT" -map 0:s:0 subtitles.ass

# Extract all subtitle tracks
ffprobe -v error -select_streams s -show_entries stream=index,codec_name:stream_tags=language -of json "$INPUT"
# Then extract each: ffmpeg -n -i "$INPUT" -map 0:s:INDEX output.srt

Subtitle Format Conversion

# SRT to ASS
ffmpeg -n -i input.srt output.ass

# ASS to SRT
ffmpeg -n -i input.ass output.srt

# SRT to WebVTT (add "WEBVTT" header)
echo "WEBVTT" > output.vtt && echo "" >> output.vtt && cat input.srt >> output.vtt

Subtitle Timing Adjustment

# Delay subtitles by 2 seconds
ffmpeg -n -i input.srt -itsoffset 2 output.srt

# For manual offset, edit SRT timestamps with awk or Python

ASS Styling Reference

Key ASS override tags for caption customization:

\fn{FontName} — change font
\fs{Size} — font size
\c&HBBGGRR& — primary color (BGR format, not RGB)
\3c&HBBGGRR& — outline color
\bord{Width} — outline width
\shad{Depth} — shadow depth
\fscx{%}\fscy{%} — scale X/Y (for "pop" animation: \fscx120\fscy120)
\fad(FadeIn,FadeOut) — fade timing in ms
\kf{Duration} — karaoke smooth fill (duration in centiseconds)
\K{Duration} — karaoke instant fill
\an{Position} — alignment (1-9 numpad layout, 2=bottom-center, 8=top-center)
\pos(x,y) — absolute position

Common Presets

TikTok/Reels style (large, centered, bold):

FontName=Impact,FontSize=28,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=3,Shadow=0,Alignment=2,MarginV=300

YouTube style (clean, bottom):

FontName=Arial,FontSize=20,PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=2,Shadow=1,Alignment=2,MarginV=40

Podcast/interview (minimal, lower third):

FontName=Helvetica,FontSize=16,PrimaryColour=&H00FFFFFF,BackColour=&H80000000,BorderStyle=4,Alignment=2,MarginV=20