SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscreate-cinematic-music-video

create-cinematic-music-video

Marketing & growth

View repository

Updated 10-15 21:54

About this skill

create-cinematic-music-video

Originally validated in content-goose at clients/gorgie/ad-runs/run-02-club-gorgie-music-video/, clients/hype-and-vice/ad-runs/run-01-game-day-girls/, and clients/hype-and-vice/ad-runs/run-02-night-game/. Adapt paths to your project.

Purpose

Turn a concept.json into a beat-locked 20–30 s cinematic music-video ad with original vocals, brand-correct visuals, and burned-in captions. The molecule generalizes everything we learned across the Gorgie + Hype and Vice reference runs into one pipeline that a fresh session can invoke against a new brand in ~30 minutes wall-clock with ~$15-25 of paid gen credits.

Three pillars make this format work:

  1. Original anthem with word timestamps. ElevenLabs Music API --with-timestamps gives 80–90 word-level anchors per 28 s song. Every visual cut lands on a vocal beat. This is the difference between "music video" and "stock music with cuts."
  2. 14-tableau structure with assigned narrative roles. Not a montage. A 3-act arc — anticipation → energy → reflection. Each tableau has a job (intro / wide / kinetic-close / hook / product / origin / outro).
  3. Look pack as the swap point for aesthetic. Kodak Portra warm-day, CineStill 800T neon-night, paper-cut craft — same molecule, three completely different finished products. Adding a new pack is a one-file PR.

The output is platform-ready: 1080×1920 9:16, 30 fps, h264 + aac, loudnorm -14 LUFS, captions burned, ~20-30 MB.

When to use this molecule

Use when:

  • Brand has cultural ownership of a setting / moment / vibe (tailgate, energy-drink-3pm, summer-friday)
  • Format needs to deliver emotional payoff, not feature callouts
  • Spot length 20-30s for paid social (TikTok, Reels, Shorts, Pinterest video pin)
  • Music can be original (ElevenLabs Music) — i.e. not licensed-music-required
  • Brand has a strong visual signature compatible with one of the existing look packs (or you'll add a new one)

Do NOT use for:

  • UGC talking-head ads (use ugc-ad/create-ugc-creatify or ugc-ad/create-ugc-higgsfield)
  • Motion-graphic kinetic typography (use motion-graphics/create-motion-graphic-video)
  • Sub-15s short cuts (use ugc-ad/create-shortform-cuts-from-video after a master ships)
  • Photoreal product sizzle (use product-video/create-product-sizzle-reel)
  • When licensed music is mandatory (this molecule generates original)

Inputs

A single concept.json (schema below) plus:

  • Brand brief (shared/brand-brief.md in the brand's workspace)
  • Lyrics in source/lyrics-locked.md (also a content_field in concept.json so you don't have to keep them in sync)
  • Look pack selection (one of KODAK_PORTRA_DAY | CINESTILL_800T_NIGHT | PAPER_CUT_CRAFT | a custom one you add)

Environment:

  • ELEVENLABS_API_KEY for music generation
  • Higgsfield CLI authenticated for keyframes + i2v
  • imageio-ffmpeg installed via pip for libass-enabled caption burn (Homebrew ffmpeg lacks libass on macOS — known issue, see Workarounds below)

concept.json schema

{
  "brand": "hype-and-vice",
  "run_slug": "run-02-night-game",
  "title": "Night Game",
  "campaign": "every day is game day",
  "duration_s": 28,
  "look_pack": "CINESTILL_800T_NIGHT",
  "palette_anchors": ["deep indigo", "neon magenta", "sodium amber", "scarlet pop"],

  "music": {
    "bpm": 125,
    "vibe": "indie-electronic anthem, dark synth foundation, female vocal with grit, big stadium drum, neon-night driving energy",
    "structure_note": "Synth pad opening, drums punch in at the pre-chorus, full electronic-band drop at the chorus, returns to synth pad for outro."
  },

  "lyrics_locked_md": "source/lyrics-locked.md",
  "accent_words": ["Hype","Vice","every","day","game","HER","girls","started","finish"],

  "tableaux": [
    {
      "id": "T01",
      "role": "INTRO",
      "lyric_anchor": "(intro / 'Friday night')",
      "prompt": "Hero subject: a scarlet HYPE AND VICE Ohio State basketball jersey draped over a desk chair in a college dorm room at night. Warm desk lamp glow on the jersey. A pink neon 'OPEN' sign through a frosted window casts magenta light. Movie posters on the wall. Snow visible outside. Composition: low angle on the chair, intimate and anticipatory. The jersey is the hero.",
      "motion_hint": "The desk lamp glow flickers ever so slightly. The neon sign pulses gently. Draft moves the curtain edge. The jersey stays still as hero."
    }
    // ... 13 more tableaux
  ]
}

Tableau roles (assigned, structure-locked):

# Role What it is
1 INTRO Cold-open, no music yet, product hero hint
2 PERSONAL_BEAT_1 Character + product close beat
3 PERSONAL_BEAT_2 Walk-up / movement
4 WIDE_ENSEMBLE Group scene wide, golden/neon flare
5 KINETIC_CLOSE_1 Emotional close-up
6 KINETIC_CLOSE_2 Secondary energy beat
7 SUSPENDED_TIME Slow-mo confetti / abstract beat
8 HOOK_HERO Chorus drop — quadrant split / hero
9 KINETIC_PEAK Biggest energy moment
10 PRODUCT_HERO Garment / product alone in moody light
11 MICRO_MONTAGE Rapid 4-cut beat-locked inside one slot
12 ORIGIN_WINK Brand history Easter egg
13 TWILIGHT_OUTRO Reflective walk-out
14 END_CARD Wordmark + tagline + hero SKU

This structure is load-bearing. Don't skip slots — even a 20 s cut compresses each tableau to ~1.4s, not removes them. The 3-act arc is what separates this format from generic AI-music-video slop.

Look packs

A look pack locks the photographic style across all 14 tableaux. The pack provides:

  • STYLE_OPENER string injected at the head of every keyframe prompt
  • NEGATIVE_TAIL string injected at the end of every keyframe prompt
  • Palette anchor list
  • Music structure default
  • Caption style (placement, chunk-size, font, accent)

Built-in packs (all under lookpacks/):

Pack When to pick Reference run
KODAK_PORTRA_DAY Warm afternoon, autumn, golden hour, lifestyle, "Saturdays" hype-and-vice/run-01-game-day-girls
CINESTILL_800T_NIGHT Neon-noir, winter, urban, nightlife, indoor venues hype-and-vice/run-02-night-game
PAPER_CUT_CRAFT Y2K, candy-coded, kids/wellness/CPG brands, "craft" energy gorgie/run-02-club-gorgie-music-video

Adding a new pack: drop a file in lookpacks/<NAME>.md with the four fields above. The scripts read packs by name lookup — no code changes.

Workflow

S0 INTAKE          ← read concept.json + brand-brief.md + lyrics-locked.md
S1 STORYBOARD      ← gen storyboard.html (Gate 2 review surface)
S2 MUSIC_GEN       ← ElevenLabs Music with --with-timestamps → music.mp3
S3 WORDS_ADAPTER   ← convert ms→s, ElevenLabs shape → Whisper shape
S4 KF_T01_TEST     ← generate single keyframe (Gate 3, aesthetic validation)
S5 KF_BATCH        ← 13 parallel keyframes
S6 I2V_BATCH       ← 14 parallel image-to-video
S7 STITCH          ← concat segments to silent master
S8 MUX_AUDIO       ← mix music + loudnorm to -14 LUFS
S9 CAPTION_BURN    ← ASS gen + libass-binary burn → master-final.mp4

Each state is idempotent — re-running the molecule will skip completed steps (file-existence checks). The driver scripts/one_shot.py walks the whole state machine; you can also invoke individual scripts to re-run a single state.

Gates

Only two human gates (vs the 5 of motion-graphics) because the cheapest review surface is the storyboard:

  • Gate 2 — Storyboard approval (after S1). Open storyboard.html in browser. Approve the 14 tableaux + music vibe + look pack before any paid gen starts.
  • Gate 3 — T01 aesthetic test (after S4). Look at assets/keyframes/T01.png. Decide if the look pack landed correctly before committing to the parallel batch of 13.

Everything else is automatic + idempotent.

Output

<brand>/ad-runs/<run-slug>/
├── intake-brief.md
├── concept.json
├── source/
│   └── lyrics-locked.md
├── storyboard.html               ← Gate 2
├── audio/
│   ├── music.mp3
│   ├── music_metadata.json       ← raw ElevenLabs response
│   └── words.json                ← Whisper-shape adapter
├── assets/
│   ├── keyframes/T01..T14.png    ← Gate 3 = T01.png
│   └── clips/T01..T14.mp4
├── working/
│   ├── segments/seg_T01..T14.mp4
│   ├── concat.txt
│   └── logs/                     ← per-job logs
├── clips/
│   ├── master-silent.mp4
│   └── master-no-captions.mp4
├── finals/
│   ├── master-final.mp4          ← the deliverable
│   └── master-final.ass
└── manifest.json

Workarounds baked in (lessons from reference runs)

These are real failure modes hit during Gorgie + H&V development. The scripts handle each automatically — no operator action needed.

  1. ElevenLabs Music ToS filter on artist names. Prompts containing "Charli XCX", "Lana Del Rey", "Petra Collins", etc. return HTTP 400 with bad_prompt. The scripts strip artist names from the music vibe field. Use descriptive language ("indie-electronic anthem with female vocal grit") not name references.

  2. gpt_image_2 NSFW false-flags. "Young woman", "twin XL bed", "sip", "reach into frame", "Petra Collins" all triggered false NSFW rejections during reference runs. The scripts default to object-hero framing — "Hero subject: a [garment] on a [surface]" — and only introduce people in supporting roles. The shared STYLE_OPENER per look pack avoids the trigger words.

  3. Higgsfield 502s. Transient. The scripts retry once on 502/timeout per job. If a single keyframe still fails, the operator re-runs only that one (python3 gen_keyframes.py T12).

  4. ElevenLabs returns ms timestamps; sync.py expects seconds. The adapter build_word_timestamps.py converts ms→s and renames start_ms/end_ms → start/end + adds text (Whisper shape). Always run S3 before S9.

  5. Homebrew ffmpeg lacks libass. Caption burn fails with No such filter: 'subtitles'. The scripts use the imageio-ffmpeg static binary (pip install imageio-ffmpeg) which has libass enabled. Path resolution is auto.

  6. Captions need libass-binary path injection. The sync-captions-to-music atom shells out to Homebrew ffmpeg. Workaround: let sync.py generate the .ass file (it succeeds at that part), then we manually run the burn step with the libass binary. This is what stitch.py does after S9.

  7. Beat-anchored TIMELINE varies per song. ElevenLabs doesn't deliver exactly on the spec — verse can stretch a few hundred ms, hook lands at 15.4 s not exactly 14.0 s. The scripts read music_metadata.json and compute the TIMELINE per-run. Never hard-code timings across runs.

Composed Atoms

  • atoms/music/create-music-elevenlabs — original anthem with word timestamps (S2)
  • atoms/image-generation/create-image-nano-banana-fal — keyframes for T01..T14 (S4, S5)
  • atoms/video-generation/create-video-kling — image-to-video, kling3_0 (S6)
  • atoms/assembly/stitch-videos-ffmpeg — concat segments to silent master (S7)
  • atoms/audio-editing/mix-master — mix music + loudnorm to -14 LUFS (S8)
  • atoms/captions/burn-in-captions — ASS gen + libass-binary burn (S9)

Quality Checks

  • audio/music.mp3 exists and audio/music_metadata.json returns word-level timestamps.
  • audio/words.json is in Whisper shape ({words:[{text,start,end}]}) with start/end in seconds.
  • All 14 keyframes T01..T14 exist under assets/keyframes/.
  • All 14 i2v clips exist under assets/clips/ and pass through ffprobe.
  • finals/master-final.mp4 is 1080×1920, 30 fps, h264 + aac, duration 20-30 s.
  • Loudness normalized to -14 LUFS with no clipping.
  • Captions burned cleanly (no No such filter: 'subtitles' errors — libass binary path resolved).

Decision Rules

  • One look pack per run. Don't mix Kodak Portra and CineStill across tableaux — the through-line is what makes the spot read as one ad.
  • Same brand mark on every tableau. Vary SKU / school / color but never the wordmark logic.
  • Chorus mantra is the campaign anchor. When making a series (Run 01 → Run 02 → Run 03), the chorus + outro stay identical, only verse + pre-chorus + tableaux change.
  • Hook lands at exactly the chorus drop word boundary. The TIMELINE auto-computes this from the music metadata — don't override unless you're trying to break the format on purpose.
  • Captions = lower-third serif italic. Music-video preset with --placement low --chunk-size 4 for cinematic feel. Word-burst pop is for paper-cut, not 35mm.

Failure Modes

  • S2 music gen returns HTTP 400 bad_prompt. Check vibe field for artist names. The API response includes a prompt_suggestion you can use verbatim.
  • S4 T01 keyframe NSFW-flagged. Sanitize the prompt: drop "young woman", "twin XL bed", "Petra Collins"; use object-hero framing.
  • S5 keyframe batch has 1-2 fails. Re-run only those: python3 gen_keyframes.py T12 T08.
  • S6 i2v fails on one clip. Re-run only that one. The kling3_0 timeout is 20 minutes.
  • S9 caption burn fails with No such filter: 'subtitles'. Homebrew ffmpeg installed instead of libass binary. pip install imageio-ffmpeg and stitch.py auto-finds it.
  • TIMELINE durations don't sum to total. Re-derive from music_metadata.json — see build_timeline_from_metadata helper in scripts/timeline.py.

Reference runs

The molecule was extracted from three working spots:

Run Look pack Output What it taught the molecule
gorgie/ad-runs/run-02-club-gorgie-music-video/ PAPER_CUT_CRAFT 20 s · 18.9 MB Word-burst captions; saturated paper-cut works for CPG
hype-and-vice/ad-runs/run-01-game-day-girls/ KODAK_PORTRA_DAY 28 s · 25 MB 14-tableau narrative arc beats 8-tableau montage; lower-third captions; founder-wink Easter egg
hype-and-vice/ad-runs/run-02-night-game/ CINESTILL_800T_NIGHT 28 s · in flight Campaign series with shared chorus; cool palette + neon halation

Quick start (for a new brand)

# 1. Set up workspace
mkdir -p <brand>/ad-runs/<run-slug>/{source,assets/keyframes,assets/clips,clips,finals,working/segments,working/logs,audio}

# 2. Write the brand brief
$EDITOR <brand>/shared/brand-brief.md

# 3. Write the concept.json (see schema above)
$EDITOR <brand>/ad-runs/<run-slug>/concept.json

# 4. Write lyrics
$EDITOR <brand>/ad-runs/<run-slug>/source/lyrics-locked.md

# 5. Run the one-shot driver
python3 skills/molecules/create-cinematic-music-video/scripts/one_shot.py \
    --concept <brand>/ad-runs/<run-slug>/concept.json

# Pauses at Gate 2 (storyboard) and Gate 3 (T01).
# Resume after each gate with: python3 .../one_shot.py --concept <...> --continue

The driver writes a .state.json file in the run folder so it can resume from any state.