SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillstalking-goose-creator-video

talking-goose-creator-video

Design & creation

View repository

Updated 10-15 21:54

About this skill

talking-goose-creator-video

Originally validated in content-goose at internal/gooseworks-ads/goose-talking-head-01..03/. Adapt paths to your project.

Purpose

The production line for the 9:16 fast-paced explainer-with-mascot format. Codifies the lessons from <your-project>/goose-talking-head-01 → 02 → 03 (three iterative builds of the same format).

Specifically tuned for:

  • list-style explainers — "5 tools you need", "3 mistakes to avoid", "the SaaS stack in 30s"
  • product / tool walkthroughs where each beat needs a logo + a sample/visual or a stand-alone text card
  • founder/mascot rundowns where credibility comes from naming the actual things, not from cinematic production value

NOT the right molecule for:

  • Long-form (>40s) talking head — use molecules/podcast-clip-animated-ad
  • Brand spots / concept films — use molecules/motion-graphics/create-motion-graphic-video
  • Pure UGC aesthetics (handheld, real apartment, no mascot) — use molecules/ugc-ad/create-ugc-diary-ad

Inputs

  • <concept> — required. Free-form paragraph describing what the mascot is explaining. Should imply a list (5 tools, 3 steps, etc.).
  • <beats> — required if <concept> doesn't enumerate. List of {label, vo_line, tool?, sample_image?, fillText?, logo?, samples?}. The molecule produces one scene per beat plus a hook + CTA. Practical max: 6 tips + hook + CTA = 8 total. More beats and the VO crowds.
  • <character> — default goose-finger-puppet. {name, anchor_reference, descriptor_block}. anchor_reference is a path to a single high-quality reference still on solid bg. descriptor_block is a verbatim paragraph naming every distinguishing feature (see references/character-descriptors.md).
  • <background_style> — default cozy-library. See references/background-presets.md for the 4 shipped presets.
  • <voice> — required. {voice_id, model?, settings?}. Default model: "eleven_v3" with stability 0.34, similarity 0.80, style 0.18. Voice ID must be in VOICES.md OR explicitly approved.
  • <tools> — required for tool scenes. List of {name, logo_path, sample_path?, samples?: [array], fillText?, accent_color?}. NEW v03 behavior: if no real sample image makes sense (e.g. "get credits", "ask claude", "use ffmpeg"), pass fillText instead — the spotlight renders a clean beige card with brown text. If a tip has multiple samples (e.g. "look at this and this and this"), pass samples (array) — the molecule will split that beat into N sub-overlays timed to the VO's word boundaries.
  • <music_bed> — optional. Path to instrumental track. If absent, molecule emits a TBD warning.
  • <output_dir> — destination project folder, e.g. <your-project>/goose-talking-head-NN/.
  • <reuse_anchor> — optional path to a previously-locked anchor. When set, skips S3 (anchor lock) entirely. Saves $0.24 + ~30s. All three shipped projects (v01/02/03) used the same anchor, so this is the common case for series content.
  • <reuse_lipsync_drafts> — optional bool, default false. If true, builds a cheap draft by concatenating previous projects' talking-head-master.mp4 files trimmed to v03's duration, instead of calling Veed Fabric. The goose's mouth motion won't match the new VO phonemes but stays alive. Use for fast iteration on layout/overlays before committing to real lip-sync. See "Cheap-draft mode" below.
  • <auto> — bool, default false. If true, passes through human gates with prior operator preferences.

Composed Atoms

  • atoms/voiceover/create-voiceover-elevenlabs — eleven_v3 with audio tags for the main VO
  • atoms/image-generation/create-image-gpt-image-fal — character anchor; preserves identity on textured/felt characters
  • atoms/image-generation/create-image-nano-banana-fal — fallback for cartoon/illustrated mascots only
  • atoms/lipsync/create-lipsync-veed-fal — veed/fabric-1.0 at 720p, always 720p (480p reads soft in PIP)
  • atoms/overlays/render-video-overlay-track — Playwright renderer for tool-spotlight + kinetic-title-clean overlays
  • atoms/captions/burn-in-captions — frosted-subtle default; pair with a brand-spelling patch step
  • atoms/assembly/stitch-videos-ffmpeg — per-scene composite + concat + audio mix (VO + SFX pops via adelay + amix)

In addition the molecule uses the ElevenLabs sound-generation API directly for a single pop SFX mixed at every tip boundary (see references/generate-sfx.sh), and the control-plane scripts at scripts/video_control_plane/{init_project,record_asset,record_state,validate_project}.py.

Workflow

The molecule walks a 9-state machine. Each state's contract and outputs are defined below.

S0 INTAKE  →  S1 SCRIPT LOCK*  →  S2 RAW MATERIALS  →  S3 ANCHOR LOCK*  →  S4 LIP-SYNC
   ↓                                                                            ↓
S8 DELIVER*  ←  S7 CAPTIONS  ←  S6 COMPOSITE + PROMOTE*  ←  S5 OVERLAYS  ←  ─ ─ ─ ─ ─ ─┘

* = human gate

S0 — Intake

  1. Validate <character.anchor_reference> exists.
  2. Validate <voice.voice_id> against VOICES.md; if missing AND not explicitly approved, FAIL.
  3. Validate every <tools[].logo_path> and <tools[].sample_path> if provided.
  4. Run init_project.py --video-folder <output_dir> --title "<title>" --format 9x16.
  5. Copy <character.anchor_reference> to <output_dir>/raw-materials/reference/.
  6. If <reuse_anchor> is set, copy it directly to <output_dir>/generated/character-lock/anchor-portrait.png (+ .meta.json and LOCKED.json) and mark S3 pre-approved.
  7. Write idea-brief.md and design-brief.md (templates in references/intake-templates.md).

S1 — Script lock (GATE 1)

  1. Author script/script-locked.md with three views: clean VO, tagged VO (light [curious], [deliberate], [emphasized], [reassuring]), per-scene mapping table.
  2. Budget: ~60 words ≈ 17–22s clean; with v3 tags expands to ~24–32s. Plan for the longer end.
  3. HUMAN GATE. Any change after this gate invalidates every downstream artifact.
  4. Sequential reveal pattern: if a beat says "this... and this... and this" (or any N-reveal pattern), keep it as ONE script line — the molecule will derive sub-overlay timings from word-level Whisper timestamps in S5.

S2 — Raw materials

  1. VO (main thread): eleven_v3 with with-timestamps endpoint, voice settings from <voice>. Save audio/voiceover.mp3 + audio/voiceover_timestamps.json.
  2. Slice into per-scene segments: parse char-level timestamps, find each scene's start anchor (e.g. "One —", "Two —", "Three —"), compute end as the previous-char-of-next-scene-start, slice at silence-midpoints. Write audio/scene_timing.json AND audio/segments/plan.json (with tip1SubBoundaries if any beat has multi-reveal).
  3. SFX: run references/generate-sfx.sh to generate ONE pop sound via ElevenLabs sound-generation API. Save to audio/sfx-pop.mp3. ~$0.05.
  4. Tool assets (sub-agent if not pre-provided): brief asks for official transparent PNG logo + ONE clean hero image per tool. Hard rule for samples: no promotional collages with text baked in — they fight the spotlight card layout.

S3 — Anchor lock (GATE 2)

  1. Skip if <reuse_anchor> is set. Just mark approved.
  2. Build prompt from character descriptor + background preset (see references/character-descriptors.md + references/background-presets.md).
  3. Generate 3 takes in parallel via gpt-image-1/edit-image at medium quality (~$0.24 total).
  4. HUMAN GATE: operator picks one. Copy chosen to generated/character-lock/anchor-portrait.png. Write LOCKED.json with chain_strategy: "anchor-only".

S4 — Lip-sync

  1. If <reuse_lipsync_drafts> is true: skip S4 entirely. In S6 the composite uses a concat of prior projects' masters instead. Mark lipsync_method: "reused-draft" in state. Use for cheap layout iteration only.
  2. Sequential, with 3s pauses. Do NOT parallelize. The Veed Fabric storage upload endpoint (rest.fal.ai/storage/auth/token) 403s under high parallel load if the account has churned through many calls recently. Sequential with 3s sleep is reliable.
  3. 7 calls (one per scene segment) at --resolution 720p. ~$0.15/sec, so ~$5 per 30s ad. Wall time ~25 min.
  4. Sub-overlays don't need their own lip-sync clip — the same scene-NN.mp4 covers all sub-reveals; only the overlay PNG swaps in S5.

S5 — Overlays

  1. Write production/overlay-plan.raw.json. One entry per overlay segment:
    • Hook + CTA → kinetic-title-clean with {upper, lower, accent: "upper"|"lower"|"both"}. NO caption field — captions are now burn-in (S7).
    • Tool scenes → tool-spotlight with EITHER samples: [path...] (real images) OR fillText: "SHORT POSTER TEXT" (beige card). NEVER both.
    • Multi-reveal beats (e.g. "look at this... and this... and this") → split into N sub-overlays. First sub has noSfx: false (default — gets the pop), subs 2..N have noSfx: true. Sub timings come from char-level "this" word timestamps in voiceover_timestamps.json.
  2. Inline-encode logo + sample paths as data: URIs via references/inline-overlay-images.py.
  3. Render via node atoms/overlays/render-video-overlay-track/scripts/render-previews.js --plan <path> --out generated/overlays --width 1080 --height 1920 --transparent.
  4. Layout invariants (the whole point of the v03 redesign):
    • Sample/fill zone: top ~6% to ~51% of frame
    • Chip: top ~53% (logo + tool name, on a cream pill)
    • Goose PIP zone (composited later, NOT in overlay PNG): bottom ~58% to ~97%
    • Overlay PNG occupies only the top half — bottom half is reserved for the lip-sync PIP. Verify: open any tool overlay PNG, the bottom 40% must be transparent.

S6 — Composite + promote (GATE 3)

  1. Stitch the 7 lip-sync clips: ffmpeg concat → scale=-2:1920,crop=1080:1920 → talking-head-master.mp4. (Veed returns 768×1152; this step puts it at exact 1080×1920 9:16.)
  2. Cheap-draft path: if <reuse_lipsync_drafts>, concat prior projects' masters and trim to current VO duration (see "Cheap-draft mode" below) — skip step 1.
  3. Run references/composite.py. For each overlay:
    • kinetic-title-clean (hook/CTA): full-frame goose + transparent title PNG.
    • tool-spotlight: blur+dim backdrop (boxblur=30:1, eq=brightness=-0.22:saturation=0.55) + sharp goose PIP at 444×790 centered horizontally, bottom-aligned at y=1100 (PIP center is in the lower-third "safe area" the storyboard reserves) + transparent overlay PNG on top.
  4. Audio: mix VO directly with SFX pops via [1:a]adelay=BOUNDARY|BOUNDARY,volume=0.55[sN]; [0:a][s0][s1]...amix=inputs=N+1,volume=1.2[aout]. SFX skipped for any overlay with noSfx: true. The first scene also skipped (no pop on frame 1).
  5. Concat per-scene composites → mux mixed audio → edits/master-final-v<N>.mp4. Promote by cp v<N>.mp4 master-final.mp4. Never write master-final.mp4 directly.
  6. HUMAN GATE. Operator approves the pre-captions video.

S7 — Captions

  1. Run atoms/captions/burn-in-captions/scripts/caption.py --video edits/master-final-v<N>.mp4 --style frosted-subtle. fal.ai Whisper transcribes, ASS template renders, libass burns.
  2. Always run the brand-spelling patch step afterward via references/patch-captions.py. The patch handles:
    • MarginV (caption Y-position): default 0.18 × 1920 = 345px from bottom. For the talking-goose-creator-video layout, drop this to ~295px (50px lower than default) so captions don't cover the goose's mouth in PIP.
    • Brand-name fixes: each known mishearing (e.g. "Hicksfield" → "Higgsfield"). Add per-project mappings in the patch script's top-level config block.
    • Whisper word splits: compound brand names like "Higgsfield" get tokenized as "Higgs" + "Field" by Whisper. The patch script merges them back (both same-line and cross-cue cases) with summed karaoke kf timings to preserve the fill effect.
  3. Re-burn the patched ASS with ffmpeg -i precaption.mp4 -vf "ass=patched.ass" -c:v libx264 -crf 20 -c:a copy patched.mp4.
  4. Always keep both master-final-v<N>-precaption.mp4 AND the .captions/ sidecar dir. Lets you re-render captions with different style/wording without re-rendering the composite.

S8 — Deliver (GATE 4)

  1. Optional: molecules/platform-variants/* for 1:1 / 16:9 reformats.
  2. HUMAN GATE. Operator confirms ship.
  3. Record final cost in .control-plane/project-state.json spendUsd.

Layout reference (the v03 redesign)

9:16 frame · 1080 × 1920
┌──────────────────────────────────┐
│                                   │ ← top 5% breathing room
│   ┌──────────────────────────┐   │
│   │                          │   │
│   │  SAMPLE IMAGE            │   │ ← ts-sample: top:120 height:860 (45%)
│   │  or                      │   │   beige card if no sample
│   │  3-IMAGE GRID            │   │
│   │  or                      │   │
│   │  beige fillText card     │   │
│   │  (#F4DEB0 bg, #5A3D1F)   │   │
│   │                          │   │
│   └──────────────────────────┘   │
│  ┌───────────────────────────┐   │
│  │ [logo] TOOL NAME          │   │ ← ts-chip: top:1010 (53%)
│  └───────────────────────────┘   │
│                                   │
│            ┌──────────┐           │
│            │          │           │
│            │  GOOSE   │           │ ← PIP composited at video stitch:
│            │   PIP    │           │   x=318, y=1100, 444×790 (9:16)
│            │ (lipsync)│           │   centered, white border, drop shadow
│            │          │           │
│            └──────────┘           │
│                                   │ ← bottom 2% breathing room
│  ┌─────────────────────────┐     │
│  │ word-level caption pill │     │ ← burn-in captions (S7)
│  └─────────────────────────┘     │   MarginV=295 (lower than default)
└──────────────────────────────────┘

Cheap-draft mode

For fast layout iteration without burning $5 + 25 min on Veed Fabric every run:

  1. Concat one or more prior projects' talking-head-master.mp4 files using ffmpeg's concat demuxer.
  2. Trim to current VO duration: -t <vo_duration>.
  3. Drop the inherited audio (-an) — we'll add our own VO + SFX in S6.
  4. Save as edits/talking-head-master.mp4 and run S6 composite normally.

Result: goose's mouth moves throughout (mouth shapes match the OLD VO, not the new one) but the layout/overlays/captions are real. Good enough for review and storyboard. Re-run S4 properly for the final.

Cost saved: ~$5/run. Time saved: ~25 min.

Output

<output_dir>/
├── idea-brief.md
├── design-brief.md
├── script/
│   └── script-locked.md
├── raw-materials/
│   ├── reference/                # character anchor source still
│   ├── logos/                    # transparent PNGs per tool
│   ├── samples/                  # one hero image per tool (or N for multi-reveal beats)
│   └── SOURCES.md
├── audio/
│   ├── voiceover.mp3
│   ├── voiceover_timestamps.json # char-level from ElevenLabs
│   ├── scene_timing.json         # per-scene start/end + tip1SubBoundaries
│   ├── sfx-pop.mp3               # single pop, mixed at every tip boundary
│   └── segments/
│       ├── scene-01.mp3 .. scene-07.mp3
│       └── plan.json
├── generated/
│   ├── anchor-options/           # all takes from S3 (kept for traceability)
│   ├── character-lock/
│   │   ├── anchor-portrait.png
│   │   └── LOCKED.json
│   └── overlays/
│       └── scene-NN-*.png        # one transparent overlay per overlay segment
├── clips/
│   └── scene-NN.mp4              # per-scene lip-synced 720p (or reused-draft)
├── edits/
│   ├── talking-head-master.mp4   # 7 lip-sync clips stitched, 1080x1920
│   ├── segments/                 # per-scene composited
│   ├── master-final-v<N>.mp4     # versioned masters
│   ├── master-final-v<N>-precaption.mp4  # before burn-in-captions (re-burnable)
│   ├── master-final-v<N>-precaption.captions/
│   │   ├── words.json            # Whisper word-level
│   │   ├── captions.{srt,ass}    # raw sidecars
│   │   └── *__frosted-subtle__{ass,fixed}.ass  # rendered + brand-patched
│   └── master-final.mp4          # promoted current
├── production/
│   ├── scene-contract.json
│   ├── overlay-plan.raw.json     # file paths (editable)
│   ├── overlay-plan.json         # data: URIs (renderer-ready)
│   ├── audio-plan.json
│   └── asset-manifest.json
└── .control-plane/
    └── project-state.json

Quality Checks

  • All 9 states either ran or were explicitly skipped (with reason in state file).
  • Every clips/scene-NN.mp4 duration is within ±10% of its audio segment (if <reuse_lipsync_drafts> is false).
  • generated/character-lock/LOCKED.json exists.
  • Overlay PNGs: bottom 40% of every TOOL spotlight PNG is transparent (PIP zone uncovered).
  • edits/master-final.mp4 exists, is 1080×1920, h264 + AAC, duration matches audio/voiceover.mp3 ±0.5s.
  • Captions don't cover the goose's mouth in PIP — MarginV ≥ 275px from bottom.
  • Every known brand name in the script appears spelled correctly in the burned-in captions (visual review).
  • The .captions/ sidecar dir is preserved.

Failure Modes

Symptom Likely cause Fix
Veed Fabric 403 on storage auth Too many parallel calls recently Drop to sequential with sleep 3 between calls (see S4 step 2)
FAL "User is locked. Exhausted balance" Account credits depleted Top up at fal.ai/dashboard/billing. The error is clear in the response body.
Whisper hears brand name wrong (e.g. Hicksfield) Compound name with uncommon morphology Add to patch-captions.py REPLACEMENTS dict; brand-name patching is now a default S7 sub-step
Whisper splits brand into 2 words ("Higgs" + "Field") Same compound-name issue patch-captions.py handles both same-line and cross-cue merges with combined karaoke kf timings
Captions cover the goose's mouth Default MarginV=345 puts captions at ~72% Y; goose PIP head is at ~70% Y in this layout patch-captions.py drops MarginV to 295 by default for this molecule
Sub-overlay reveal lands BEFORE the word Slice timing didn't include lead-in Add 0.15s lead-in past each "this" word: cuts.append(this_end + 0.15)
Tool name wrapping ugly in chip Multi-word tool name Shorten to one word + · separator (e.g. "Higgsfield · Marketing Studio" → just "Higgsfield" with marketing in eyebrow)
Caption shows "Higgs" alone at end of cue Whisper tokenization split across cue boundary The cross-line merge in patch-captions.py handles this — verify it's enabled
Cheap-draft master is shorter than VO v01+v02 concat = 56s; trim to VO duration Already handled — -t <vo_duration> flag. If your VO is longer than the sum of prior masters, add more prior projects to the concat.
Overlay PNG covers the goose PIP New tool-spotlight component edited away the 40% bottom transparency Verify CSS: .ts-sample height ≤ 860px out of 1920 (~45%); .ts-chip top ≤ 1080
Character drift on every GPT-image take Descriptor block missing distinctive shape/proportion words Add specifics: "tall and slender, NOT chubby"; "long droopy tube beak, NOT triangular bird beak"; "round white felt eyes SEPARATED, NOT touching"
Composite errors on sub-overlay scenes Scene segment in PLAN["overlays"] has noSfx AND the script doesn't skip it correctly Verify composite.py build_audio_with_sfx continues on overlay.get("noSfx")

Cost reference (per typical 30s run, real lip-sync)

Line item Provider Unit Count Cost
ElevenLabs VO (eleven_v3 w/ timestamps) ElevenLabs per char ~500 ~$0.06
ElevenLabs SFX pop (sound-generation) ElevenLabs per request 1 ~$0.05
Character anchor takes (skipped if <reuse_anchor>) FAL gpt-image-1/edit-image medium $0.08/img 3 ~$0.24 (or $0 if reused)
Veed Fabric lip-sync (720p, sequential) FAL veed/fabric-1.0 $0.15/sec 30s ~$4.50
Burn-in-captions Whisper transcription fal.ai Whisper per minute 0.5 min ~$0.005
Playwright overlay rendering local — 10 $0
ffmpeg composite + caption burn local — — $0
Total (with reused anchor) ~$4.65
Total (fresh anchor) ~$4.90
Cheap-draft mode (no lipsync) ~$0.15

When to escalate

  • 3 review iterations completed and operator still flagging P0s: escalate to user with punch list.
  • Character identity drift > 30% on 6+ takes: surface Soul ID training option.
  • Voice ID isn't on VOICES.md AND operator hasn't approved: stop at S0 and ask.
  • FAL balance exhausted mid-run: surface immediately with top-up link.

Decision Rules

  • Hard cap practical beats at 6 tips + hook + CTA = 8 total scenes. More beats and the VO crowds.
  • Prefer gpt-image-1/edit-image for the character anchor on felt/textured mascots; only fall back to Nano Banana for cartoon/illustrated characters.
  • Reuse a previously-locked anchor whenever the same mascot is being used (<reuse_anchor> skips S3 entirely — saves ~$0.24 + 30s).
  • Run Veed Fabric lip-sync calls sequentially with a 3s sleep between calls — parallel calls 403 from the storage auth endpoint.
  • Always pair burn-in-captions with a brand-spelling patch step; Whisper mishears compound brand names (e.g. Higgsfield → "Hicksfield" / "Higgs Field").
  • Tool-spotlight overlay PNG must keep the bottom 40% transparent so the goose PIP is uncovered. Verify before composite.
  • Use <reuse_lipsync_drafts> for cheap layout iteration only — never ship a draft where mouth motion doesn't match the new VO.

Reference runs

  • <your-project>/goose-talking-head-01/ — first build of the format. Old-style overlay (cream card with eyebrow + tagline + sample inside). No SFX, no burn-in captions.
  • <your-project>/goose-talking-head-02/ — reused v01 anchor. Same old overlay style. First validation that anchor inheritance saves time.
  • <your-project>/goose-talking-head-03/ — the v03 redesign. Clean two-zone layout, beige fillText cards, sequential multi-reveal, SFX pop on tip transitions, burn-in word-level captions with brand-spelling patches. This is the canonical reference for new runs — see the per-project HOW_TO.md.