create-voiceover-elevenlabs

create-voiceover-elevenlabs

Purpose

Create ElevenLabs voiceover audio with timing data, pronunciation checks, and expressive delivery control. Default to Eleven v3 plus audio-tag prompting unless the job explicitly needs lower-latency or stricter stability.

📌 READ FIRST: skills/orchestrators/higgsfield/vo-naturalness-learnings.md — the authoritative naturalness rule set for every TTS call. Settings recipe (stability=0.40, similarity_boost=0.75, style=0.05), script naturalness cues (…, —, ALL-CAPS, contractions, density 2.3–2.6 wps), audio tags syntax, post-processing chain, and the 10 AI tells listening pass. Skipping this is how we shipped the vidiq video-01-apex robotic-VO incident (2026-05-15).

Implementation status: extracted from existing repository skills. The workflow is based on working source material and still requires the standard smoke and human tests after migration.

Sources: voiceover-product-ad, ad-studio, ai-ugc-ad-generator.

Extraction notes: ElevenLabs TTS, timestamps, pronunciation, atempo readiness, Eleven v3 prompting, and audio-tag-based intonation control.

Inputs

• A clear user brief or source asset path.
• Brand, product, audience, platform, and approval constraints when relevant.
• Required credentials or provider access for any external service used by this skill.
• Output directory or test-run directory where artifacts should be saved.

Workflow

1. Read the brief and confirm all required inputs are present.
2. Check the requested voice against the approved voice list when one exists, usually VOICES.md at the repo root.
3. Prefer ElevenLabs timestamped TTS for video work so the edit can align scene cuts to phrase timing.
4. Default model choice: use eleven_v3 for most ad/video voiceovers because it is the most expressive model and supports audio tags for emotion, pacing, and delivery direction. Only fall back to eleven_multilingual_v2 / turbo / flash when the user explicitly wants a flatter read, lower latency, or maximum consistency.
5. Default prompting style for v3: add light-touch audio tags and punctuation shaping when the read needs more intonation. ElevenLabs v3 responds to:
   • emotional tags like [excited], [curious], [reassuring], [understated]
   • delivery tags like [pause], [deliberate], [rushed], [drawn out], [emphasized]
   • reaction tags like [sighs], [laughs], [breathes]
   • punctuation and capitalization for rhythm and emphasis
6. Use direct ElevenLabs HTTP or an approved provider wrapper. For direct HTTP, call /v1/text-to-speech/<VOICE_ID>/with-timestamps with the approved voice ID, model, and voice settings.
7. For v3 reads, prefer Natural or Creative behavior by lowering stability enough that tags can work. Very high stability can flatten the performance.
8. Save audio, raw timestamp JSON, and script text under the requested output folder.
9. Use ffprobe to verify audio duration and write phrase timing notes from returned character timestamps.
10. Write or update a manifest.json for executable runs with status, provider, outputs, warnings, and errors.

Timestamped direct API shape — default v3:

curl -sS -X POST \
  "https://api.elevenlabs.io/v1/text-to-speech/$VOICE_ID/with-timestamps?output_format=mp3_44100_128" \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -cn --arg text "$VO_TEXT" '{
    text: $text,
    model_id: "eleven_v3",
    voice_settings: {
      stability: 0.34,
      similarity_boost: 0.80,
      style: 0.15,
      use_speaker_boost: true
    }
  }')"

Example v3-style script shaping:

[curious] If your collagen still looks like this... you need to watch this.
[understated] But honestly... that is not even why I stayed on it.
[emphasized] This is so easy because it's literally one tablespoon a day.
[reassuring] Take the quiz and you can try the full-size product before you buy.

Prompting guidance:

• Use tags sparingly. Over-tagging can make the read theatrical.
• Match the tags to the chosen voice. A warm conversational voice responds better to subtle cues than extreme character tags.
• For creator-style UGC, aim for curious, warm, reassuring, understated, deliberate, and occasional emphasized.
• Use ellipses for soft pauses and thoughtfulness.
• Use all-caps very sparingly for emphasis.

Fallback shape for flatter / more stable reads:

curl -sS -X POST \
  "https://api.elevenlabs.io/v1/text-to-speech/$VOICE_ID/with-timestamps?output_format=mp3_44100_128" \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -cn --arg text "$VO_TEXT" '{
    text: $text,
    model_id: "eleven_multilingual_v2",
    voice_settings: {
      stability: 0.48,
      similarity_boost: 0.82,
      style: 0.32,
      use_speaker_boost: true
    }
  }')"

Decode audio_base64 to voiceover.mp3, save the full response as voiceover_timestamps.json, and save the exact script as voiceover_script.txt.

Known limitations

character_start_times is unreliable on eleven_v3 scripts with audio tags

For eleven_v3 reads that include audio tags ([curious], [pause], [emphasized], etc.), the returned character_start_times_seconds payload is unreliable: tag characters get interpolated garbage timestamps, and adjacent spoken characters can cluster at the same value. Observed example (from 11-walking-felt-goose, run 2026-05-24): "Now this is" at char 390 and "ffmpeg" at char 281 both reported the same timestamp 16.71s — clearly invalid.

Do NOT use this payload for caption sync, SFX retiming, or overlay placement on tagged scripts.

Instead: run Whisper on the rendered audio (fal-ai/whisper with chunk_level: "word") to get accurate per-word timestamps. This is what atoms/captions/burn-in-captions already does. The Whisper round-trip costs ~$0.01 and lands within ±50ms on most words.

Raw render duration is non-deterministic (5-15% run-to-run variance)

eleven_v3 is non-deterministic on pacing — pause length, breath duration, syllable speed all vary run-to-run even with stability=0.40. Same script can come back at 40.75s / 49.16s / 50.83s across three renders. Do not target a VO duration and then build a video to match it — VO is a moving target. Render the video first to a fixed duration (e.g. via Seedance clip durations + a freeze-frame tail), then atempo the VO with atempo = vo_raw_duration / video_duration.

Output

• Primary artifact or written plan requested by the skill.
• Voiceover audio file, normally MP3.
• Timestamp JSON with character timing when provider supports it.
• Exact script text used for generation.
• manifest.json for executable runs.
• verification.md or a short verification summary that names the checks performed.
• Any generated source assets, intermediate files, or final exports in the run folder.

Quality Checks

• Required files exist and paths in the manifest are valid.
• Voice ID is present in the approved voice list or explicitly approved by the user.
• Audio duration is measured and compatible with the target video timeline.
• Timestamp JSON is present for video sync workflows.
• Chosen model is documented in the manifest.
• If using eleven_v3, the script was reviewed for appropriate audio-tag use and not over-tagged.
• Output matches the requested format, platform, duration, dimensions, or text structure.
• Brand claims, captions, on-screen text, and CTAs follow the provided brand rules.
• Provider failures, skipped integrations, and human-review needs are explicit.

Failure Modes

• Missing credentials, provider access, or source files.
• Requested voice is not approved.
• Provider returns audio without timestamps when timestamps are required.
• Eleven v3 read comes back too flat because stability is too high or there are no delivery cues.
• Eleven v3 read comes back too theatrical because tags are too dense or too strong for the voice.
• Output does not match requested dimensions, duration, structure, or brand constraints.
• Generated media contains artifacts, unreadable text, unsafe claims, or caption collisions.
• Scaffolded skills cannot run production workflows until implementation details are added.