Goal
Turn a ready script into a finished podcast episode that meets broadcast audio standards (Apple Podcasts / Spotify). The user should be able to upload the MP3 directly to a podcast host.
Core Principle
Every audio decision serves clarity and listener comfort. Voices must be distinct, pacing must feel natural, loudness must be consistent, and transitions must be smooth. Production quality is not decoration — it's the difference between "AI-generated audio" and "a podcast."
Requires Rebyte API auth — $AUTH_TOKEN and $API_URL are set up per the agent's system prompt; use them as Bearer token and base URL.
Part 1: Pipeline Overview
Script text
| Parse (check directive markers FIRST, then speaker markers)
v
Segment list [{ speaker, text, directives }]
| Assign voices (Part 2)
v
Segment list with voice assignments
| Preflight: probe TTS with tiny text to verify model availability
| Chunk segments > 3500 chars (Part 3)
v
Chunk list
| Synthesize via relay TTS API (sequential, with retry)
| Decode base64 -> WAV, normalize to canonical format
v
Raw chunk WAVs ($WORKDIR/chunks/)
| Reassemble same-speaker chunks (concat)
| Per-segment processing: highpass -> compressor -> limiter
v
Processed segments ($WORKDIR/segments/)
| Generate silence files, download music + ambience ($WORKDIR/assets/)
v
All assets in canonical WAV (44100 Hz, stereo, s16)
| Assemble speech + silences (ffmpeg concat demuxer)
v
all_speech.wav
| Mix background ambience under speech (volume + amix)
v
episode_with_ambience.wav
| Prepend intro music, append outro music (concat)
v
episode_raw.wav
| Final mastering: two-pass loudnorm (-16 LUFS)
v
episode_mastered.wav
| Encode to MP3 (192kbps)
v
podcast-episode-<slug>.mp3
| Upload to Artifact Store
v
Done
Key invariant: All intermediate audio is WAV (PCM, 44100 Hz, stereo, s16). MP3 encoding happens once at the very end. This avoids generation-loss and format mismatch issues.
Setup
Before starting, verify tools and create working directory:
for cmd in ffmpeg curl jq base64 python3; do
command -v "$cmd" >/dev/null || { echo "FATAL: $cmd not found"; exit 1; }
done
WORKDIR=$(mktemp -d /tmp/podcast-XXXXXX)
mkdir -p "$WORKDIR/chunks" "$WORKDIR/segments" "$WORKDIR/assets"
On success: delete $WORKDIR after upload. On failure: preserve $WORKDIR and report its path.
Part 2: Voice Selection & Configuration
Model
Use gpt-4o-mini-tts. It produces the most natural prosody.
Preflight check: Before synthesizing any real segments, make one probe call with a short text (e.g., "Hello") to verify the model is available. If it fails, switch to tts-1-hd and remap voices BEFORE any real synthesis begins. Never mix models within an episode.
Voice Palette
| Voice | Character | Best For |
|---|---|---|
| marin | Female, warm, confident | Primary host, narration (default) |
| cedar | Male, calm, authoritative | Co-host, expert segments |
| ash | Male, energetic, youthful | News, upbeat topics |
| coral | Female, clear, professional | Interviews, corporate |
| ballad | Male, storytelling, expressive | Narrative, dramatic |
| sage | Female, wise, measured | Educational, explainer |
Voice Pairing Rules
Select voices that contrast in register and timbre:
| Format | Recommended Pairing | Why |
|---|---|---|
| Two-host discussion | marin + cedar | Warm female + calm male — distinct but harmonious |
| Interview | coral + cedar | Professional interviewer + authoritative guest |
| Debate | ash + sage | Energetic vs. measured — creates tension |
| News roundup | marin + ash | Confident anchor + energetic reporter |
Rule: Never pair two voices of the same character. The listener must instantly distinguish who is speaking.
Voice Assignment by Speaker Count
Voices are assigned to speakers in order of first appearance:
| Speakers | Assignment |
|---|---|
| 1 | marin |
| 2 | marin, cedar (or template pairing) |
| 3 | marin, cedar, ash |
| 4 | marin, cedar, ash, coral |
| 5 | marin, cedar, ash, coral, sage |
| 6+ | Cycle: marin, cedar, ash, coral, ballad, sage. Warn if reusing. |
Fallback Remap (gpt-4o-mini-tts -> tts-1-hd)
If falling back to tts-1-hd, remap ALL voices before any synthesis:
| Original | Fallback | Rationale |
|---|---|---|
| marin | nova | Both female, warm |
| cedar | onyx | Both male, authoritative |
| ash | echo | Both male, conversational |
| coral | nova | Both female, professional |
| ballad | fable | Both expressive |
| sage | shimmer | Both female, calm |
Part 3: Script Parsing & Chunking
Script Grammar
Directive markers are checked FIRST (take precedence over speaker markers):
Directive: ^\[(INTRO|OUTRO|TRANSITION|SEGMENT|SILENCE)[^\]]*\]\s*$
Speaker: ^\[([A-Za-z0-9 _-]+)\]\s*$
If a line matches a directive pattern, it is a directive — never a speaker. This prevents [INTRO] from being treated as a speaker named "INTRO".
Parser Rules
- Check directive regex first. If match -> store as structural cue.
- Check speaker regex. If match -> start new segment.
- All other non-empty lines -> speech text for current speaker.
- Empty lines within a speaker block -> paragraph breaks (0.3s silence).
- Text before any speaker marker -> default narrator.
- No speaker markers at all -> entire text is single narrator, Solo Narration template.
- Empty speaker blocks -> skip (no audio).
Chunking Long Segments
TTS limit: 4096 characters. Split segments > 3500 chars using this priority:
- Paragraph boundary (
\n\n) - Sentence boundary — regex
(?<=[.!?])\s+(?=[A-Z])— avoids splitting on "Dr. Smith", "3.14", ellipses - Clause boundary —
;or, - Hard split — last space before 3500 chars
Concatenate same-speaker chunks with no gap (same voice, continuous thought).
Part 4: TTS Synthesis
API Call
Always send all five fields explicitly. Never rely on relay defaults.
RESPONSE=$(curl -s -X POST "$API_URL/api/data/tts/synthesize" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"text\": \"$CHUNK_TEXT\",
\"voice\": \"$VOICE\",
\"model\": \"$MODEL\",
\"format\": \"wav\",
\"speed\": $SPEED
}")
Request WAV format to avoid lossy MP3 round-trip. If relay rejects WAV, fall back to flac, then mp3.
Decode Response
# Validate
SUCCESS=$(echo "$RESPONSE" | jq -r '.success')
AUDIO=$(echo "$RESPONSE" | jq -r '.audio.base64 // empty')
if [ "$SUCCESS" != "true" ] || [ -z "$AUDIO" ]; then
# Handle error / retry
fi
# Save and normalize to canonical format
echo "$AUDIO" | base64 -d > "$WORKDIR/chunks/chunk_${N}_raw.wav"
ffmpeg -i "$WORKDIR/chunks/chunk_${N}_raw.wav" -ar 44100 -ac 2 -sample_fmt s16 "$WORKDIR/chunks/chunk_${N}.wav"
Fallback State Machine
1. Call with (model=gpt-4o-mini-tts, format=wav)
2. success=true AND audio.base64 non-empty -> save, done
3. success=false:
a. text_too_long -> re-split, retry smaller
b. rate_limit -> wait 5s, retry (max 3)
c. authentication_error -> re-auth, retry once
d. invalid_model -> switch ALL to tts-1-hd + remap voices, retry
e. invalid_format -> try flac, then mp3 (global switch)
f. other error -> retry once, then FATAL
4. success=true but audio.base64 empty -> retry once
Model and format fallbacks apply globally to ALL subsequent calls.
Rate Limiting
| Rule | Value |
|---|---|
| Concurrency | 1 (sequential) |
| Delay between calls | 0.5s |
| Retry on 429 | Wait 5s, up to 3 retries |
| Per-call timeout | 60 seconds |
| Total TTS phase timeout | 15 minutes |
| Max TTS calls | 30 |
| Max script length | Warning at 15k chars, fatal at 25k chars |
Part 5: Audio Processing
Per-Segment Processing Chain
Apply to each assembled segment (after same-speaker chunk concatenation):
ffmpeg -i "$WORKDIR/segments/segment_${N}_raw.wav" \
-af "highpass=f=80,acompressor=threshold=-20dB:ratio=3:attack=5:release=100,alimiter=limit=-1.0" \
"$WORKDIR/segments/segment_${N}.wav"
| Filter | Purpose |
|---|---|
highpass=f=80 |
Remove rumble and TTS artifacts below 80Hz |
acompressor |
Even out dynamics — threshold -20dB, ratio 3:1 |
alimiter |
Hard ceiling at -1.0 dBTP — no clipping |
Do NOT apply loudnorm per-segment. It causes pumping at boundaries.
Silence Generation
Pre-generate needed silence durations:
ffmpeg -f lavfi -i anullsrc=r=44100:cl=stereo -t 0.6 -sample_fmt s16 "$WORKDIR/assets/silence_0_6s.wav"
| Transition | Silence | Rationale |
|---|---|---|
| Same speaker, paragraph break | 0.3s | Brief breath |
| Different speaker responds | 0.6s | Natural turn-taking |
| New segment/topic | 1.2s | Signals topic shift |
| After intro music | 0.8s | Let music breathe |
| Before outro music | 1.0s | Signal wrap-up |
Part 6: Music, Ambience & Sound Design
Professional podcasts use two layers of audio beyond speech: music (intro/outro/transitions) and background ambience (atmospheric sounds that set the mood throughout). Both dramatically improve production quality.
Two Layers
| Layer | Purpose | Examples | Volume |
|---|---|---|---|
| Music | Brand identity, structure | Intro jingle, transition stings, outro | Full volume when solo, -15dB under speech |
| Ambience | Mood, atmosphere, warmth | Ocean waves, rain, coffee shop, forest, lo-fi hum | 0.2-0.3x speech volume (subtle, never distracting) |
Getting Audio: Download Free Sounds
Do NOT generate sine-wave tones. Download real music and ambience from free sources instead. These are royalty-free and sound infinitely better:
| Source | URL | License | Best For |
|---|---|---|---|
| Pixabay Music | pixabay.com/music/ | CC0 (no attribution) | Podcast intro/outro music, ambient tracks |
| Pixabay Sound Effects | pixabay.com/sound-effects/ | CC0 (no attribution) | Ocean waves, rain, birds, city, coffee shop |
| Freesound | freesound.org | CC0 / CC-BY (varies) | Huge library of ambience and effects |
| Mixkit | mixkit.co/free-sound-effects/ | Free license | Clean sound effects and music |
Download approach: Use curl or wget to download audio. Convert to canonical WAV format:
# Download ocean waves from Pixabay (example — find real URL by searching the site)
curl -L -o "$WORKDIR/assets/ambience_raw.mp3" "<url>"
ffmpeg -i "$WORKDIR/assets/ambience_raw.mp3" -ar 44100 -ac 2 -sample_fmt s16 "$WORKDIR/assets/ambience.wav"
Choose ambience that matches the episode tone:
| Episode Tone | Suggested Ambience |
|---|---|
| Calm, reflective | Ocean waves, gentle rain, wind |
| Energetic, news | None or subtle city hum |
| Educational | Soft lo-fi background, library ambience |
| Storytelling | Forest, fireplace, subtle music bed |
| Professional/corporate | None or very subtle office ambience |
Music Positions
| Position | Purpose | Duration |
|---|---|---|
| Intro | Brand identity, energy setter | 5-10s |
| Transitions | Signal topic change | 2-4s |
| Outro | Sign-off, wrap-up | 5-10s |
Mixing Background Audio Under Speech
This is the most important sound design technique. Use volume + amix — simple and reliable:
# Mix narration (full volume) with background ambience (0.25 volume, fade out at end)
DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$WORKDIR/segments/all_speech.wav" | cut -d. -f1)
ffmpeg -i "$WORKDIR/segments/all_speech.wav" -i "$WORKDIR/assets/ambience.wav" \
-filter_complex "[0:a]volume=1.0[speech];[1:a]volume=0.25,afade=t=out:st=$((DURATION-5)):d=5[bg];[speech][bg]amix=inputs=2:duration=shortest" \
-ac 2 -ar 44100 "$WORKDIR/episode_with_ambience.wav"
Key parameters:
- Speech volume: 1.0 (never reduce speech)
- Ambience volume: 0.2-0.3 (subtle background, not competing with speech)
- Fade out ambience near end of speech (5s fade)
duration=shortestensures output matches speech length
This is the same technique Manus and other production tools use. It's simpler and more reliable than sidechaincompress.
When to Mix vs. Concat
| Scenario | Technique |
|---|---|
| Intro/outro music (plays alone, then speech starts) | Concat — sequential, no overlap |
| Background ambience under entire episode | Mix — volume + amix |
| Music bed under intro speech | Mix — music at 0.2-0.3 volume |
| Transition stings between segments | Concat — too short to mix |
Assembly Order
When using background ambience, the pipeline changes slightly:
- Assemble all speech segments + silences into one file via concat (no music/ambience yet)
- Mix background ambience under the assembled speech using
amix - Prepend intro music and append outro music via concat
- Apply final mastering
Fade Rules
Apply fades to music/ambience BEFORE mixing or assembly:
| Element | Fade In | Fade Out |
|---|---|---|
| Intro music | 1.0s | 2.0s |
| Transition sting | 0.3s | 0.3s |
| Outro music | 2.0s | 3.0s |
| Background ambience | 2.0s | 5.0s |
Degradation Policy
Music and ambience are additive polish, not load-bearing:
| Scenario | Action |
|---|---|
| Download fails (network issues) | Skip ambience/music, produce speech-only episode, warn user |
| User-provided audio corrupt | Warn, try downloading from free source instead |
| No suitable ambience for tone | Skip ambience — not every episode needs it |
Part 7: Episode Assembly & Mastering
Step 1: Concat Speech + Silences
All files must be canonical WAV. Use alphanumeric filenames only (no spaces/quotes):
file 'segments/segment_001.wav'
file 'assets/silence_0_6s.wav'
file 'segments/segment_002.wav'
file 'assets/silence_1_2s.wav'
file 'segments/segment_003.wav'
ffmpeg -f concat -safe 0 -i "$WORKDIR/speech_list.txt" -c copy "$WORKDIR/all_speech.wav"
Step 2: Mix Background Ambience (if available)
DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$WORKDIR/all_speech.wav" | cut -d. -f1)
ffmpeg -i "$WORKDIR/all_speech.wav" -i "$WORKDIR/assets/ambience.wav" \
-filter_complex "[0:a]volume=1.0[speech];[1:a]volume=0.25,afade=t=out:st=$((DURATION-5)):d=5[bg];[speech][bg]amix=inputs=2:duration=shortest" \
-ac 2 -ar 44100 -sample_fmt s16 "$WORKDIR/episode_with_ambience.wav"
If no ambience, just copy: cp all_speech.wav episode_with_ambience.wav
Step 3: Prepend Intro + Append Outro
file 'assets/intro.wav'
file 'assets/silence_0_8s.wav'
file 'episode_with_ambience.wav'
file 'assets/silence_1_0s.wav'
file 'assets/outro.wav'
ffmpeg -f concat -safe 0 -i "$WORKDIR/episode_list.txt" -c copy "$WORKDIR/episode_raw.wav"
Two-Pass Loudnorm Mastering
Pass 1: Measure
LOUDNORM_STATS=$(ffmpeg -i "$WORKDIR/episode_raw.wav" \
-af "loudnorm=I=-16:TP=-1.0:LRA=7:print_format=json" \
-f null /dev/null 2>&1 | grep -A 20 '"input_i"' | head -20)
Extract values with jq or grep:
INPUT_I=$(echo "$LOUDNORM_STATS" | grep '"input_i"' | grep -o '[-0-9.]*')
INPUT_TP=$(echo "$LOUDNORM_STATS" | grep '"input_tp"' | grep -o '[-0-9.]*')
INPUT_LRA=$(echo "$LOUDNORM_STATS" | grep '"input_lra"' | grep -o '[-0-9.]*')
INPUT_THRESH=$(echo "$LOUDNORM_STATS" | grep '"input_thresh"' | grep -o '[-0-9.]*')
TARGET_OFFSET=$(echo "$LOUDNORM_STATS" | grep '"target_offset"' | grep -o '[-0-9.]*')
Pass 2: Apply
ffmpeg -i "$WORKDIR/episode_raw.wav" \
-af "loudnorm=I=-16:TP=-1.0:LRA=7:measured_I=${INPUT_I}:measured_TP=${INPUT_TP}:measured_LRA=${INPUT_LRA}:measured_thresh=${INPUT_THRESH}:offset=${TARGET_OFFSET}:linear=true" \
"$WORKDIR/episode_mastered.wav"
Final Encode
ffmpeg -i "$WORKDIR/episode_mastered.wav" -codec:a libmp3lame -b:a 192k -ar 44100 "podcast-episode-${SLUG}.mp3"
Part 8: Episode Structure Templates
Solo Narration
[INTRO MUSIC: 6s] [SILENCE: 0.8s]
[HOST] Intro (30-60s)
[SILENCE: 1.2s]
[HOST] Segment 1 [TRANSITION: 3s]
[HOST] Segment 2 [TRANSITION: 3s]
[HOST] Segment 3
[SILENCE: 1.2s]
[HOST] Outro (30-60s)
[SILENCE: 1.0s] [OUTRO MUSIC: 8s]
Voice: marin | Speed: 0.93
Two-Host Discussion
Voice: marin + cedar | Speed: 0.95 | Speaker gaps: 0.6s
Interview
Voice: coral + cedar | Speed: 0.93 | Speaker gaps: 0.8s
News Roundup
Voice: marin or ash | Speed: 1.0 | Transition stings between stories
Part 9: Decision Logic
Precedence (highest first)
- User explicit instruction
- Script markers (e.g.,
[HOST A: voice=ash]) - Template defaults
- Auto-detection from script patterns
- Skill defaults
Auto-Detect Format
| Script Pattern | Template |
|---|---|
| Single speaker | Solo Narration |
| Two speakers, balanced turns | Two-Host Discussion |
| One asks questions, other answers | Interview |
| Single speaker, many short sections | News Roundup |
Part 10: Delivering Output
Upload the final MP3 to the Artifact Store. Report:
| Field | Value |
|---|---|
| Title | From script or user |
| Duration | From final MP3 (ffprobe -show_entries format=duration) |
| Format | MP3, 192kbps, 44100 Hz, stereo |
| Loudness | -16 LUFS (±1) |
| Voices | Speaker -> voice mapping |
| Segments | Count + timestamps (from cumulative WAV durations) |
Offer adjustments: swap voices, change speed, re-generate a segment, change music.
Part 11: Error Handling
Fail early, fail loudly. TTS and audio assembly errors are fatal. Music errors degrade gracefully.
| Scenario | Action |
|---|---|
| TTS fails after retries | FATAL — report segment text + error |
| Model unavailable | Preflight catches this. Fallback to tts-1-hd. Both fail -> FATAL |
| ffmpeg missing | FATAL — detected in setup |
| ffmpeg filter/concat fails | FATAL — report step + error |
| Script > 25k chars | FATAL — ask user to split |
| Script > 15k chars | WARNING — suggest splitting |
| No speaker markers | Treat as solo narrator |
| Music corrupt | Warn, use generated pad |
| Auth expired | Re-auth, retry once |
On fatal failure, preserve $WORKDIR and report its path for inspection.
What This Skill Does NOT Do
- Content creation: No topic research, no script writing
- Transcription: No speech-to-text
- Video: Audio only
- Distribution: No RSS feeds, no podcast host uploads
- Live streaming: Offline generation only