audio-static-debugging

Debugging Embedded Audio Static / Streaming-PCM Noise

Static on an I2S speaker (MAX98357A, PCM510x, etc.) has several distinct causes that sound similar to the human reporting them, and the wrong ones are seductive because they're about hardware — the amp, the clock, the ground. This skill exists because in the robocar-unified voice bring-up (2026-07) two hardware theories were wrong (uninitialised DMA content; BCLK stop/start pops) and cost several reflash cycles each. The fix only came from measuring the actual samples at each stage. Measure first; theorise last.

The one law

You cannot hear the audio; the user can, and the samples can. Do not reason about the amp/clock/DMA from the symptom word "static." Get the PCM bytes and compute statistics on them — that localises the corruption to a pipeline stage in one pass, with no hardware and no guessing.

Step 1 — get a precise symptom from the user (it discriminates the cause)

"Terrible static" underdetermines everything. Ask which:

Symptom	Most likely stage
Static before/after the line, silence-adjacent	Amp idle / clock-gating / floating DIN (analog)
Speech intelligible but with static bursts, or alternating clean↔noise	Sample misalignment in the transport (see Step 3)
Speech fully garbled, no words	Format mismatch: endianness, bit width, or sample rate
Faint constant hiss under clean speech	Amp noise floor / gain / power rail

Step 2 — prove whether the source data is clean (before blaming the pipeline)

Capture the PCM the device receives (e.g. the base64 inlineData from a TTS response, or a dump from the decode buffer) and characterise it. Clean speech and byte-swapped/garbage have unmistakably different statistics:

python3 - <<'PY'
import json, base64, struct, math
raw = base64.b64decode(json.load(open("resp.json"))
        ["candidates"][0]["content"]["parts"][0]["inlineData"]["data"])
n = len(raw)//2
def stats(fmt, label):
    s = struct.unpack(("<" if fmt=="LE" else ">")+"%dh"%n, raw[:n*2])
    rms = math.sqrt(sum(x*x for x in s)/n)
    big = sum(1 for x in s if abs(x) > 16384)/n            # fraction past half-scale
    zc  = sum(1 for i in range(1,n) if (s[i-1]>=0)!=(s[i]>=0))/(n-1)  # zero-cross rate
    print(f"{label}: RMS={rms:7.0f}  >half-scale={big*100:4.1f}%  zero-cross={zc*100:4.1f}%")
stats("LE","little-endian"); stats("BE","big-endian")
PY

• Clean speech: low zero-cross rate (~5–15%), few samples past half-scale (~1–3%), moderate RMS. The interpretation that yields this is the correct one.
• Byte-swapped / garbage: zero-cross ~40–50% (near-random), ~40% past half-scale, high RMS.

Real finding: Gemini TTS advertises audio/L16 (nominally big-endian per RFC) but actually returns little-endian — the LE stats were clean, BE was noise. Verify endianness from the data before swapping bytes; a "fix" based on the label would have broken working audio. If the source is clean, the corruption is downstream — go to Step 3.

Step 3 — the streaming-PCM alignment trap (the robocar root cause)

16-bit samples must never be split across a transport boundary. The classic break, and the one that produced "speech + terrible static, alternating":

• A FreeRTOS RINGBUF_TYPE_BYTEBUF (byte-granular) carries the PCM.
• The producer feeds it odd-length chunks — a base64 decoder emits 3-byte groups per quartet, and 3 is odd.
• The consumer reads got bytes and does samples = got / sizeof(int16_t), then returns the whole item — so an odd got drops the straggler byte.
• That drop shifts every subsequent sample by one byte. The shift is cumulative: after an even number of drops the stream is re-aligned (clean), after an odd number it's misaligned (static) — hence the audio oscillates between clean speech and loud static, which is the tell.

Fix — keep the ring 16-bit aligned end to end. Hold back an odd trailing byte in the writer and prepend it to the next write, so only even-length runs enter the ring (splice the carried byte + the first new byte into one 2-byte sample, send the even remainder, carry any new odd tail). Reset the carry at utterance end/abort so a half-sample can't bleed into the next utterance. With even writes into an even-sized buffer and even-sized reads, alignment holds through wraps. See packages/robocar/unified/main/audio_player.c (audio_player_write) for the reference implementation.

Related sizing gotcha found the same day: the speech text buffer (SPEECH_TEXT_MAX) must fit the rendered sentence — Finnish words are long, so a 25-word line ran ~250 chars and a 160-char buffer truncated both text and audio mid-word. Size for the language.

Step 3b — starvation: the network can be slower than real time (2026-07)

Distinct from misalignment, and the cause of the second robocar voice bug. A streamed source can produce audio more slowly than the DAC consumes it, and the resulting underrun does not sound like a gap — it sounds like broadband noise. Two facts, both verified against ESP-IDF v5.4 source:

• Measure the real-time factor before theorising. Across seven live Gemini TTS captures the RTF (audio-ms produced per wallclock-ms after first byte) was 0.55 / 0.79 / 0.84 / 1.88 / 2.70 / 3.08 / 3.81 — three of seven below 1.0. The slow ones were all the short lines: a fixed 0.9–2.7 s time-to-first-byte amortised over less audio. Log an rtf= figure per utterance; below 1.00 is normal and must be absorbed, not prevented.
• Underrun tears, it does not merely gap. i2s_common.c creates msg_queue with depth desc_num - 1 and, when it is full, the TX ISR drops the oldest entry (xQueueIsQueueFullFromISR → xQueueReceiveFromISR into a dummy) before pushing the finished descriptor. After a writer stall longer than desc_num-1 descriptor periods, i2s_channel_write() is handed the oldest free descriptor — the very next one the DMA will transmit — so its memcpy lands inside a buffer being read out. auto_clear zeroing the buffer after send does not save you; that is why the symptom is noise rather than silence.

Fix shape: gate playback on a preroll — start when N ms are banked OR the fetch completes, whichever comes first. The OR is the load-bearing half: a short utterance (the class that starves) then plays from a complete buffer and cannot underrun at all. Make the gate a property of the ring, not of an utterance, or back-to-back speech inherits an already-open gate and skips it. Widen dma_desc_num for margin, and batch the producer's writes — a decoder emitting 3 bytes per call forces a portYIELD_WITHIN_API() to the higher-priority player on every ring send (~25 000/s), starving the producer exactly when it needs to get ahead.

Step 4 — only now consider the analog side

If the source is clean AND the transport is aligned AND it still hisses/pops in the silence around speech, it's the amp: the MAX98357A has no shutdown GPIO on this board (budget full), so it amplifies continuously; gating BCLK / letting DIN float between utterances pops. Options, in order: drive continuous zeros (keep the channel enabled, feed silence when idle — trades pop for a possible faint hiss), or gate the amp's SD pin via a freed/expander GPIO. Judge by ear — this is the one part only the user can settle.

Rationale

Every wrong turn in this bug was a hardware theory reasoned from the word "static"; every correct step was a measurement of the bytes. The statistics test in Step 2 costs one python3 run and tells you which stage is lying. This is diagnose-at-the-failure-point for audio: the samples at each stage are the authoritative source, above any theory about the amp.