cochlea

Cochlea is a headless, deterministic audio engine designed for AI agents. The entire workflow is: compose → render → probe → verify, with no need for an audio device, human ears, or ffmpeg.

Purpose

Cochlea enables agents to write scores as data, render them offline into deterministic PCM audio, and then "listen" to the audio through numerical features (loudness, onsets, pitch, tonality, spectrograms, etc.) and make assertions. It solves the problem that agents cannot directly process raw audio samples: a 7-second 48kHz 32-bit float PCM clip is 2.7 MB, while a 3-minute clip of the same kind is about 66 MB, making it unsuitable to hand directly to an agent. What Cochlea provides is a few KB of JSON reports and a small spectrogram.

Main capabilities

• Composition: Scores are pure data, containing tracks, notes, per-parameter automation, tempo maps, and optional master sections. It supports RON serialization (version: 1) and a Rust builder. Position representation such as bar(3).beat(2), and durations use exact fractions such as Dur::quarter()、"3/16".
• Rendering: Audio is rendered in 64-sample blocks, split at event boundaries, with note timing accurate at the sample level. It outputs deterministic PCM, which can be rendered as WAV or FLAC.
• Probing:cochlea probe generates JSON reports, including loudness (integrated LUFS, momentary max, true peak, LRA), onsets, YIN pitch, quantized melody, MFCC timbre summaries, chroma and tonality, tempo and rhythm (reported separately), stereo width and correlation, silence, tails, clipping, structural boundaries, etc.
• Spectrograms:cochlea spectro renders mel spectrogram PNGs, can overlay beat grids, onsets, and pitch annotations, supports difference heatmaps, and can use --sheet to tile long tracks into a single contact sheet.
• Assertions and verification: Scores have embedded verify: assertions,cochlea render score.ron --verify runs assertions, and on failure exits with a non-zero status and outputs a machine-readable JSON failure report.
• Difference comparison:cochlea diff compares two files in feature space, reports timbre, loudness, and pitch differences, and can generate a signed difference heatmap.--tier2 can turn difference decisions into gates (exit 0 if byte-identical or Tier-2 equivalent, otherwise exit 1), suitable for CI or agents to catch regressions.

Usage

Command line

• Render from a RON score:cochlea render score.ron --out mix.wav
• Probe to JSON:cochlea probe mix.wav --json report.json
• Generate a spectrogram:cochlea spectro mix.wav --out spec.png
• Verify assertions:cochlea render score.ron --verify
• Probe any WAV, FLAC (bit-exact decoding), mp3, or ogg without needing a score.
• All reading tools support --from/--to for frame-accurate window analysis.
• probe --digest prints a deterministic text summary, limited to about 40 lines.

MCP server

cochlea-mcp is a stdio-based MCP server exposing 12 tools:render_score、probe_audio、spectrogram、lint_score、probe_digest、loudness_timeline、beat_grid、audio_diff、import_midi、export_midi、transcribe_audio、score_reference. Each tool is a thin wrapper around the matching library call. Features include:

• Self-teaching:score_reference returns the full composition reference, including RON syntax, the live preset catalog, all automatable parameters, assertions, and runnable examples.
• Direct display:spectrogram returns images via inline MCP image content (base64 PNG, size-limited), supporting annotations for beats, onsets, and pitch.
• Scalable:probe_audio and spectrogram accept from_s/to_s, analyzing only one segment of a long file.
• Scoped:cochlea-mcp --root DIR rejects any read or write resolving outside DIR (including symlinks and ..).

Installation

All nine crates are published on crates.io. They can also be installed from source:git clone https://github.com/richer-richard/cochlea && cd cochlea && cargo install --path crates/cli.

Use cases

• AI agents automatically compose, render, and verify audio works.
• Automated audio quality checks and regression testing, without manual listening review.
• Catch unexpected changes in audio rendering in CI pipelines.
• Perform feature analysis on existing audio files, without a score.
• Provide audio processing capabilities to agents through the MCP protocol.

Limitations

• Audio is "folded" rather than "mapped", and filters and delays carry state, so sample-by-sample purity is not the contract; determinism is divided into three tiers, and Tier 1 requires specific compilation and runtime choices.
• WAV and FLAC decoding is bit-exact; mp3 and ogg are only used for analysis input, and are reproducible for a given build, but cannot claim to be completely identical to the original samples.
• By design, it does not depend on ffmpeg, with no subprocess calls, no system codecs, no GPU, and no audio device.
• Writing a score at a position that is not on the tick grid is an error and will not be silently rounded.
• Prices, fees, platforms, etc. are not provided in the materials.