SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsaudio-and-haptics-feedback

audio-and-haptics-feedback

General & other

View repository

Updated 10-18 18:04

About this skill

Audio and haptics feedback

Every sound an Expo app plays as feedback is synthesised from a literal recipe and played through one ambient AudioContext behind AudioPort; every pulse is a cue from a fixed table played through HapticsPort on iOS and Android, gated by the user's preference and throttled. This skill builds the ports, the cue tables, the banks, and the two checkers that prove them.

Rules that must hold

  1. One engine each, behind ports. Only src/services/audio/audio-api-audio-adapter.ts imports react-native-audio-api and only src/services/haptics/expo-haptics-adapter.ts imports expo-haptics; nothing imports react-native-sound or react-native-haptic-feedback, the feedback layer imports no media player, and Vibration never appears in the feedback layer or a view (elsewhere only with // vibration-exempt: <reason>). Tests use createFakeAudio()/createFakeHaptics(). Why: one file changes on a bump, two engines fight over the session, and tests run without native code.
  2. Pin react-native-audio-api exactly and override all five plugin defaults (iosBackgroundMode: false, androidPermissions: [], androidForegroundService: false, disableFFmpeg: true, disableStaticExternalLibs: true) through AUDIO_API_PLUGIN in app.config.ts; a new version goes through "Before any bump" and is recorded in contracts/audio-feedback.json. Why: the library is pre-1.0, and the defaults add background audio, Android foreground-service permissions and build-time downloads.
  3. One AudioContext, created in the adapter after setAudioSessionOptions({ iosCategory: 'ambient' }), and no audio focus on Android (observeAudioInterruptions(true) only under Platform.OS === 'ios'). Why: ambient respects the silent switch and mixes with the user's music; on Android that call requests AUDIOFOCUS_GAIN, which fades out and mutes the user's music.
  4. Every sound is a literal SoundRecipe that passes recipeProblems (finite, peak 0.05-0.9, first sample < 0.05, last < 0.01, later layers fade in over at least 1 ms, 10-1500 ms, music up to 30 s); the app's synth renders byte-identically to the checker's port. No streaming and no URL audio in the feedback layer; every recorded audio file in the repo is listed with its licence and source in the contract (bundledSounds for feedback, otherAudio for a notification sound or a feature's media). Why: no clicks, no clipping, no licences to chase, previews that are exactly what the device plays.
  5. Banks and ids: kebab-case nouns for what happened; ui. is reserved for UI_SOUNDS; each feature or game bank lives in <owner>/sounds/<owner>-sounds.ts as <OWNER>_SOUNDS, is tested, and is composed once in the root layout with composeSoundBank (which throws on a clash). Every cue names an id of its owner's bank and every sfx id is cued. Why: a typo is silence in production, a clash silently replaces another sound.
  6. Keep the voice policy: three category gains (sfx and ui follow sound effects, music follows music), volume², 50 ms ramps, at most 8 overlapping voices, the same sound at most once per 30 ms (or its minIntervalMs, at most 1000), voices as intervals on the audio clock, released by cancelPending. Why: even loudness, no clicks, no pile-ups, no sound lost after a fast action.
  7. Music is off by default, never activelyReclaimSession, and every music control is hidden when hasMusic(bank) is false. Why: the library cannot tell whether the user's own audio plays, so the app never plays over it unasked.
  8. Feedback fires from events on JS: a feature's cue table played by its model (playCue), a canvas track cue scheduled on the audio clock, a UI moment (playUiFeedback), or drained sim events; never from a view's own handler, a sim, a 'worklet' module or per frame, and each event in one place only; views (src/ui, a feature's screen or components) never call audio.play, playCue or playUiFeedback. Why: sync with the animation, and the UI thread never touches a port.
  9. Mount useAudioLifecycle once, in the root layout; the context runs only while the app is active and nothing full-screen plays; iOS interruptions suspend it and resume it only if the lifecycle still wants it running (with a direction gate, mount it and load the bank behind the gate); await audio.dispose() before every JS reload, or await an injected beforeReload hook that the caller wires to () => services.audio.dispose(), or say why with // audio-dispose-exempt: <reason>. Why: no sound in the background or over a video, and the native engine closes before the bundle reloads.
  10. Haptics follow the cue table on both platforms (iOS feedback generators; Android performAndroidHapticsAsync constants, an API 30 table and a legacy one), only while the user's preference is on, at least 40 ms apart, each call ending in .catch(() => undefined) inside the adapter; isSupported is false on iPad. Views use only selection (switches, pickers). Why: bursts feel like a buzz, a newer Android constant rejects on older phones, and buttons should not buzz.
  11. The app-wide moments go through UI_FEEDBACK (by default toggle, success, error; the table is the app's), each moment it keeps played from a model; buttons are silent by default, and an app that adds a tap moment renders <PressFeedbackProvider playsSound onPress={...}> that plays it (the canonical press surface then sets android_disableSound itself; no synced file is edited). Why: one forgotten handler is a silent switch, and Android would play its own click on top of the tap.
  12. Add no microphone purpose string up front; only if an App Store upload reports ITMS-90683, set iosMicrophonePermission (localised, with a comment naming ITMS-90683). Why: feedback never records, and an unused purpose string invites review questions.
  13. A green gate proves a sound is clean, not right. Render the WAV previews, list the listening and the feel on a real iPhone and Android phone for a human, and never block on the answer; test builds prove requests with the recording decorators. Why: an agent cannot hear and simulators have no haptic engine.

Workflow

  1. Pick the job. A new feedback layer: steps 2-6. Sounds for a feature or game: steps 7-9. Tuning a sound: steps 8-9. Bumping the library: step 10. Always finish with step 11.
  2. Read references/audio-engine.md (library, plugin, port, adapter, session and focus, lifecycle, music, wiring, testing) and references/haptics.md (cue table on both platforms, throttle, devices).
  3. Copy the templates at the paths their first line names (tests included): templates/src/services/audio/ (with synth/), templates/src/services/haptics/, templates/src/services/feedback/, templates/src/services/test-hooks/recording-feedback.ts (test builds only), templates/src/config/audio-plugin.ts, templates/__mocks__/react-native-audio-api.ts, templates/contracts/audio-feedback.json (+ schema). src/hooks/use-is-app-active.ts, src/ui/press-feedback-context.tsx and src/core/rng/sfc32.ts belong to other skills; copy them only if the app lacks them. A haptics-only app copies haptics/ alone; the checker then notes that the audio rules do not apply.
  4. Install: npx expo install expo-haptics and npm install --save-exact react-native-audio-api@0.13.6 (dependency-management records both in its policy). Add AUDIO_API_PLUGIN to app.config.ts plugins; check that .gitignore ignores reports/.
  5. Wire the composition root (the lines are in audio-engine.md, "Wiring"): audio and haptics in Services, createDeviceServices and createTestServices; the haptics preference read at call time (settings-and-preferences owns it). In the root layout, services.audio.load(composeSoundBank(...)) in the startup initializer and useAudioLifecycle(...) once; imitate examples/root-layout-wiring.tsx. Make every reload path dispose first.
  6. Wire the moments (references/feedback-moments.md): playUiFeedback(services, 'toggle') in switch handlers (the canonical one: the settings model's onToggled), 'success'/'error' where a requested action ends. Tap sounds only if the app wants them (the opt-in there).
  7. Plan the sounds. Read references/sound-design.md. Write the event → sound → haptic table: most events get nothing, three to six sounds per feature, the most repeated the quietest, loud and long only for rare moments.
  8. Write the bank and the cue table test-first. Copy templates/src/features/__FEATURE__/ (bank, cue table, their tests), replace __FEATURE__, __FEATURE_PASCAL__, __FEATURE_SOUNDS__ and __FEATURE_FEEDBACK__, write one literal recipe per sound, play each row from the model with playCue (or put it on the canvas track), add the bank to composeSoundBank, run npx prettier --write src/features/<feature> on the copied files, and run npx jest src/features/<feature> --ci --selectProjects unit until green (paths before the flag).
  9. Render and hand over the listening: node ${CLAUDE_SKILL_DIR}/scripts/check-sound-banks.mjs . --wav-dir reports/sfx, fix every FAIL, then list the listening as in examples/listening-checklist.md. Carry on; when a human answers, tune one number at a time and re-render.
  10. Bumping react-native-audio-api: follow "Before any bump" in audio-engine.md (release notes, plugin defaults, mock, focus code, adapter tests, a smoke test on a named iOS simulator and Android emulator, a listening check), then set audioApiVersion in contracts/audio-feedback.json with the pin.
  11. Run the checks until both print RESULT: PASS: node ${CLAUDE_SKILL_DIR}/scripts/check-sound-banks.mjs . then node ${CLAUDE_SKILL_DIR}/scripts/check-audio-haptics.mjs .. Each FAIL names the file, the rule and the fix. Then npx jest src/services src/features/<feature> --ci --selectProjects unit android.

Definition of done

  • The audio, haptics and feedback services, the config plugin and the root mock exist as in templates/, and their tests pass in the unit and android Jest projects.
  • react-native-audio-api is pinned to the contract's audioApiVersion (default 0.13.6), expo-haptics has the SDK's ~ range, AUDIO_API_PLUGIN is in app.config.ts, and there is no microphone purpose string without ITMS-90683.
  • Every feature or game with sounds has a tested literal bank composed in the root layout, and a cue table (or track cues) that names only its ids.
  • Music is off by default and its controls are hidden without music; the haptics row is hidden when isSupported is false (settings-and-preferences builds the rows).
  • useAudioLifecycle is mounted once; every reload disposes the engine or carries an exemption reason.
  • Every UI_FEEDBACK moment is played from a model; tap sounds only as the documented opt-in.
  • The WAV previews are rendered and the listening on an iPhone and an Android phone is listed for a human (not blocking).
  • node ${CLAUDE_SKILL_DIR}/scripts/check-sound-banks.mjs . prints RESULT: PASS.
  • node ${CLAUDE_SKILL_DIR}/scripts/check-audio-haptics.mjs . prints RESULT: PASS (a haptics-only app prints the note audio rules not applicable: ...; an app with no feedback layer prints NOT APPLICABLE)

Anti-patterns

  • An AudioContext in a screen or hook. Two contexts fight over the session and double the memory; the adapter owns the only one.
  • AudioManager.observeAudioInterruptions(true) on every platform. On Android it takes audio focus and silences the user's podcast at the first tap sound.
  • audio.play(...) in a view's onPress or a useEffect for an animated event. The sound drifts from the animation and the event plays twice; play the cue table row from the model, or the track cue (the checker's view-feedback rule).
  • Wiring a settings switch to usePressFeedback(). Without the tap opt-in it is a no-op, so the switch is silent; with it, the switch plays tap. The model's onToggled plays toggle.
  • A cue on every particle track, or a pulse per completed item. One visible event, one cue; the voice limit and the throttle would drop the rest at random.
  • Pulsing on every button press. Buttons are silent to the hand; only switches and pickers use selection.
  • impactAsync on Android. It drives the Vibrator directly; performAndroidHapticsAsync follows the system's touch-feedback setting.
  • Shipping a downloaded sound "just this once". It brings a licence, a file the checker rejects and possibly a network path; synthesize a recipe, or list a bundled file in the contract.
  • Turning music on by default to make the app feel livelier. It plays over the user's music; ask the developer, and build the silence-hint check first.
  • Tuning by reading numbers, or blocking a change on the listening. The gate proves clean, not right; render the previews, list them for a human, and carry on.
  • Raising MAX_VOICES, lowering the 40 ms throttle or widening the gate to pass a check. Fix the recipe or the cue; the limits exist because the failures are audible.
  • Importing expo-haptics or react-native-audio-api in a test. jest.mock + jest.requireMock in the adapter test; fakes everywhere else.

Files in this skill

File What it is Read/run when
references/audio-engine.md Versions, the plugin on iOS and Android, ITMS-90683, the one iOS session next to media, port, adapter (session, focus), voice policy, banks, lifecycle and the reload seam, music, wiring, testing, recorded audio (notification, media, bundled), licence rows, bump list, multi-package Workflow steps 2-5 and 10
references/haptics.md Haptics port, the cue table for iOS and Android (with the legacy table), rules of use, throttle, devices and failure modes, tests Workflow step 2, and when choosing a cue
references/sound-design.md Recipe model, quality gate, design starting points, ids and banks, cue tables and track cues, real-time games, WAV previews, the human check, the Tasks example Workflow steps 7-9
references/feedback-moments.md UI_FEEDBACK and the settings model's toggle, the tap opt-in (playsSound, android_disableSound), feature cue tables, settings and hidden rows, recording decorators, where the checkers are heuristic Workflow step 6; when a checker misses or flags a shape
templates/src/services/audio/ Port, adapter (+ unit and android tests: no audio focus on Android), voice policy (+ test), fake (+ test), lifecycle hook (+ test), composeSoundBank/hasMusic (+ test), synth/ (synth, gate, UI_SOUNDS, tests) Step 3 (copy)
templates/src/services/haptics/ Port, shouldPulse (+ test), expo-haptics adapter with the iOS table and the three Android tables (+ unit and android tests), fake (+ test) Step 3 (copy)
templates/src/services/feedback/ playCue, UI_FEEDBACK/playUiFeedback, each with its test Steps 3 and 6
templates/src/services/test-hooks/ recording-feedback.ts: the test-build recording decorators (a test-only folder, reached only through the test-only entry), with its test Step 6
templates/src/features/__FEATURE__/ A feature's bank (+ gate test) and cue table (+ test), filled for the Tasks app's tasks feature Step 8
templates/src/config/audio-plugin.ts AUDIO_API_PLUGIN with every default overridden (+ test) Step 4
templates/src/config/audio-plugin.test.ts Pins the five overrides Step 4
templates/contracts/ audio-feedback.json (verified version, bundled sounds) and its JSON schema Step 3; on a bump or a bundled sound
templates/mocks/react-native-audio-api.ts Root Jest mock completing the library mock's AudioManager (synced; this skill owns the canonical copy) Step 3
templates/src/hooks/ use-is-app-active.ts (+ test): the foreground check (synced from react-components-and-hooks; do not edit here) Step 3, only if missing
templates/src/ui/ press-feedback-context.tsx (+ test): the press-feedback seam (synced from design-system-structure; do not edit here) Step 6, tap opt-in only
templates/src/core/rng/ sfc32.ts (+ test): the seeded PRNG the synth's noise uses (synced from seeded-determinism-and-golden-vectors; do not edit here) Step 3, only if missing
examples/root-layout-wiring.tsx Root layout shape: compose the bank, mount the lifecycle once, dispose before a reload, the crash exemption Step 5
examples/bundled-sound-loading.ts Shape of the opt-in recorded-sound decoding inside the adapter Only when shipping a recorded sound
examples/listening-checklist.md The human listening and feel follow-up for a change report Step 9
scripts/check-audio-haptics.mjs Engines, pins, plugin, mic string, context and focus, file audio, voice and haptic policies (iOS and Android), call sites and views, UI moments, tap opt-in, lifecycle, dispose-before-reload (and the seam), root mock Steps 3 and 11
scripts/check-sound-banks.mjs Renders every sound with the byte-exact synth port, gate, ids, categories, cues against banks, bank tests and wiring, synth drift; --wav-dir writes previews, --owner narrows Steps 9 and 11
scripts/selftest.mjs Proves both checkers pass the templates and catch every planted bug After changing a checker or a template
scripts/lib/synth.mjs Synth, sfc32, gate and WAV port with a pinned golden hash When changing the synth
scripts/lib/ts-literal.mjs Reads TypeScript literal data without a compiler (synced; do not edit here) Never by hand
scripts/lib/app-modules.mjs Loads the app's pure TypeScript for the synth-drift rule (synced; do not edit here) Never by hand
scripts/check-lib.mjs Shared script helper (synced; do not edit here) Never by hand
assets/shared.json The shared files this skill copies in When adding a shared file
assets/template-placeholders.json Placeholder values verify-templates and the self-test fill in When adding a placeholder
tests/ bases/app/ (stand-ins for the app's root layout, models, a game) and fixtures/ (overlay cases for both checkers) When adding a rule

Related skills

  • settings-and-preferences - the sound, music and haptics rows, their defaults, the settings-to-audio glue.
  • skia-canvas-rendering - timeline track cues and the cue scheduler that calls these ports.
  • design-system-structure - the press surface and the press-feedback context.
  • architecture-and-boundaries - ports, adapters and the composition root.
  • dependency-management - installing and pinning, the policy file.
  • error-handling-typed-results - the one licensed best-effort swallow.
  • react-components-and-hooks - useIsAppActive.
  • realtime-game-loop - drained sim events that become sounds and haptics.
  • e2e-maestro, debug-menu-and-test-hooks - asserting recorded feedback in test builds.
  • ios-release-testflight - upload warnings such as ITMS-90683.