Build complete AI behavior: intentional input, provider, validated response, observable result, and owned cleanup.
1. Write the AI contract
Name the input, provider branch, response modality, application consumer, data sent off-device, user disclosure, concurrency rule, and visible waiting, unavailable, empty, disconnected, and error states. Treat prompts, microphone, camera, screenshots, and scene context as separate disclosures.
Complete this step when success changes scene, UI, speech, or an asset and each disclosed input is necessary for that result.
2. Prove the provider path
Read ../../docs/docs/manual/AI.mdx and
references/current-api.md. Verify planned symbols
in ../../src/xrblocks.ts,
../../src/ai/AI.ts, and the selected provider. For Live,
tools, camera, or context grounding, also read
references/live-tools-grounding.md.
Use ../../templates/06_ai_query/ for bounded
queries and
../../templates/07_ai_live_assistant/
for the common Live loop.
Complete this step when every symbol is public and the selected provider supports every required operation and modality.
3. Select one primary branch
- Use
xb.ai.query({prompt})for a bounded provider-neutral text request. - Use Gemini multipart input when a bounded request needs an image or typed parts.
- Use Gemini Live for an ongoing audio or video conversation.
- Use
xb.ai.generate()when the image result becomes an actual app asset. - Use a narrow
xb.Toolwhen the model requests a validated application action.
Complete this step when latency, modality, provider support, data disclosure, and resource ownership match the contract.
4. Implement the complete behavior
Configure and enable the selected provider before initialization. Check
availability at use time, catch provider failures, and prevent unintended
concurrent starts. Accept missing text, null, rejected requests,
interruption, and disconnection as normal observable states.
For Live, register callbacks before connecting, make readiness depend on
onopen, use provider-typed nested realtime inputs, and route remote close,
local stop, error, and disposal through one idempotent cleanup owner. Stop the
Live session, microphone, playback, frame timers, and app-owned tracks.
Validate tool arguments again at execution and resolve targets through an allowlist or current scene context. Keep consequential actions behind application authorization or confirmation.
Complete this step when one intentional input reaches its consumer, duplicate starts are controlled, every failure is visible, and every acquired resource has one cleanup owner.
5. Prove and hand off provider behavior
Build or type-check the app and run startup smoke that does not require a long conversation. Check configuration, availability guards, request and response types, visible states, tool validation, disclosures, and cleanup. Use existing safe prototype credentials only for one narrow connectivity check.
Give the user the trigger, required credentials and permissions, data sent off-device, waiting/success/error states, and recovery steps. For Live, include open, interruption, remote close, local stop, and restart. For tools, include valid, invalid, unknown, denied, and failed outcomes.
Finish when the user can evaluate real provider behavior without discovering setup, disclosure, expected states, or cleanup from source.