agenticmail

AgenticMail is a communications infrastructure platform for AI agents, with the goal of giving AI agents real email addresses and phone numbers, and programmatically completing the sending and receiving of emails, text messages, and verification codes, as well as agent-driven outbound voice calls. The project is released as an npm package, and the repository root provides AGENTS.md as a complete runbook for agents to autonomously complete installation and configuration.

[Main Capabilities]

1. Email and SMS: Agents can send and receive real emails and text messages, and obtain verification codes.
2. Telephony and voice: Supports outbound call control and real-time voice conversations. The telephony transport provider can be 46elks or Twilio, selected during phone setup; Twilio and 46elks are functionally equivalent, including outbound call control and real-time voice. Twilio outbound calling is implemented through the Calls.json REST API, TwiML webhooks, and status callback cost tracking; inbound webhooks are verified using the X-Twilio-Signature header (HMAC-SHA1, timing-safe, fail-closed). Real-time voice is bridged through Twilio Media Streams and OpenAI Realtime; Twilio audio is G.711 µ-law @ 8 kHz, and the OpenAI session uses audio/pcmu, so no transcoding is required.
3. Real-time voice tool calling: The voice agent can call tools during a call. The OpenAI Realtime session declares session.tools, and RealtimeVoiceBridge dispatches the model's function calls through an injected ToolExecutor and returns function_call_output, keeping the phone line alive during slow tools through a safety-net timeout and an in-flight call limit.
4. ask_operator human intervention: During a real-time call, the agent records an operator query, notifies the operator (email by default, channel-agnostic), polls for up to about 5 minutes, and continues with the answer. If the caller hangs up while the query is pending, the task is marked as a dropped-call callback, and after the operator replies, the system redials with a continuity task.
5. Lookup tools: web_search (keyless DuckDuckGo, with results fenced as untrusted content), recall_memory (the agent's general memory), get_datetime, and operator query AP scoped by agent keyI endpoint.
6. Telegram channel: After a user registers a Telegram bot token and links a chat, they can send messages to the AgenticMail agent via Telegram and receive replies. Inbound webhooks are authenticated using constant-time secret token comparison, and also carry ask_operator notifications and approvals.
7. General memory: Each agent has persistent, evolving memory, organized by category, with confidence decay, BM25F search support, and survives across all conversations. AgentMemoryManager provides CRUD, text recall, 9 memory categories, importance levels, confidence decay for unaccessed entries, access tracking, pruning, and generateMemoryContext() to render memories sorted into a markdown block for prompt injection. The memory API includes /memory (set/list/search/get/delete), /memory/reflect, /memory/context, /memory/stats; each endpoint is scoped to the authenticated agent, and an agent can only read and write its own memory. Deleting an agent clears its agent_memory rows.
8. Multi-agent collaboration: call_agent achieves 53x faster response times compared to traditional sub-agent spawning. The layered wake context system (Layer 1 ThreadCache shares the envelopes and previews of the most recent 10 messages; Layer 2 AgentMemory is a markdown file written by each agent at the end of a wake) reduces multi-agent thread cost from growing linearly with thread length to roughly flat. Waking defaults changed from "all Cc recipients" to "recipients only"; Cc'd local agents still receive the email but do not trigger a Claude turn unless explicitly named in wake; wake: 'all' can be used to restore the old behavior. Wake coalescing: multiple wake events for the same (agent, thread) within 30 seconds are merged into a single Claude turn, configurable via wakeCoalesceMs.
9. Task contract: call_agent / POST /tasks/assign accepts outputSchema (a JSON Schema draft-7 subset), submit_result validates against it, and a mismatch returns 400 with validator errors.
10. Mediatoolset (unreleased): A local, opt-in media/video editing toolset containing nine media tools: media_tts / media_tts_voices (Edge text-to-speech), media_image_edit, media_video_edit, media_audio_edit, media_info, media_video_understand, media_voice_clone, media_capabilities. Provided as MCP media_* tools and OpenClaw agenticmail_media_* tools, all thin clients over the /media/* API routes on the core MediaManager. Supports cinematic video editing: color grading presets, crossfade/wipe transitions, timed text overlays, picture-in-picture, split screen, Ken Burns, frame-interpolated slow motion, watermarks, concatenation, audio mixing, and whisper.cpp-powered automatic subtitles. The underlying binaries (ffmpeg, ffprobe, ImageMagick, whisper.cpp, Python) are not bundled; each tool detects the required binaries and returns actionable installation hints when missing, and the server does not crash. All binaries are invoked via execFile with argument arrays, with no shell and no string interpolation; untrusted input paths are validated; numeric options are clamped; each invocation has bounded timeouts and output buffering; output files land only within the configured media directory.
11. Visibility and operations: check_activity shows dispatcher heartbeat status (alive/unhealthy/missing, uptimeMs, channels, coalesceQueueSize, etc.); the skipped-wake ring buffer records every filtering decision and its reason, showing the most recent 100; per-agent wake_on_cc: false flag; the Web UI displays To/Cc/Bcc as separate labeled lines; docs/wake-patterns.md documents all wake forms and 5 recommended patterns.
12. Workers and autonomous mode: Workers can run for hours, with each round logged to ~/.agenticmail/worker-logs/<id>.log, a heartbeat every 30 seconds, and an independent cwd; the MCP tool tail_worker can read running worker logs in real time. The email hook is also registered on the Stop event; long-running headless Claude Code runs return decision: 'block' at round boundaries when they see teammate replies, forcing continuation. compact-and-continue synthesizes a breadcrumb checkpoint when context overflows and loops (limit 4 times).

[How to use] The project is published as an npm package. Installation and setup can be completed via the runbook in AGENTS.md at the repository root, supporting a zero-prompt, approximately 2-minute, fully autonomous approach. MCP tools and OpenClaw tool integration are provided. Real-time voice requires setting OPENAI_API_KEY (environment variable or config.json). The telephony transport provider is selected during phone setup as either 46elks or Twilio.

[Applicable scenarios] Scenarios where an AI agent needs a real email address and phone number, programmatically sending and receiving emails/SMS/verification codes, initiating outbound voice calls, performing multi-agent collaboration and persistent memory management; it can also be used for real-time call flows requiring human intervention (ask_operator).

[Limitations and notes] The media toolset has not yet been released. The underlying media binaries are not bundled and must be installed separately. The real-time Twilio ↔ OpenAI call path and real-time Telegram delivery still require operator smoke testing before npm release. The real-time voice end-to-end path requires a valid OPENAI_API_KEY and a configured 46elks websocket number; the bridging logic, memory injection, and WebSocket upgrade/authentication glue have been unit tested with mock sockets, but real-time calls must be smoke tested by an operator. Version 0.9.54 has 1038 passing tests, with a fully green build.