ag2-multimodal-input

Multimodal inputs

When to use

The user wants the agent to process non-text input: an image to describe, audio to transcribe, video to summarise, or a PDF / document to extract from. The same factory pattern works across providers; per-provider support varies.

60-second recipe

from ag2 import Agent
from ag2.config import GeminiConfig
from ag2.events import ImageInput

agent = Agent(
    "vision",
    "You describe images.",
    config=GeminiConfig(model="gemini-3-flash-preview"),
)

image = ImageInput("https://example.com/photo.jpg") reply = await agent.ask("Describe this image in detail.", image) print(reply.body)


Multiple inputs in one ask are fine:

```python
reply = await agent.ask(
    "Compare these two images.",
    ImageInput("https://example.com/before.jpg"),
    ImageInput("https://example.com/after.jpg"),
)

Input factories

Factory	Formats
`ImageInput(...)`	JPEG, PNG, GIF, WebP
`AudioInput(...)`	WAV, MP3, OGG, FLAC, AAC
`VideoInput(...)`	MP4, WebM, MOV, MKV, MPEG
`DocumentInput(...)`	PDF, TXT, HTML, Markdown, CSV, JSON, Office formats

Each accepts the same four data sources:

from ag2.events import ImageInput

ImageInput("https://example.com/photo.jpg")     # URL
ImageInput(path="photo.jpg")                    # local file
ImageInput(data=raw_bytes, media_type="image/png")  # bytes
ImageInput(file_id="file-abc123")               # provider-uploaded

Provider matrix

Input type	OpenAI	OpenAI Responses	Gemini	Anthropic
Text	✓	✓	✓	✓
Image (URL)	✓	✓	✓	✓
Image (binary)	✓	✓	✓	✓
Audio (URL)	–	–	✓	–
Audio (binary)	✓	–	✓	–
Video (URL)	–	–	✓	–
Video (binary)	–	–	✓	–
Document (URL)	–	✓	✓	✓
Document (binary)	✓	✓	✓	✓
File ID	✓	✓	✓	✓

Unsupported combinations raise UnsupportedInputError with a clear message.

Gemini has the broadest multimodal support. If you don't know which provider to pick for a multimodal task, start there.

Provider-specific niceties

Gemini — YouTube URLs work directly

from ag2.events import VideoInput

video = VideoInput("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
reply = await agent.ask("Summarize this video.", video)

Gemini — large files (> 20MB) via Google Files API

from google import genai
from ag2.events import VideoInput
import time

client = genai.Client()
uploaded = client.files.upload(file="large_video.mp4")
while uploaded.state.name == "PROCESSING":
    time.sleep(2)
    uploaded = client.files.get(name=uploaded.name)

video = VideoInput(uploaded.uri)

Gemini — `vendor_metadata`

Key	Purpose
`media_resolution`	`MEDIA_RESOLUTION_LOW/MEDIUM/HIGH/ULTRA_HIGH` — token vs cost
`video_metadata`	Clipping (`start_offset`, `end_offset`) and `fps`
`display_name`	Display name for the file

ImageInput(data=raw, media_type="image/jpeg", vendor_metadata={"media_resolution": "MEDIA_RESOLUTION_LOW"})

VideoInput(path="lecture.mp4", vendor_metadata={
    "video_metadata": {"start_offset": "60s", "end_offset": "120s", "fps": 0.5},
})

OpenAI — image detail

ImageInput(data=raw, media_type="image/png", vendor_metadata={"detail": "low"})  # "low" | "high" | "auto"

Anthropic — File ID + prompt caching

import anthropic
from ag2.events import ImageInput, DocumentInput

client = anthropic.Anthropic()
uploaded = client.beta.files.upload(file=("photo.jpg", open("photo.jpg", "rb"), "image/jpeg"))

# filename determines block type (image vs document)
image = ImageInput(file_id=uploaded.id, filename="photo.jpg")

# Cache an attachment so subsequent turns skip re-uploading
doc = DocumentInput(path="report.pdf", vendor_metadata={"cache_control": {"type": "ephemeral"}})

`FilesAPI` — upload lifecycle, provider-agnostic

For any provider that has a file API (OpenAIConfig, OpenAIResponsesConfig, AnthropicConfig, GeminiConfig):

from ag2 import FilesAPI
from ag2.config import OpenAIResponsesConfig

files = FilesAPI(OpenAIResponsesConfig(model="gpt-5-mini"))

uploaded = await files.upload(path="report.pdf", purpose="assistants")
print(uploaded.file_id)

# Or from bytes (filename required)
uploaded = await files.upload(data=b"...", filename="hello.txt", purpose="assistants")

# List, read, delete
all_files = await files.list()
data = await files.read(uploaded.file_id)        # NotImplementedError on Gemini
await files.delete(uploaded.file_id)

Pass the file_id to DocumentInput, ImageInput, etc.:

from ag2.events import DocumentInput

doc = DocumentInput(file_id=uploaded.file_id)
reply = await agent.ask("Summarize this report.", doc)

Going deeper

website/docs/user-guide/multimodal/inputs.mdx — full provider matrix and vendor_metadata reference.
website/docs/user-guide/advanced/files.mdx — FilesAPI reference (upload / list / read / delete).
For tools that return images / binary back to the LLM, see ag2-add-custom-tool (ImageInput, BinaryInput, ToolResult).

Common pitfalls

Picking a provider that doesn't support your input type — silently you'll get UnsupportedInputError. Check the matrix; Gemini is broadest.
FilesAPI.read() on Gemini — raises NotImplementedError. Gemini doesn't expose download.
Calling files.upload(data=...) without filename= — raises ValueError. Filename is required for in-memory uploads.
Supplying more than one source to a factory — not an error. The factory resolves in priority order url > file_id > path > data, so extra sources are silently ignored. Pass exactly one to get what you intend. Supplying zero sources raises ValueError.
Anthropic ImageInput(file_id=...) without filename= — Anthropic decides block type (image vs document) by filename extension. Pass it.
Gemini vendor_metadata keys are nested — video_metadata itself takes a dict. Check the doc table for shape.
Forgetting to wait for Gemini file processing — large uploads have a PROCESSING state. Poll client.files.get(name=...) until ready before referencing the URI.