magentic

Magentic is an open-source Python library designed to seamlessly integrate large language models (LLMs) into Python code. By using the @prompt and @chatprompt decorators, developers can create functions that return structured output from an LLM and combine LLM queries and tool use with traditional Python code to build complex agent systems.

Main capabilities:

• Structured output: Supports pydantic models and built-in Python types.
• Streaming: Can stream structured output and function calls, usable during generation.
• LLM-assisted retries: Improves LLM adherence to complex output schemas.
• Observability: Supports OpenTelemetry and natively integrates with Pydantic Logfire.
• Type annotations: Works well with linters and IDEs.
• Multi-LLM provider configuration: Includes OpenAI, Anthropic, Ollama, and more.
• Other features: Chat prompts, parallel function calling, vision, formatting, Asyncio, etc.

Usage:

1. Set the OPENAI_API_KEY environment variable to configure the OpenAI API key; for other LLM providers, refer to the configuration instructions.
2. Use the @prompt decorator to define an LLM prompt template as a Python function. When the function is called, the arguments are inserted into the template, the prompt is sent to the LLM, and the function output is generated. The decorator follows the function's return type annotation (supports any type supported by pydantic).
3. The @chatprompt decorator is similar to @prompt, but allows passing chat messages as a template (such as a system message or few-shot prompt). Format fields represented by curly braces {example} are filled in all messages (except FunctionResultMessage).
4. The LLM can decide to call functions. In this case, a function decorated with @prompt returns a FunctionCall object, which can be called to execute the function using the arguments provided by the LLM.
5. When the LLM needs to call one or more functions to generate a final answer, the @prompt_chain decorator automatically resolves FunctionCall objects and passes the output back to the LLM until a final answer is obtained.
6. Using @prompt, @chatprompt and @prompt_chain to create LLM-driven functions that can be passed as arguments to other @prompt / @prompt_chain decorators, just like ordinary Python functions, thereby building more complex LLM-driven functionality.
7. Use the StreamedStr (or AsyncStreamedStr) class to stream LLM output, allowing processing as text is generated, and also allowing multiple StreamedStr instances to be created to concurrently stream LLM output.
8. Use the return type annotation Iterable (or AsyncIterable) to stream structured output, with each item processable as the next one is generated.
9. Use async functions/coroutines to query the LLM concurrently, which can significantly improve generation speed and allow other async code to run while waiting for LLM output.

Additional features:

• The functions parameter of @prompt can include async/coroutine functions; when calling the corresponding FunctionCall object, the result must be awaited.
• Use Annotated type annotations to provide descriptions and other metadata for function parameters (refer to the pydantic documentation).
• The @prompt and @prompt_chain decorators accept a model parameter, allowing an OpenaiChatModel instance to be passed to use GPT4 or configure a different temperature.
• Other types can be registered as return type annotations for @prompt functions (refer to the Pandas DataFrame example notebook).

Backend/LLM configuration: Magentic supports multiple LLM providers (backends), referring to the Python packages used to interact with LLM APIs. Supported backends:

• OpenAI (default backend): Uses the openai Python package, supports all features, requires no additional installation, and imports the OpenaiChatModel class directly.
• Ollama: Supports the OpenAI-compatible API, allowing Ollama models to be used through the OpenAI backend. First install ollama from ollama.com, pull the model, then specify the model name and base_url when creating an OpenaiChatModel instance.
• When using the openai backend, set MAGENTIC_OPENAI_BASE_URL environment variable or use OpenaiChatModel(..., base_url="http://localhost:8080")，即可搭配任何 OpenAI-compatible APIs (such as Azure OpenAI Service, LiteLLM OpenAI Proxy Server, LocalAI). Note: if the API does not support tool calling, prompt functions that return Python objects cannot be created, but other features are still available.
• Azure: when using the openai backend, you need to set the MAGENTIC_OPENAI_API_TYPE environment variable to "azure" or use OpenaiChatModel(..., api_type="azure"), and set the environment variables required by the openai package to access Azure.
• Anthropic: uses the anthropic Python package and supports all features. When installing the magentic package, add the anthropic extra, or directly install the anthropic package, then import the AnthropicChatModel class.
• LiteLLM: uses the litellm Python package and can query LLMs from multiple providers. Note: some models may not support all magentic features (such as function calling/structured output and streaming). When installing the magentic package, add the litellm extra, or directly install the litellm package, then import the LitellmChatModel class.
• Mistral: uses the openai Python package with minor modifications for compatibility with the Mistral API, and supports all features. However, tool calling (including structured output) is not streamed, but received in one go. Future versions may switch to the mistral Python package. No additional installation is required; directly import the MistralChatModel class.

Configuring the backend: The default ChatModel can be configured according to the following priority:

1. A ChatModel instance passed as the model parameter to the magentic decorator.
2. The current chat model context created using with MyChatModel:.
3. A global ChatModel created from environment variables and the default settings in src/magentic/settings.py. The environment variables that can be set are described in the documentation.

Handling by type checkers: Because functions decorated with @prompt have no functionbody or return value, many type checkers will issue warnings or errors. How to handle:

• Disable checking globally, for example disable the empty-body error code in mypy: # pyproject.toml [tool.mypy] disable_error_code = ["empty-body"]
• Make the function body ... (does not satisfy mypy) or raise. @prompt("Choose a color") def random_color() -> str: ...
• Use the comment # type: ignore[empty-body] on each function; in this case a docstring can be added instead of .... @prompt("Choose a color") def random_color() -> str: # type: ignore[empty-body] """Returns a random color."""

Use cases: Magentic is suitable for developers who need to integrate LLMs into Python applications, especially scenarios involving building agent systems, requiring structured output, streaming, function calling, or support for multiple LLM providers. Limitations:

• Some backends (such as certain models in LiteLLM, or OpenAI-compatible APIs that do not support tool calling) may not support all features of magentic.
• Tool calling in the Mistral backend (including structured output) does not support streaming.
• When using the @prompt decorator, type checkers may report errors, which need to be handled as described above.