llm

LLM is a command-line tool and Python library for interacting with large language models, supporting calling models through remote APIs as well as installing and running models locally. It can connect to OpenAI, Anthropic Claude, Google Gemini, Qwen, Gemma, Kimi, DeepSeek, Mistral, and dozens of other large language models.

Purpose

LLM is aimed at developers and users who want to use large language models directly from the command line or in Python programs, covering both remote API models and locally run models.

Main capabilities

• Run prompts from the command line.
• Store prompts and responses in SQLite.
• Generate and store embeddings.
• Extract structured content from text and images.
• Grant models the ability to execute tools.
• Support JSON output, schemas, fragments, continuing conversations, completion prompts, and more.
• Can start interactive chat via llm chat.
• Support model aliases and default option configuration.
• Support extension through plugins: local models, remote APIs, tools, fragment and template loaders, embedding models, additional commands, and more.
• Provide a Python API supporting basic prompt execution, system prompts, attachments, tools, schemas, fragments, model options, API key passing, plugin models, access to the underlying JSON, token usage, streaming responses, structured messages and streaming events, async models, conversations (including tool use), model lists, and more.
• Support writing logs to SQLite, and browsing logs via Datasette, searching, filtering (by ID, model, fragment, tool), backing up the database, and querying the message store.

Usage

Installation: use pip or Homebrew or pipx or uv. For example, install via pip, or install via Homebrew (see the warning note). You can also use uvx to run LLM without installing it.

Running prompts: you can run prompts or start a chat against any endpoint compatible with OpenAI Chat Completions (such as LM Studio). When using uvx, there is no need to install LLM first. If the endpoint requires authentication, add --key your-api-key. For more options, see "Run against an endpoint without configuring it".

If you already have an OpenAI API key, you can run it directly. For Gemini or Anthropic, you need to use the corresponding plugin. You can also install plugins to access models that can run on local devices, for example using Ollama.

Start an interactive chat: use llm chat.

Plugin installation: installing plugins is supported, listing installed plugins is supported, and running with a subset of plugins is supported. The plugin directory can include local models, remote APIs, tools, fragment and template loaders, embedding models, additional commands, and more. Plugin hooks include register_commands(cli), register_models(register, model_aliases), register_embedding_models(register), register_tools(register), register_template_loaders(register), register_fragment_loaders(register).

Developing model plugins: includes initial structure, installing and trying it out, building a Markov chain, executing the Markov chain, adding it to a plugin, understanding execute(), logging prompts and responses to a database, adding options, distributing the plugin (GitHub repository, publishing to PyPI, adding metadata), and troubleshooting. Advanced model plugins involve lazily loading expensive dependencies, models that accept API keys, asynchronous models, models that do not support conversation, supporting schemas, supporting tools, supporting server-side tools, multimodal model attachments, structured messages and streaming events, using json_replacements to compress log payloads, consuming prompt.messages in build_messages, restoring opaque metadata in subsequent requests, tracking token usage, tracking the resolved model name, LLM_RAISE_ERRORS. Plugin utility functions include llm.get_key(), llm.user_dir(), llm.ModelError, Response.fake().

Python API: supports basic prompt execution, system prompts, attachments, tools, schemas, fragments, model options, passing API keys, plugin models, accessing the underlying JSON, token usage, streaming responses, structuredmessages and streaming events, async models (AsyncResponse, tool functions can be synchronous or asynchronous, tool use with async models), conversations (including tool use), listing models, running code after a response completes. Other functions include set_alias(alias, model_id), remove_alias(alias), set_default_model(alias), get_default_model(), set_default_embedding_model(alias) and get_default_embedding_model().

Embeddings: command-line embeddings include llm embed, llm embed-multi, llm similar, llm embed-models, llm collections list, llm collections delete. Embeddings can also be used from Python, working with collections, retrieving similar items, viewing the SQL schema, writing plugins to add new embedding models (EmbeddingModel, embedding binary content, embedding storage format).

Templates: support --save to get started, using templates, listing available templates, using YAML files as templates, system prompts, fragments, options, tools, schemas, additional template variables, specifying default parameters, configuring code extraction, setting a default model for a template, template loaders from plugins.

Fragments: using fragments in prompts, using fragments in chat, browsing fragments, setting aliases for fragments, viewing fragments in logs, using fragments from plugins, listing available fragment prefixes.

Model aliases: listing aliases, adding new aliases, deleting aliases, viewing the aliases file.

Configuration: setting a custom default model, setting a custom directory location, turning SQLite logging on or off.

Logging to SQLite: viewing logs (-s/--short mode), conversation logs, searching logs, filtering after a specific ID, filtering by model, filtering by prompts that used a specific fragment, filtering by prompts that used a specific tool, browsing data collected using patterns, browsing logs with Datasette, backing up the database. Message storage involves threads, turns, messages and parts, with a complete example, content addressing as a contract, forking and shared history, storage by reference, raw provider payloads, table-by-table explanation, logging from Python, SQL schema.

Applicable scenarios

• Quickly invoking large language models in the terminal for prompt experimentation.
• Scenarios that require persisting prompts, responses and embeddings to SQLite and performing retrieval and analysis.
• Tasks that require extracting structured data from text or images.
• Scenarios that require providing tool-calling capabilities to models.
• Scenarios that require running models locally (for example via Ollama) or connecting to local OpenAI-compatible endpoints (for example LM Studio).
• Development scenarios that require extending capabilities such as models, tools, templates, fragments and embeddings through plugins.

Limitations

The material does not provide information about LLM's own charges, prices or payment plans. The complete list of which models, plugins and features are specifically supported, as well as the specific operational details of each feature, must be referenced in the official documentation and release notes. The material does not provide specific explanations regarding platform compatibility, performance or effectiveness.