open-index

Open Index is an open-source tool designed to build domain-specific, accurate, structured data layers for AI agents, and to continuously keep the data correct as things change. It is used to build a "brain": a searchable, continuously improving domain context graph. This brain is domain-agnostic and can model anything that supports an organization (product → "has common questions" → issue), a sales pipeline (customer → order), infrastructure (service → runbook), and more. Users define concepts, and Open Index stores, searches, and maps them.

Main capabilities

The Open Index brain is made up of four primitives:

• doc_type: the concept to track and maintain (such as service, customer, issue).
• doc_schema: the fields stored for a given doc_type.
• entity: an instance of a doc_type, stored according to the schema. Each entity can be linked to other entities through related_to (target) + relationship_edge_meaning (free-text edge semantics).
• connector: an optional data source from which entities are extracted (such as an MCP server).

It is designed as the context layer for domain-specific agents (legal, marketing, customer support, sales, infrastructure, and so on). The MCP server provides structured context for these agents and provides a validated way to keep context up to date:

• agent prompt: publishes dynamic domain navigation through MCP server instructions, which can be injected by supported hosts before the first turn of a conversation.
• read: navigation_guidelines() refreshes instructions; search_brain() and get_entity() retrieve domain context.
• write: put_entity() (add/update entities), create_doc_type() (define concepts).

The default MCP mode is read-write, and domain agents can both use knowledge and maintain knowledge. Adding --read-only lets agents only consume context without modifying it. Claude Code is supported as an optional MCP client; open-index init generates .mcp.json, CLAUDE.md, and an editing skill for it.

Usage

1. Create a brain：open-index init <name> Generate the directory, then write doc_types (schemas) and entities (instances). Examples include support-brain, infra-brain, and personal-brain.
2. Define a doc_type: create a YAML file under doc_types/, where you can set per-field search weights (boost) and relationship vocabulary (relationships). You can use open-index add-doc-type service Create.
3. Add entities: for storage: file type, write JSON files under entities/<doc_type>/, with id in the format <doc_type>:<slug>, and related_to defines graph edges. Then run open-index index and open-index validate.
4. Populate at scale: manual/agent writes, bulk import (supports JSON, JSONL, CSV,open-index import issues.csv --doc-type issue --asserted-by import:jira), connectors (connectors/*.py pull from MCP servers on a schedule,open-index ingest <name> or open-index run), agent write-back (the Stop hook records learnings through put_entity).
5. Explore:open-index ui opens a read-only browser with four tabs: Explore, Map, Analytics, and Jobs.
6. MCP server:open-index mcp runs the MCP server (stdio), defaulting to read-write.open-index mcp --read-only or open-index serve --read-only Read-only is allowed.

Storage and Backends

Storage uses SQLite + FTS5 by default (zero external services). The backend sits behind a pluggable interface with two implementations: SQLite (default, local/dev) and OpenSearch (selected via search.backend: opensearch).

Each doc_type declares its true source:

• storage: index (default): the search database owns these entities and does not write to files. Suitable for connector-pulled, high-volume, or time-series data.
• storage: file: JSON files under entities/<doc_type>/ are the true source and are tracked by git and reviewed via PR. Suitable for carefully curated entities written by humans/agents.

open-index index Each run reconciles file-backed types from disk and preserves index-backed entities (written by connectors/agents) unchanged. brain.db is the persistent state for index-backed types and is gitignored by default.

Cloud Agent Usage (Production)

Using Docker (recommended for a shared brain):docker compose --profile sqlite up. Both methods serve http://localhost:8080/mcp。brain.yaml identically, and the profile sets OPEN_INDEX_SEARCH_BACKEND, overriding the file. Add --profile ui to use the browser at :8501.

serve exposes the same read/write tools over streaming HTTP with bearer-token authentication. Without a token, the endpoint is unauthenticated—anyone who can reach the port can write to the brain. Use --read-only to get a queryable endpoint that agents cannot modify.

Choosing a backend: SQLite is single-writer. When a second agent needs to write, migrate to OpenSearch. It also provides native per-field boosting, fuzzy (tolerant) search, and k-NN semantic search, and can scale beyond SQLite's brute-force ceiling of roughly 10k entities.

Search Control

Schema (per field): data type (string/number/boolean/timestamp), processing (keyword/text/timestamp), and search type (syntactic = keyword+prefix, semantic = vector-backed dense search, none = not indexed). After marking a field search: semantic, the backend automatically embeds it at index time.

Ranking: per-field boosters, where each field's boost weight determines the importance of a match. For hybrid queries, keyword and semantic scores are blended using search.semantic_weight (default 0.3).

Embedding models: installing the [semantic] extra (pip install 'open-index[semantic]') enables local embeddings. The default model is BAAI/bge-small-en-v1.5 (384-D). It can be overridden via search.embedding_model in brain.yaml, or by using an OpenAI-compatible API (set OPEN_INDEX_EMBEDDING_BASE_URL, etc.). Changing the embedding dimension requires rebuilding the index: on SQLite open-index index --reembed, and on OpenSearch, after recreating the index, a full open-index index --reembed.

Limitations

SQLite semantic cap: the SQLite backend performs a brute-force cosine scan over the entities in scope, suitable for up to about 10,000 entities; larger brains need to switch to the OpenSearch backend or a future sqlite-vec integration.

Not yet implemented (declarable seams exist): type-level and temporal boosters.

Contributing

Contributions are welcome—new doc_type examples, connectors, backends, documentation fixes, or bug reports. Ask or discuss in the Discord community first.

Change process: first open an issue to discuss non-trivial changes (new backend, schema changes, CLI surface); branch from main (feat/…, fix/…, docs/…); add tests under tests/ and run pytest; if you modified a brain in examples/, run open-index validate --brain examples/<name>; Update README/docs; submit a PR describing what changed and why, and link the issue. CI runs the test suite on Python 3.10 and 3.13 and validates every brain in examples/.

Note: pytest skips these suites when the MCP and UI extras are not installed, so a green run on a partial installation means little—use [all] locally (or at least [ui,mcp]). CI installs them explicitly and fails if they are missing.