What is Airweave
Airweave is an open-source tool that can synchronize data from any application (including APIs, databases, websites, etc.) into graph databases and vector databases, enabling data to be retrieved based on intelligent agents or search mechanisms. Airweave simplifies the process of data retrieval and management with features such as data chunking, hash detection, and automatic synchronization. Airweave provides core features such as no-code integration, multi-tenant support, and multi-source data integration, and supports operation through a React frontend interface or FastAPI interface.
Main features of Airweave
- No-code integration: Users can quickly turn application data into searchable content without writing code.
- Multi-tenant support: Suitable for SaaS developers, supports multi-tenant data synchronization based on OAuth2, while ensuring data privacy and security.
- Data chunking: Each data source (such as a database, API, or file system) defines an async def generate_chunks() method, used to generate data chunks in a consistent format.
- Automatic synchronization: Supports scheduled synchronization or on-demand data synchronization.
- Version control and hashing: Detects data changes based on hashing, and only updates the modified parts in the vector store.
- Multi-source support: Supports connecting multiple data sources and unifying them into a single queryable layer.
- Scalability: Supports local deployment based on Docker Compose (future support for Kubernetes for production-grade deployment).
Technical principles of Airweave
- Data collection: Connects to various data sources (such as APIs, databases, file systems, etc.) to collect data. Each data source defines an async def generate_chunks() method, used to process data in chunks and ensure that data can be output in a consistent format.
- Data processing and chunking: After data is processed in chunks, text or other data types are converted into vector form based on embedders. Vectors can be efficiently stored and retrieved by vector databases.
- Data storage：
- Graph database: Used to store relationships between data.
- Vector database: Such as Chroma, Milvus, Pinecone, Qdrant, Weaviate, etc., used to store vectorized data and support efficient similarity search.
- Data synchronization and updates: Detects data changes based on hashing, and only updates modified data chunks, reducing unnecessary data synchronization. Supports scheduled synchronization and on-demand synchronization, allowing users to flexibly configure synchronization schedules according to their needs.
- Retrieval and querying: After data is stored, users can retrieve it based on intelligent agents or search mechanisms. Airweave supports querying through a frontend interface or API interface, allowing users to quickly obtain the data they need.
- Asynchronous task processing: ARQ Redis is used for background task processing, supporting asynchronous processing of large-scale data synchronization and improving system performance and scalability.
- Multi-tenancy and privacy protection: Supports multi-tenant data synchronization based on OAuth2, ensuring data isolation and privacy protection between different tenants.
Airweave project address
- GitHub repository：https://github.com/airweave-ai/airweave
Airweave application scenarios
- Enterprise developers: Build semantic search applications, synchronize enterprise data sources, and achieve efficient data retrieval and management.
- Data Analyst: Quickly retrieve and analyze data, improve data analysis efficiency with Airweave, and generate business intelligence reports.
- SaaS Platform Developer: Provide data retrieval services for multiple tenants, ensuring data isolation and privacy.
- Content Creator and Manager: Quickly retrieve and manage content, improving content management efficiency.
- Intelligent Customer Service Team: Quickly retrieve customer support data, provide accurate answers, and improve customer satisfaction.