What is SurfSense
SurfSense is an open-source AI research tool, similar to NotebookLM and Perplexity, with stronger extensibility. The tool can integrate with various external data sources (such as search engines, Slack, Notion, YouTube, GitHub, etc.), supports users in uploading files in multiple formats, and integrates content into a personal knowledge base. SurfSense provides powerful search capabilities and natural language interaction, allowing users to quickly find and cite saved content.
Main Features of SurfSense
- Powerful Search: Combines semantic and full-text search to quickly find content in the knowledge base.
- Multi-format Support: Supports uploading more than 50 file formats (such as documents, images, etc.).
- Natural Language Interaction: Users ask questions in natural language, and the system generates cited answers based on the content in the knowledge base, providing a natural and smooth interactive experience.
- External Data Source Integration: Integrates with various external data sources such as search engines, Slack, Notion, YouTube, GitHub, etc., broadening the channels for information acquisition.
- Privacy and Local Deployment: Supports local LLMs, ensuring data privacy, and can be self-hosted.
- Browser Extension: Through the browser extension, users can conveniently save web page content, especially pages that require login, for later organization and citation.
- Document Management: Conveniently manages uploaded files and supports multi-document interaction.
Technical Principles of SurfSense
- RAG: Based on vector embeddings and full-text retrieval technology, it quickly retrieves document fragments most relevant to the user's query from the knowledge base. The retrieved fragments are input as context into the LLM to generate natural language answers. It combines semantic search (based on vector embeddings) and full-text search (based on keyword matching), and uses Reciprocal Rank Fusion (RRF) to optimize the relevance of search results.
- Vector Embeddings and Indexing: Performs efficient vector similarity operations based on pgvector (the vector extension for PostgreSQL). Document content is embedded into the vector space, facilitating quick retrieval of semantically related fragments.
- Hierarchical Indexing: Based on a hierarchical index structure (2-layer RAG architecture), it improves retrieval efficiency and accuracy. The first-layer index is used to quickly filter relevant documents, and the second-layer index is used to further refine the retrieval results.
- Backend Tech Stack: Uses FastAPI to build a modern API framework, supporting high-performance web services. The database uses PostgreSQL, combined with pgvector to provide vector search capabilities. Based on LangChain and LiteLLM for LLM integration, it supports multiple language models.
- Frontend Tech Stack: Uses Next.js and React to build the user interface, providing a smooth interactive experience. Based on Tailwind CSS and Framer Motion, it provides customized UI design and animation effects.
- Browser Extension: Uses Plasmo to build the browser extension, supporting cross-browser functionality, allowing users to conveniently save web page content.
SurfSense Project Address
- Project Website：https://www.surfsense.net/
- GitHub Repository：https://github.com/MODSetter/SurfSense
Application Scenarios of SurfSense
-
Personal Knowledge Management: Helps users organize notes, documents, etc., quickly retrieve knowledge base content, and improve the efficiency of personal knowledge management.
-
Academic Research: Assists with literature retrieval and data organization, generates research reports, and improves the efficiency and quality of academic research.
-
Enterprise knowledge sharing: Enterprises can upload internal materials, promote team collaboration and knowledge circulation, and enhance their knowledge management capabilities.
-
Content creation: Helps creators collect creative inspiration, organize reference materials, assist with writing, and spark creative inspiration.
-
Information collection: Users can save web content, integrate external information, improve the efficiency of information organization, and better respond to various information needs.