BigSet - TinyFish's Open-Source Multi-Agent Real-Time Web Scraping Tool

What is BigSet

BigSet is an open-source multi-agent system launched by the American startup TinyFish, licensed under AGPL-3.0. Users only need to describe the required data in natural language, and the system can automatically infer the table structure, dispatch agents to scrape data from the live web, deduplicate and verify it, and ultimately generate an exportable CSV/XLSX structured dataset. The tool supports scheduled refreshes ranging from every 30 minutes to weekly, keeping the dataset updated in real time.

Main Features of BigSet

Natural Language Table Creation: Just describe the required data in one sentence, and the AI automatically infers column names, data types, and primary keys, with no need to manually design the table structure.
Multi-Agent Collection: The orchestrator agent discovers target entities, and sub-agents scrape individual records in parallel, with each limited to at most 6 tool calls, automatically completing the entire process from discovery to population.
Automatic Deduplication and Traceability: Automatic deduplication based on primary keys, with each row of data accompanied by a source URL, ensuring all data can be traced and verified.
Scheduled Refresh: Supports five automatic update frequencies: every 30 minutes, 6 hours, 12 hours, daily, and weekly, keeping the dataset synchronized in real time.
Format Export: Generated results support direct download in both CSV and XLSX formats, making it easy to connect to Excel or data analysis tools.
Preset Datasets: Includes 9 curated public datasets, ready to use out of the box.

Technical Principles of BigSet

Multi-Agent Orchestration Architecture: BigSet builds a multi-agent system based on the Mastra framework, with its core consisting of an orchestrator and sub-agents (Worker Agents). The orchestrator is responsible for discovering target entities and generating scraping tasks, while sub-agents execute individual record scraping in parallel, with each limited to at most 6 tool calls, uniformly scheduling Claude Sonnet and Qwen models through the Vercel AI SDK and OpenRouter.
Automatic Table Structure Inference: After the system receives a natural language description, Claude Sonnet analyzes the semantic intent and automatically infers column names, data types, and primary key constraints. This process requires no manual specification of the Schema; the LLM generates a structured table definition based on the entity relationships and data characteristics in the description, and uses it as the verification benchmark for subsequent collection.
Data Collection and Deduplication Verification: The orchestrator splits target entities into independent tasks and assigns them to sub-agents for parallel scraping. Each sub-agent calls the TinyFish Search / Fetch / Browser API to obtain data from the live web; after results are returned, the system automatically deduplicates based on primary keys and attaches a source URL to each row of data, ensuring data traceability.
Security Protection Mechanism: To prevent prompt injection attacks, the dataset ID is not passed to the LLM through the system prompt, but is injected at runtime using a JS closure. The LLM has no direct access to or ability to leak the dataset identifier throughout the process, achieving isolation between the collection logic and data permissions.

How to Use BigSet

Environment Preparation: Install Docker and Make, and register accounts on the TinyFish, OpenRouter, and Clerk websites to obtain API keys.
Clone the Repository: Run git clone https://github.com/tinyfish-io/bigset.git to download the project code, and copy .env.example to .env file.
Configure Keys: In .env Fill in three types of API keys—TinyFish, OpenRouter, and Clerk—in the file to ensure the service has permissions for data collection, model invocation, and identity authentication.
Start the service: Run the make dev command, and the system will automatically install dependencies, start the Postgres and Convex databases, and complete service deployment.
Access and use: Open a browser and visit localhost:3500, register and log in, then describe the required data in natural language in the input box, and the system can automatically generate a structured dataset.
Load preset data (optional): Execute the make seed-public-datasets command to load the 9 built-in curated public datasets with one click and quickly experience the core features.

BigSet's core advantages

Zero-configuration collection: No need to write crawlers, selectors, or specify URLs; a pure natural language description can automatically generate a structured dataset.
Secure isolation: Dataset IDs are injected through JS closures and are completely inaccessible to the LLM throughout the process, effectively preventing prompt injection attacks.
Open source and self-hostable: Uses the AGPL-3.0 license, supports one-click Docker deployment, and gives you complete control over your data.
Enterprise-grade infrastructure: Built on TinyFish's enterprise-grade search and scraping API, which has processed 40 million+ agent operations, providing stability and reliability.

BigSet's project address

GitHub repository：https://github.com/tinyfish-io/bigset

Comparison of BigSet with similar competing products

Comparison dimension	BigSet	Firecrawl
Product positioning	A multi-agent-driven dataset generation platform, from natural language descriptions to exportable structured datasets	An API-first web scraping and crawling platform that converts websites into LLM-ready Markdown or structured data
Input method	Describe the desired data topic in natural language (e.g., "AI recruiting company information")	Crawl by specifying a URL, or use the `/agent` endpoint to describe requirements in natural language (no URL required)
Data source discovery	AI automatically infers the table structure and discovers entity sources, without the need for users to provide any links	`/agent` can autonomously search and navigate, but the regular mode requires the user to specify a starting URL or sitemap
Table structure definition	The LLM automatically infers the schema (column names, data types, primary keys) and generates a relational dataset	Supports JSON Schema or natural language prompts to define the extraction structure, with output as JSON objects
Collection scope	Aggregates multi-source data across sites and automatically deduplicates to generate a unified dataset	Recursively crawls or extracts from a single page based on a given URL, focusing on content from known sites
Degree of automation	Orchestrator + sub-agent closed loop: discovery → scraping → deduplication → validation → table creation → export, with the entire process autonomous	Executes scraping and conversion in a single pipeline or Agent mode, and requires external tools to complete dataset management
Scheduled refresh	Native support (30 minutes / 6 hours / 12 hours / daily / weekly)	Supports Scheduled Jobs for periodic execution, but requires configuration through the API or an external scheduler
Output format	CSV, XLSX (with source URLs included, ready for direct use in Excel/BI)	Markdown, JSON, HTML, screenshots, links (optimized for LLM/RAG pipelines)
Data deduplication	Automatic deduplication based on primary keys, ensuring dataset uniqueness	Users need to handle deduplication logic themselves downstream

BigSet application scenarios

Competitor monitoring: Automatically track competitors' product prices, feature updates, job postings, etc., and generate structured intelligence reports that can be refreshed on a schedule.
Market research: Collect company information, financing developments, and market share data within a specific industry to quickly build an industry database.
Investment analysis: Track stocks, cryptocurrencies, startup funding rounds, and valuation changes to support data collection for investment decisions.
Recruitment intelligence: Monitor job postings, skill requirements, and salary ranges at target companies or in target industries, and analyze talent market trends.
E-commerce price comparison: Scrape product prices, inventory, and review data from multiple platforms to build a price comparison dataset that updates in real time.