Purpose: Diffbot's goal is to transform raw web content into structured knowledge for programmatic querying and use, covering single-page extraction, site-level data collection, natural-language structuring, knowledge graph queries, and web search. The material uses "research Prometheus and draft a sales brief" as an agent operation example, illustrating that these capabilities can be invoked in sequence, and that each step is a real, inspectable Diffbot API call.
Main capabilities:
- Single-page extraction: classify and extract structured, opinionated JSON from a single URL. The official documentation emphasizes that this process does not rely on large language models, so it does not produce hallucinations.
- Whole-site crawling: traverse an entire site from a single seed URL, extracting thousands of pages into the same structured dataset.
- Natural-language structuring: parse entities, relationships, facts, and sentiment from plain text.
- Knowledge graph queries: run structured queries against what it calls "the world's largest public web knowledge graph," querying content such as companies, people, and news.
- Web search: ultra-low-latency search, with no ads and no telemetry, designed for self-hosting and API access. The official documentation states that the Web Search API currently has P90 latency below 300 milliseconds and performs comparably on NDCG to similar "fast" modes; if self-hosted, latency can improve by another 20-100 milliseconds depending on hardware and sharding optimization.
Usage: The material mentions access via API and also offers a self-hosting option; the example workflow is "prompt—RUN AGENT—view transcript," where each step when the agent runs corresponds to a real Diffbot API call. The index can be packaged in compact form (for example, a 4TB core set of news, documents, homepages, recipes, etc.) and can be carried for offline use. The material does not provide specific installation commands or code examples.
Applicable scenarios: collection and dataset construction that require converting web pages into structured data in bulk; extracting entities, relationships, facts, and sentiment from text; structured querying of public web information such as companies, people, and news; and latency-sensitive web search that requires no ads, no telemetry, and self-hosting.
Limitations: The official documentation states that the extraction process does not rely on LLMs, which is the mechanism it describes for avoiding hallucinations. The search index is a compact version (for example, 4TB), not a complete web index. The latency and performance figures are official descriptions; actual performance depends on hardware and sharding optimization. The material does not provide facts such as pricing, charging model, platform support, installation commands, or specific API parameters.
