SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

DataClaw - Open-source AI conversation data export tool, one-click conversion to standard training sets

Author: AI小集 · Original source: AI工具集

What is DataClaw

DataClaw is an AI conversation data export tool open-sourced by developer Peter O'Malle. It can automatically capture the complete conversation history between users and AI programming assistants such as Claude Code, Codex CLI, and Gemini CLI, convert it into structured JSONL training datasets, and support one-click publishing to the Hugging Face platform for community use. The tool has built-in PII (personally identifiable information) detection and sensitive key filtering features, automatically removing private content such as passwords and API keys before sharing to ensure data security.

DataClaw

Main features of DataClaw

  • Automatic conversation history capture: Supports exporting complete conversation records from mainstream AI programming assistants such as Claude Code, Codex CLI, and Gemini CLI, with no need for manual copy-pasting.
  • Intelligent privacy redaction: Built-in PII (personally identifiable information) detection engine automatically identifies and removes sensitive content such as passwords, API keys, and email addresses to ensure data security.
  • Structured format conversion: Converts raw conversations into standard training data formats such as JSONL, making them directly usable for large language model fine-tuning.
  • One-click publishing to Hugging Face: Supports directly pushing processed datasets to Hugging Face Hub for download and use by the open-source community.
  • Multi-platform data integration: Compatible with the data formats of multiple AI programming tools, enabling unified management of cross-platform conversation data.
  • Open-source and customizable: Developed in Python with fully open-source code, users can customize data processing rules and redaction strategies according to their needs.

Technical principles of DataClaw

  • Local file system monitoring: By monitoring the conversation log files generated locally by tools such as Claude Code and Codex CLI (such as JSON or SQLite databases), it captures the complete interaction records between users and AI in real time.
  • PII detection and regex matching: Uses rule-based regular expressions and keyword matching algorithms to identify and filter sensitive information such as API keys, passwords, email addresses, and ID numbers to ensure redaction.
  • Structured conversation parsing: Parses unstructured natural language conversations into a standard JSONL format containing fields such as role (user/assistant), content, and timestamp, compliant with OpenAI fine-tuning data specifications.
  • Hugging Face API integration: Enables one-click dataset upload through the Hugging Face Hub Python SDK, automatically handling authentication, repository creation, and version management.
  • Incremental synchronization mechanism: Supports incremental data capture, exporting only newly added conversation content to avoid duplicate processing and full overwrites.
  • Cross-platform adaptation layer: For the differing formats of different AI tools (such as Claude's XML logs and Codex's JSON format), built-in adapters perform unified conversion.

DataClaw project address

  • GitHub repository：https://github.com/peteromallet/dataclaw

Application scenarios of DataClaw

  • Open-source model fine-tuning: Provides developers with high-quality real programming conversation data for fine-tuning open-source code models such as CodeLlama and DeepSeek-Coder, improving their performance on specific programming languages or frameworks.
  • AI programming assistant research: Researchers can use the collected conversation data to analyze interaction patterns between users and AI programming assistants, optimize prompt engineering strategies, or evaluate model performance.
  • Educational Training Dataset Construction: Programming education institutions can organize dialogue data into a teaching case library to train students on how to effectively collaborate with AI in programming.
  • Competitor Model Distillation: Other AI companies or research teams can use publicly available dialogue data as a distillation source to train smaller, more efficient programming-specific models.
  • Data Democratization Movement: Support the open-source community in countering the data-closed strategies of large AI companies, and promote the open sharing and fair use of AI training data.