RepoAgent

RepoAgent is a repository agent framework driven by large language models (LLMs), focused on repository-level code documentation generation. Its goal is to create an intelligent documentation assistant that helps people read and understand code repositories and generate documentation, thereby improving efficiency and saving time.

Purpose

In computer programming, complete project documentation (including detailed explanations of each Python file) is crucial for understanding, maintaining, and enhancing a codebase. Traditionally, creating and maintaining software documentation requires a large amount of manpower and specialized expertise, which is a challenge for small teams without dedicated personnel. RepoAgent leverages large language models such as GPT to let AI take on most of the documentation process, so that human developers only need to focus on verification and fine-tuning, greatly reducing the burden of manually writing documentation.

Main Capabilities

• Automatically detect changes in Git repositories, tracking file additions, deletions, and modifications.
• Independently analyze code structure through AST, generating documentation for individual objects.
• Accurately identify bidirectional call relationships between objects, enriching the global perspective of the documentation content.
• Seamlessly replace Markdown content based on changes, maintaining documentation consistency.
• Perform multithreaded concurrent operations to improve documentation generation efficiency.
• Provide a sustainable, automated way to update documentation for team collaboration.
• Present code documentation in an excellent way (each project is equipped with a documentation book powered by Gitbook).
• Support exploring the prototype of the downstream application "Chat With Repo", such as automatic Q&A for Issues and code explanation.
• Planned or supported features include: automatically generating README.md in combination with global documentation, supporting multiple programming languages such as Java/C/C++, and supporting local models such as Llama, chatGLM, Qwen, and GLM4.

Usage

Installation

The repository supports GitHub Actions for automated workflows (such as build, test, and deployment). For detailed instructions, please refer to actions/run-repoagent. The repoagent package can be installed directly with pip. If you wish to contribute code or set up a development environment:

1. Install PDM (if not already installed).
2. Use CodeSpace or clone the repository:
   • Use CodeSpace: click to use GitHub Codespace, then proceed to the next step.
   • Clone the repository:

     git clone https://github.com/LOGIC-10/RepoAgent.git
     cd RepoAgent
     

3. Configure with PDM:
   • Run the following command in the /RepoAgent directory to initialize the Python virtual environment:

     pdm venv create --name repoagent
     

   • Activate the virtual environment.
   • Install dependencies with PDM:

     pdm install
     

Configuration

Before configuring specific parameters, make sure to configure the OpenAI API as an environment variable in the command line. Go to the RepoAgent root directory and try the corresponding command in the terminal. The run command supports the following optional flags (if set, they will override the configuration defaults):

• -m, --model TEXT: specify the model used for completion. Default: gpt-3.5-turbo
• -t, --temperature FLOAT: set the generation temperature of the model. Lower values make the model more deterministic. Default: 0.2
• -r, --request-timeout INTEGER: define the timeout for API requests (in seconds). Default: 60
• -b, --base-url TEXT: the base URL for API calls. Default:https://api.openai.com/v1
• -tp, --target-repo-path PATH: The filesystem path of the target repository. Used as the root directory for document generation. Default: path/to/your/target/repository
• -hp, --hierarchy-path TEXT: The name or path of the project hierarchy file, used to organize the document structure. Default: .project_doc_record
• -mdp, --markdown-docs-path TEXT: The folder path where Markdown documents are stored or generated. Default: markdown_docs
• -i, --ignore-list TEXT: A comma-separated list of files or directories to ignore during document generation.
• -l, --language TEXT: The ISO 639 code or language name for the documentation. Default: Chinese
• -ll, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]: Sets the application's log level. Default: INFO

First Generation and Automatic Maintenance

If this is the first time generating documentation for the target repository, RepoAgent will automatically create a JSON file that maintains global structure information, and create a folder named Markdown_Docs in the root directory of the target repository to store the documentation. After the initial generation of global documentation for the target repository, or if the cloned project already contains global documentation information, you can seamlessly and automatically maintain the project's internal documentation with your team by configuring a pre-commit hook in the target repository. RepoAgent currently supports generating documentation for projects, and requires some configuration in the target repository:

1. Ensure that the target repository is an initialized git repository.
2. Install pre-commit in the target repository to detect changes in the git repository.
3. Create a file named .pre-commit-config.yaml in the root directory of the target repository (see the original text for an example).
4. For the specific configuration method of the hook, please refer to pre-commit. After configuring the yaml file, run the corresponding command to install the hook. In this way, every git commit will trigger RepoAgent's hook, automatically detect changes in the target repository, and generate the corresponding documentation. Subsequently, you can make some modifications to the target repository (for example, add new files or modify existing files), and simplyFollow the normal git workflow: git add, git commit -m "your commit message", git push. The RepoAgent hook will be automatically triggered on git commit, detect the files added in the previous step, and generate the corresponding documentation. After execution, RepoAgent will automatically modify the staged files in the target repository and formally commit them. After execution is complete, a green "Passed" will be displayed. The generated documentation will be stored in the specified folder under the root directory of the target repository.

Examples and Quality

The official documentation for the XAgent project (approximately 270,000 lines of code) was generated using the default model gpt-3.5-turbo, and the results can be viewed in the Markdown_Docs directory on the XAgent project GitHub. To improve documentation quality, it is recommended to consider more advanced models, such as gpt-4-1106 or gpt-4-0125-preview. Finally, you can flexibly adjust the output format, template, and other aspects of the documentation through custom prompts.

Applicable Scenarios

• Small teams lacking dedicated documentation personnel that want to automatically generate and maintain project documentation.
• Developers who need to quickly understand unfamiliar or large code repositories.
• Teams that want to integrate documentation generation into existing Git workflows (such as pre-commit).
• Explorers interested in automated technical writing and downstream applications such as "chat with the repository".

Limitations

• Requires configuring the OpenAI API environment variables to use.
• Currently mainly generates documentation for Python projects; multi-programming-language support (Java, C/C++, etc.) is a planned feature.
• Local model support (such as Llama, chatGLM, Qwen, GLM4) is a planned feature.
• Documentation quality is affected by the chosen model; the default gpt-3.5-turbo may have limited effectiveness, and a more advanced model is recommended.
• Specific information such as fees and pricing is not yet provided.