Agent-E is an agent-based system designed to automate a user's operations on a computer, currently focused primarily on automation within the browser. The system is built on the AG2 agent framework (formerly AutoGen) and provides a way to interact with a web browser through natural language.
Main capabilities include: filling out web forms using user information or information from other websites (PDF is not currently supported); searching for and sorting products on e-commerce websites such as Amazon based on criteria such as best-seller rank or price; locating specific content and details on websites (such as ESPN sports scores or contact information on university pages); navigating to and operating web media, including playing YouTube videos and managing playback settings such as fullscreen and mute; performing comprehensive web searches to gather information on various topics; filtering issues and automating task management on project management platforms such as JIRA; and providing a personal shopping assistant that recommends products based on user needs.
Usage: You can get started quickly via scripts. On macOS/Linux, run ./install.sh from the project root directory, and add the -p flag if Playwright support is needed; on Windows, run .\win_install.ps1 in PowerShell, and add the -p flag to install Playwright. Then set the fields as instructed in the newly created .env and agents_llm_config.json. Manual installation is also possible: use uv to manage the Python virtual environment and dependencies (on macOS/Linux run curl -LsSf https://astral.sh/uv/install.sh | sh, on Windows run powershell -c "irm https://astral.sh/uv/install.ps1 | iex", or use pip install uv). Then create and activate a virtual environment, install dependencies, optionally install the Playwright driver, copy .env-example to .env, and configure environment variables such as AUTOGEN_MODEL_NAME and AUTOGEN_MODEL_API_KEY. At runtime, you can use the ./run.sh script or the corresponding command. After running, an icon will appear in the browser interface; click it to open a chat-style interface and enter natural language commands, such as "open youtube and search for funny cat videos" or "find iPhone 14 on Amazon and sort by best seller".
In addition, Agent-E provides a FastAPI wrapper that allows commands to be sent via HTTP and streaming results to be received. On Linux/macOS, run uvicorn ae.server.api_routes:app --reload --loop asyncio; on Windows, run the same command without --reload. A POST request can be sent to execute a task, and the API request may optionally include an llm_config object to apply different configurations.
In terms of configuration, Agent-E supports advanced LLM configuration through environment variables or a JSON-based configuration file. Environment variables include AUTOGEN_MODEL_NAME, AUTOGEN_MODEL_API_KEY, AUTOGEN_MODEL_BASE_URL, AUTOGEN_MODEL_API_TYPE, AUTOGEN_MODEL_API_VERSION, AUTOGEN_LLM_TEMPERATURE, AUTOGEN_LLM_TOP_P, BROWSER_STORAGE_DIR, SAVE_CHAT_LOGS_TO_FILE, LOG_MESSAGES_FORMAT, ADDITIONAL_SKILL_DIRS, PLANNER_USER_INPUT_SKILL_ENABLED, and others. If temperature, top_p, or seed is not set, the default for gpt-* models is temperature 0.0 and top_p 0.001, seed 12345; for other models, temperature 0.1 and top_p 0.1 are used by default.
Agent-E also supports using open-source models through LiteLLM and Ollama, allowing language models to run locally. The steps include: installing LiteLLM (pip install 'litellm[proxy]'), installing Ollama, pulling an Ollama model, running the LiteLLM proxy, and then configuring the model in AutoGen (modify the .env file; no model name or API key is required). However, the official documentation notes that although running Agent-E with a local LLM is feasible, it has not yet been fully tested and should be used with caution.
Architecturally, Agent-E leverages the interaction between skills and agents. Each skill represents an atomic action, and after execution returns a natural language description of its result. This granularity enables Agent-E to flexibly combine these skills to handle complex web automation workflows.
Applicable scenarios include: web form filling, e-commerce product search and sorting, website information lookup, online media operations, web search, task management on project management platforms, personal shopping assistance, and more. In terms of limitations: it currently focuses on in-browser automation, and form filling does not yet support PDF; local LLM support has not been fully tested; using a local Chrome browser requires setting BROWSER_STORAGE_DIR; on macOS, a BlockingIOError may be encountered and needs to be resolved by using the -u flag. Some details such as fees and pricing are not yet provided.
