Taxy AI is a browser extension designed to implement browser automation through large language models such as GPT-4, helping users complete repetitive operations. Its core purpose is to let users define ad hoc instructions in natural language, with an AI agent performing actions such as clicking and typing on web pages.
Main capabilities:
- Use GPT-4 or GPT-3.5 to control the browser and execute user-defined ad hoc instructions.
- Support two interaction methods: click(id) to click interactive elements, and setValue(id, text) to enter text in text input boxes.
- Extract the web page DOM through a content script, simplify and templatize the HTML, retaining only interactive or semantically important elements and assigning ids, in order to reduce token consumption.
- Send the simplified DOM and user instructions to the selected LLM, parse the returned actions, and execute them.
- The action loop executes up to 50 actions, and tasks usually involve fewer than 10 actions.
- Provide two forms of use: a popup window and a developer tools panel.
- Fully open source, does not send page content or instructions to its own servers, and stores API keys locally in the browser.
How to use: Currently it can only be obtained through the GitHub repository and must be built and installed locally. Steps: ensure Node.js >= 16; clone the repository; run yarn to install dependencies; run yarn start to build; load the unpacked extension in Chrome (chrome://extensions/, enable Developer mode, click "Load unpacked", and select the build folder). After installation, you can open the popup window with the shortcut cmd+shift+y (Mac) or ctrl+shift+y (Windows/Linux), or open the Taxy AI panel in the developer tools. You need to provide an OpenAI API key and paste it into the specified box. Then navigate to the target web page (such as the OpenAI playground) to begin experimenting.
Applicable scenarios:
- Protect the main branch in GitHub.
- Search for and play the movie "Oblivion" in Netflix.
- Create calendar events (for example, "schedule a standup at 10 a.m. tomorrow and invite david@taxy.ai").
- Write an article in the OpenAI playground.
- Other repetitive browser operations.
Limitations:
- The current status is a research preview, and many workflows will fail or confuse the agent.
- Currently only supports the Chrome browser, requires manual build and installation, and has not yet been listed on the Chrome Web Store.
- Requires the user to provide their own OpenAI API key.
- A single task executes up to 50 actions.
- Future plans include support for saving and scheduled workflows, but these are not yet provided.
- The security-first architecture automatically stops execution upon an unexpected response.
Tech stack: Chrome Extension Manifest V3, Webpack Dev Server 4, React Hot Loader, eslint-config-react-app.
