AI Town is a virtual town where AI characters live, chat, and socialize. It is a deployable starter kit that makes it easy to build and customize your own version of AI Town, inspired by the research paper "Generative Agents: Interactive Simulacra of Human Behavior."
Main goal: Besides the development process itself being fun, this project aims to provide a platform with a solid foundation that is easy to extend. The backend natively supports shared global state, transactions, and a simulation engine, making it suitable for both simple experimental projects and scalable multiplayer games. A secondary goal is to provide a JS/TS framework, since most simulators in this field (including the original paper mentioned above) are written in Python.
Technical architecture:
- Game engine, database, and vector search: Convex
- Authentication (optional): Clerk
- The default chat model is llama3, and the embedding model is mxbai-embed-large
- Local inference: Ollama
- Other cloud LLMs can be configured: Together.ai or any service compatible with the OpenAI API; PRs to add support for more cloud providers are welcome
- Background music generation: Replicate using MusicGen
- Pixel art generation: Replicate, Fal.ai
- All interactions, background music, and component rendering are powered by PixiJS
Installation methods (cloud, local, Docker, self-hosted, Fly.io, etc.):
Standard Convex setup (local or cloud development): A Convex account is required (free), and this is the easiest way to deploy to the cloud and develop seriously. After running, visit http://localhost:5173。
Docker Compose self-hosted Convex: Suitable for users who do not want to register an account and are comfortable with Docker. The frontend runs at http://localhost:5173，后端运行在 http://localhost:3210（http api is 3211), and the dashboard runs at http://localhost:6791。需要生成 admin key and add it to the .env.local file.
There is also a community fork that provides a Pinokio one-click installation (suitable for users who only want to run it without modifying it), and it can also be deployed to Fly.io (see the instructions in ./fly).
Windows installation: Windows 10/11 is required with WSL2 installed, and the Ubuntu distribution is recommended. NVM and Node.js 18 need to be installed, Python and Pip need to be installed, and then follow the steps above.
Configuring the LLM:
- Ollama (default): Download and install Ollama, run ollama serve, and execute ollama pull llama3 to download the model. You can customize the model by adjusting convex/util/llm.ts or setting npx convex env set OLLAMA_MODEL. If you need to change the embedding model, change OLLAMA_EMBEDDING_DIMENSION and set OLLAMA_EMBEDDING_MODEL. If it is slow, you can set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts.
- OpenAI: Set the OPENAI_API_KEY environment variable, and optionally choose models via OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.
- Together.ai: Set TOGETHER_API_KEY, and optionally choose models via TOGETHER_CHAT_MODEL and TOGETHER_EMBEDDING_MODEL. The embedding model dimension must match EMBEDDING_DIMENSION.
- Other OpenAI-compatible APIs (such as Anthropic, Groq, Azure): Change EMBEDDING_DIMENSION to match the embedding model dimension, edit getLLMConfig in llm.ts, or set environment variables. Note: After changing the LLM provider or embedding model, you should delete the data and start over, because memory embeddings are based on the chosen embedding model, and the vector database dimension must match the embedding model dimension.
Customizing the simulation world:
- After each change to character data, you should rerun npx convex run testing:wipeAllTables then npm run dev to re-upload to Convex. Note that this command will clear all data.
- Create your own characters and stories: all characters, stories, and their sprite sheet references are stored in characters.ts. You can first change the character descriptions.
- Update sprite sheets: define character textures and sprite data in data/characters.ts. You need to find sprite sheets for the characters and define sprite movements/assets in the corresponding files.
- Update backgrounds (environments): maps are loaded from data/gentle.js in convex/init.ts. Use Tiled to export the tile map as JSON (two layers, bgtiles and objmap), then convert it with the convertMap.js script: node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>. After generating converted-map.js, it can be used like gentle.js.
- Add background music (optional): use Replicate to generate daily background music. You need to create a Replicate account and obtain an API token, and set npx convex env set REPLICATE_API_TOKEN. You need to be able to receive Replicate webhooks. You can adjust the music by modifying the prompts in convex/music.ts, and adjust the generation frequency via convex/crons.ts.
Run/test/debug commands:
- Stop the backend (when there is too much activity): stop the engine and agent runs; queries and functions can still be run for debugging.
- Restart the backend.
- Kick the engine (when the engine or agent is not running).
- Archive the world: reset the world and start from scratch. After archiving the current world, you can view the data in the dashboard, but the engine and agent will no longer run. Then you can use init to create a new world.
- Pause backend deployment: pause/unpause in the dashboard deployment settings. This will stop all functions (client calls, scheduled or cron jobs), as a last resort.
Deploy to production:
- Run npx convex deploy to deploy Convex functions to production
- Run npx convex run init --prod
- Transfer local data: npx convex export then npx convex import --prod
- Clear existing data: npx convex run testing...
Note: if the window is idle, the simulation will pause after 5 minutes. Loading the page can unpause it, and you can also manually freeze/unfreeze the world via a UI button. If you want to run the world without a browser, you can comment out the "stop inactive worlds" cron job in convex/crons.ts.
