NLSOM (Natural Language-Based Societies of Mind) is a multi-agent collaboration framework whose core idea is inspired by Marvin Minsky's "Society of Mind" and Jürgen Schmidhuber's "learning to think." The project aims to enable multiple agents (such as large language models, neural network-based experts, APIs, and role-players) to work together through natural language interfaces, in order to overcome the limitations of a single model and improve multimodal zero-shot reasoning capabilities. NLSOM has been accepted by CVMJ 2025 and won the Best Paper Award at the NeurIPS 2023 Ro-FoMo Workshop.
In terms of use, NLSOM is mainly used for automated task solving and is especially suitable for non-technical users. After users describe a goal in natural language, the system can automatically recommend relevant AI communities and agents, forming a self-organized society of mind to solve the problem. For example, for a visual question answering task, the system can recommend the VQA community and load agents such as BLIP2_VQA, mPLUG_VQA, and OFA_VQA; for an AGI introduction task, it can call agents such as Arxiv, WolframAlpha, Wikipedia, and BingSearch in the search community; for a role-playing task, it can load character agents such as Guan Yu, Liu Bei, Zhuge Liang, and Zhang Fei. These agents conduct mutual interviews through a "brainstorming" process and collaborate to produce more comprehensive answers.
Main capabilities include: recommendation, autonomously selecting communities and agents according to user goals; brainstorming, where multiple agents collaborate more efficiently to solve tasks; a reward mechanism, giving rewards to participating agents; easy management, allowing different domains of NLSOM to be organized through template adjustments; easy expansion, supporting custom communities and agents, with 16 communities and 34 agents currently available; and an elegant UI, supporting multiple file sources (images, text, audio, video, etc.). The system also includes review and summarization steps to ensure task completion and integrate the outputs of each agent.
In terms of usage, the project provides three installation methods: the CONDA method runsconda env create -n nlsom -f nlsom.yaml; the PIP method first creates an environmentconda create -n nlsom python=3.8then executespip install -r requirements.txt; it is also necessary to create the checkpoints directory, modify the Huggingface and modelscope settings (specific lines not provided), and in.env.templatefill in the API keys (the OpenAI API key is required, others are optional), then rename it to.env. Usage flow example: user inputs a goal → the system recommends a community → self-organized loading of agents → agents respond separately → the system reviews, summarizes, and gives rewards. For example, in the Three Kingdoms question, the system loaded character agents such as Guan Yu, Liu Bei, Zhuge Liang, and Zhang Fei, each proposing strategies, and finally the system summarized them.
Applicable scenarios include: multimodal reasoning tasks (such as VQA and image processing); information retrieval and integration (such as introducing AGI); role-playing and creative generation (such as Three Kingdoms strategies); and complex problem solving requiring multi-model collaboration. The system also supports processing multi-source files such as images, text, audio, and video.
In terms of limitations, the materials do not clearly specify concrete performance metrics, charging models, or platform compatibility details. The system relies on external API keys, which may incur corresponding costs; the reward mechanism is described as "rough" and can be optimized in the future; the evaluation of model contributions may not be completely precise; in addition, the installation and configuration steps involve modifying library files, which may impose certain requirements on users' technical ability. Some extended functions (such as video generation) are not fully demonstrated in the materials, and specific implementation details are not yet provided.
