SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsOther AI toolsJARVIS

JARVIS

JARVIS (also known as HuggingGPT) is an open-source research project by Microsoft that aims to connect large language models (LLMs) with the machine learning community (Hugging Face) to collaboratively complete complex AI tasks. Using an LLM as the controller, it decomposes user requests into executable tasks, automatically selects expert models from Hugging Face and executes them, and finally integrates the prediction results to generate a response. The system workflow is divided into four stages: task planning, model selection, task execution, and response generation. The project provides multiple usage methods such as a Web API, a Gradio demo, and a CLI mode, supports a hybrid mode of local deployment and Hugging Face inference endpoints, and has released related tools and benchmarks such as EasyTool and TaskBench.

Open website

About this tool

JARVIS (HuggingGPT) is a research project initiated by Microsoft, with the mission of exploring artificial general intelligence (AGI) and providing cutting-edge research to the community. The project code is hosted on GitHub (https://github.com/microsoft/JARVIS），论文地址为 https://arxiv.org/pdf/2303.17580.pdf。

Purpose Language serves as the interface for an LLM to connect numerous AI models, used to solve complex AI tasks. JARVIS builds a collaborative system, with an LLM as the controller and numerous expert models (from the Hugging Face Hub) as collaborative executors.

Main Capabilities The system workflow consists of four stages:

  1. Task planning: Use ChatGPT to analyze user requests, understand the intent, and decompose them into potentially solvable tasks.
  2. Model selection: To solve the planned tasks, ChatGPT selects expert models from Hugging Face based on model descriptions.
  3. Task execution: Invoke and execute each selected model, and return the results to ChatGPT.
  4. Response generation: Finally, use ChatGPT to integrate the predictions of all models and generate a response.

Project Updates

  • 2024.01.15: Released Easytool for easier tool usage; see EasyTool for the code and dataset, and the paper EasyTool: Enhancing LLM-based Agents with Concise Tool Instruction.
  • 2023.11.30: Released TaskBench for evaluating the task automation capabilities of LLMs; see TaskBench for the code and dataset, and the paper TaskBench: Benchmarking Large Language Models for Task Automation.
  • 2023.07.28: Evaluation and project restructuring are being planned, and a new version of Jarvis will be released soon.
  • 2023.07.24: Released a lightweight langchain version of Jarvis.
  • 2023.04.16: Jarvis supports OpenAI services and GPT-4 models on the Azure platform.
  • 2023.04.06: Added a Gradio demo and built Web APIs for /tasks and /results in server mode. The Gradio demo is hosted on Hugging Face Space (built with inference_mode=hybrid and local_deployment=standard). The Web APIs /tasks and /results can access Stage 1 (task planning) and the Intermediate results of stages 1-3 (model selection and execution results).
  • 2023.04.03: Added CLI mode and provided parameters to configure the scale of local endpoints. You can experience the lightweight version of Jarvis without locally deploying models by running python awesome_chat.py --config configs/config.lite.yaml.
  • 2023.04.01: Updated the build code version.

System Requirements For configs/config.default.yaml:

  • Ubuntu 16.04 LTS
  • RAM > 12GB (minimum), 16GB (standard), 80GB (full)
  • Disk > 284GB: damo-vilab/text-to-video-ms-1.7b requires 42GB, ControlNet requires 126GB, stable-diffusion-v1-5 requires 66GB, others require 50GB For configs/config.lite.yaml: This configuration does not require downloading and deploying any expert models locally, but it means Jarvis is limited to models that run stably on HuggingFace Inference Endpoints.

Usage First replace openai.key and huggingface.token in server/configs/config.default.yaml with your personal OpenAI Key and Hugging Face Token, or put them into the environment variables OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN respectively. Then run the corresponding command. The Jarvis service can be accessed via Web API:

  • /hugginggpt --method POST, access the full service.
  • /tasks --method POST, access the intermediate results of stage 1.
  • /results --method POST, access the intermediate results of stages 1-3. A user-friendly web page is provided. After starting awesome_chat.py in server mode, you can run commands in the browser to communicate with Jarvis. Nodejs and npm need to be installed first. If running Web client on another machine, set HUGGINGGPT_BASE_URL in web/src/config/index.ts to http://{LAN_IP_of_the_server}:{port}/。若想使用视频生成功能，需手动编译支持 ffmpeg for H.264. Double-click the settings icon to switch to ChatGPT. The Gradio demo is hosted on Hugging Face Space, and you can also run commands locally to start the demo. CLI mode: makes it easier to run Jarvis, and provides CLI mode examples.

Configuration Parameters The server-side configuration file is server/configs/config.default.yaml. Some parameters are as follows:

  • model: LLM, currently supports text-davinci-003. More open-source LLMs are being integrated.
  • inference_mode: inference endpoint mode. local: use only local inference endpoints; huggingface: use only Hugging Face inference endpoints (no local inference endpoints required); hybrid: use both local and huggingface.
  • local_deployment: scale of locally deployed models, effective in local or hybrid inference modes. minimal (RAM>12GB, only ControlNet), standard (RAM>16GB, ControlNet + standard pipeline), full (RAM>42GB, all registered models). On a personal laptop, the recommended configuration is inference_mode: hybrid and local_deployment: minimal. However, under this setting, the available models may be limited due to the instability of remote Hugging Face Inference Endpoints.

NVIDIA Jetson Embedded Device Support A Dockerfile is included to provide experimental support for NVIDIA Jetson embedded devices. This image provides accelerated ffmpeg, pytorch, torchaudio, and torchvision dependencies. Before building the Docker image, ensure that the default Docker runtime is set to 'nvidia'. The prebuilt image is located at https://hub.docker.com/r/toolboc/nv-jarvis。 Due to memory requirements, JARVIS needs to run on Jetson AGX Orin series devices (preferably a 64G onboard memory device), with configuration options set to: inference_mode: local, local_deployment: standard. It is recommended to provide models and configurations through volume mounts from the host to the container. You can uncomment the # Download local models section in the Dockerfile to build a container that includes the models.

Citation If you find this work useful in your method, please cite the paper as follows: JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf

Restrictions Missing facts regarding fees, pricing, platforms, effects, or usage steps have not been provided.

Related tools

Browse category