SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsagentlego

agentlego

AgentLego is a tool API library open-sourced by the InternLM team, used to extend and enhance intelligent agents based on large language models (LLMs). It provides a rich set of multimodal tools covering capabilities such as visual perception, image generation and editing, speech processing, and vision-language reasoning, and supports flexible tool interfaces, allowing users to extend custom tools with any type of parameters and outputs. AgentLego can be easily integrated into mainstream agent frameworks such as LangChain, Transformers Agents, and Lagent, and also supports tool serviceization and remote access, making it convenient for use in scenarios that require heavy machine learning models (such as ViT) or special environments (such as GPU, CUDA).

Open website

About this tool

Purpose

AgentLego is an open-source tool API library designed to extend and enhance intelligent agents based on large language models (LLMs). It provides LLM agents with rich multimodal tool capabilities, enabling agents to perform tasks such as visual perception, image generation and editing, speech processing, and vision-language reasoning.

Main Capabilities

AgentLego provides a set of multimodal tools, mainly divided into the following categories:

  • General capabilities: Calculator (calculation through a Python interpreter), GoogleSearch (Google search), TextToSpeech (convert input text to audio), SpeechToText (transcribe audio to text).
  • Image processing related: ImageDescription (describe the input image), OCR (recognize text from photos), VQA (answer questions based on images), HumanBodyPose (estimate human body poses or keypoints in an image), HumanFaceLandmark (estimate facial keypoints in an image), ImageToCanny (extract image edges), ImageToDepth (generate a depth map), ImageToScribble (generate sketch scribbles), ObjectDetection (detect all objects in an image), TextToBbox (detect specific objects in an image described by text).
  • Segment Anything series: SegmentAnything (segment all items in an image), SegmentObject (segment specific objects in an image according to a given object name).
  • Image generation and editing: TextToImage (generate an image based on input text), ImageExpansion (expand the surrounding area based on image content), ObjectRemove (remove specific objects in an image), ObjectReplace (replace specific objects in an image), ImageStylization (modify an image according to instructions).
  • ControlNet series: CannyTextToImage (generate an image based on a Canny edge image and description), DepthTextToImage (generate an image based on a depth image and description), PoseToImage (generate an image based on a human pose image and description), ScribbleTextToImage (generate an image based on a sketch scribble image and description).
  • ImageBind series: AudioToImage (generate images from audio), ThermalToImage (generate images from thermal images), AudioImageToImage (generate images from audio and images), AudioTextToImage (generate images from audio and text prompts).

Features of AgentLego include:

  • A rich toolset for multimodal extension of LLM agents, including visual perception, image generation and editing, speech processing, vision-language reasoning, and more.
  • Flexible tool interfaces that allow users to easily extend custom tools, supporting parameters and outputs of any type.
  • Easy integration into LLM-based agent frameworks such as LangChain, Transformers Agents, and Lagent.
  • Support for tool servitization and remote access, especially suitable for tools that use heavy machine learning models (such as ViT) or require special environments (such as GPU and CUDA).

Usage

Install the AgentLego package and install the required dependencies for specific tools. Some tools require additional software packages; you need to check the tool's readme file and confirm that all requirements are met. For example, when using the ImageDescription tool, you need to check the Set up section of its readme and install the relevant dependencies.

There are two ways to use it: directly use the tool, or integrate it into an agent framework.

Applicable Scenarios

AgentLego is suitable for scenarios that require extending the multimodal tool capabilities of LLM agents, such as visual perception, image generation and editing, speech processing, vision-language reasoning, and more. It is also suitable for agent applications that need to remotely access heavy models or special-environment tools in a servitized manner.

Limitations

This project is released under the Apache 2.0 license. Users should also ensure compliance with the relevant licenses of the models used in this project.

Related tools

Browse category