NeMo-Guardrails

Purpose

NVIDIA NeMo Guardrails is an open-source toolkit designed to add programmable guardrails (rails) to conversational applications based on large language models (LLMs). Guardrails are specific ways to control LLM output, such as: not talking about politics, responding to specific user requests in a specific way, following a predefined conversation path, using a specific language style, extracting structured data, and so on. Developers can insert a guardrail layer between the application code and the LLM, thereby building trustworthy, safe, and reliable LLM applications and preventing the model from engaging in unwanted topics.

Main capabilities

The toolkit supports five main types of guardrails:

• Input rails: applied to user input, can reject input to stop subsequent processing, or modify input (such as masking sensitive data or rewriting).
• Dialog rails: affect how the LLM is prompted, operate based on canonical form messages, and decide whether to execute actions, whether to call the LLM to generate the next step or response, whether to use predefined responses, and so on.
• Retrieval rails: applied to text chunks retrieved in RAG (retrieval-augmented generation) scenarios, can reject or modify text chunks.
• Execution rails: applied to the input/output of custom actions (tools).
• Output rails: applied to output generated by the LLM, can reject output or modify output (such as removing sensitive data).

The toolkit also provides protection mechanisms against common LLM vulnerabilities (such as jailbreaks and prompt injection), and includes a variety of built-in guardrails, including LLM self-checks (input/output moderation, fact-checking, hallucination detection), NVIDIA safety models (content safety, topic safety), jailbreak and injection detection, as well as integrations with community models and third-party APIs.

Usage

Installation requires Python 3.10, 3.11, 3.12, or 3.13. It can be installed using pip.

Adding guardrails using the Python API requires only two steps:

1. Load the guardrail configuration and create an LLMRails instance.
2. Use the generate / generate_async method to call the LLM.

The input/output format is similar to OpenAI's Chat Completions API. The toolkit is async-first, and public methods provide both synchronous and asynchronous versions (such as LLMRails.generate and LLMRails.generate_async).

You can also use the built-in CLI to start the guardrail server, load one or more configurations from a specified folder, and expose an HTTP API. For example, use the /v1/chat/completions endpoint to get chat completions. You can also use a Docker container to start the guardrail server (a Dockerfile is provided to build the nemoguardrails image).

LangChain integration is optional and requires setting the environment variable NEMOGUARDRAILS_LLM_FRAMEWORK=langchain or calling set_default_framework("langchain") to enable it.

Guardrail configurations are organized by folder and include config.yml(general configuration options, such as the LLM model, activated guardrails, and custom configuration data),config.py(custom initialization code),actions.py(custom Python actions), and .co files (containing Colang definitions used to define various guardrails). Colang is a modeling language created specifically for designing flexible and controllable conversation flows, with a Python-like syntax.

Applicable scenarios

• Document-based question answering (RAG): enforce fact-checking and output moderation.
• Domain-specific assistants (chatbots): ensure the assistant stays on topic and follows the designed conversation flow.
• LLM endpoints: add guardrails to custom LLMs to enable safer customer interactions.
• LangChain chains (optional): wrap a guardrail layer around the chain.

Limitations

• Evaluating the safety of LLM conversational applications is a complex task and remains an open research problem. This library provides evaluation tools (nemoguardrails evaluate) Supports topic guardrails, fact-checking, moderation, and hallucination detection, and provides sample LLM vulnerability scanning reports.
• The library collects anonymous telemetry data to help NVIDIA understand deployment patterns and usage of safety features. You can opt out before the library starts by setting an environment variable or creating a do_not_track file. Telemetry does not include user content, model names, API keys, endpoints, prompts, completions, token counts, per-request metrics, file paths, usernames, or IP addresses.
• When using a third-party endpoint (such as NVIDIA Build), that endpoint has its own terms of service and privacy practices, and the library's telemetry opt-out does not extend to the endpoint. NVIDIA Build is for evaluation and testing only and must not be used in production environments; when using it, avoid submitting confidential information or personal data.
• The library is licensed under the Apache License 2.0.

Additional Information

The latest release is 0.24.1, and the development branch tracks the latest development progress. Official documentation is available at docs.nvidia.com/nemo/guardrails. The toolkit is compatible with a variety of LLMs, such as OpenAI GPT-3.5, GPT-4, LLaMa-2, Falcon, Vicuna, Mosaic, and others.