Guardrails

Purpose

NVIDIA NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails (Rails) to conversational applications based on large language models (LLMs). Guardrails are specific ways of controlling the output of a large language model, such as not talking about politics, responding to specific user requests in a specific way, following a predefined conversation path, using a specific language style, extracting structured data, etc. This toolkit adds a guardrails layer between the application code and the LLM, helping developers build trustworthy, safe, and reliable LLM applications.

Main Capabilities

• Build trustworthy, safe, and reliable LLM applications: Guardrails can be defined to guide and protect conversations, specify the application's behavior on specific topics, and prevent it from engaging in unwanted topic discussions.
• Securely connect models, chains, and other services: LLMs can be seamlessly and securely connected to other services (i.e., tools).
• Controllable conversations: Guide the LLM to follow predefined conversation paths, make interaction design conform to conversation design best practices, and enforce business processes (such as authentication and support processes).
• Defend against LLM vulnerabilities: Provide multiple mechanisms to prevent common vulnerabilities such as jailbreaks and prompt injection.

Guardrail Types

• Input guardrails: Applied to user input, can reject input to stop subsequent processing, or modify input (such as masking sensitive data, rewriting).
• Dialog guardrails: Affect how the LLM is prompted, based on canonical form messages, determine whether to execute actions, whether to call the LLM to generate the next step or response, whether to use predefined responses, etc.
• Retrieval guardrails: Applied to text chunks retrieved in RAG scenarios, can reject a chunk so that it is not used to prompt the LLM, or modify the relevant chunk (such as masking sensitive data).
• Execution guardrails: Applied to the input/output of custom actions (i.e., tools).
• Output guardrails: Applied to the output generated by the LLM, can reject the output to prevent it from being returned to the user, or modify the output (such as removing sensitive data).

Usage

• Installation: Requires Python 3.10, 3.11, 3.12, or 3.13, and can be installed using pip (see the official installation guide for specific commands).
• Python API: Load the guardrails configuration and create an LLMRails instance, then call the LLM using the generate / generate_async methods. The input and output format is similar to OpenAI's Chat Completions API. This library adopts an async-first design, with its core mechanism based on the Python async model, and its public methods provide both synchronous and asynchronous versions.
• Guardrails Server: A guardrails server can be started via the CLI, loading one or more configurations from a specified folder and exposing an HTTP API for use. The nemoguardrails image can also be built using a Docker container.
• Configuration Structure: A guardrails configuration folder typically contains config.yml (general configuration, such as the LLM model, activated guardrails, and custom configuration data), config.py (custom initialization code), actions.py (custom Python actions), and .co files (containing Colang definitions that define various types of guardrails).
• Colang Modeling Language: Created specifically for designing flexible and controllable dialogue flows, it has a Python-like syntax that is simple and intuitive, making it suitable for developers. It supports Colang 1.0 and 2.0.
• CLI: The library comes with a built-in command-line tool that can be used to start the server and perform evaluations, among other operations.
• LangChain Integration (Optional): Enabled by setting the NEMOGUARDRAILS_LLM_FRAMEWORK=langchain environment variable or calling set_default_framework("langchain"). Once enabled, you can wrap a LangChain chain (or any Runnable) with a guardrails configuration, and you can also call LangChain chains within a guardrails configuration.

Use Cases

• Document-based question answering (RAG): Perform fact-checking and output moderation.
• Domain-specific assistants (chatbots): Ensure the assistant stays on topic and follows the designed dialogue flow.
• LLM endpoints: Add guardrails to custom LLMs to enable safer customer interactions.
• LangChain chains (optional): In scenarios where LangChain is used, a guardrails layer can be added around the chain.

Built-in Guardrails Library

NeMo Guardrails comes with a set of built-in guardrails, including: LLM self-checking (input/output moderation, fact-checking, hallucination detection), NVIDIA safety models (content safety, topic safety), jailbreak and injection detection, as well as integrations with community models and third-party APIs (such as ActiveFence, PolicyAI, AlignScore, and LangChain chains).

Evaluation and Safety

The library provides the evaluation tool nemoguardrails evaluate, supporting topical guardrails, fact-checking, moderation (jailbreak and output moderation), and hallucination evaluation, along with an example LLM vulnerability scan report.

Supported LLM Models

Multiple LLMs can be used, such as OpenAI GPT-3.5, GPT-4, LLaMa-2, Falcon, Vicuna, or Mosaic. For details, see the Supported LLM Models section in the configuration guide.

Telemetry and Privacy

The library collects anonymous telemetry data to help NVIDIA understand deployment patterns and the usage of safety features. A usage event is emitted when LLMRails, IORails, or Guardrails is instantiated, after which a single daemon thread periodically sends heartbeats per process. Telemetry data includes: the installed library version, Python version, operating system and platform string; the Colang configuration language version; the configured LLM engine provider name (such as openai, nim, nvidia_ai_endpoints, never including model names or credentials); counts of various guardrail flows and activated guardrail categories; names of activated built-in library features; the count of user-defined Colang flows; whether Tracing, streaming, or a knowledge base is configured; the deployment method (library, api, or cli server); the runtime guardrail engine used; and a random per-process UUID used to correlate events from the same instance. Event payloads do not collect user content and do not include model names, API keys, endpoints, prompts, completions, token counts, per-request metrics, file paths, usernames, or IP addresses. The library also attempts to write each event payload to a local audit file ~/.config/nemoguardrails/usage_stats.json. Telemetry can be disabled by setting an environment variable or by creating do_not_track before the library starts. Third-party endpoints (such as NVIDIA Build) have separate terms and privacy practices; NVIDIA Build is intended only for evaluation and testing and must not be used in production environments.

License and Citation

NeMo Guardrails is licensed under the Apache License, Version 2.0. When using this library, please cite the EMNLP 2023 paper that introduces it.

Latest Version

The latest released version is 0.24.1. The develop branch tracks the latest development code. The official documentation is located at docs.nvidia.com/nemo/guardrails.