guidance

Guidance is an efficient programming paradigm for guiding language models. With Guidance, you can control the structure of the output and obtain high-quality output for your use case, while reducing latency and cost compared to traditional prompting or fine-tuning. It allows users to constrain generation (for example, using regular expressions and context-free grammars) and to seamlessly interleave control flow (conditionals, loops, tool use) and generation.

Main Capabilities

• Constrained generation: Use regular expressions or context-free grammars (CFG) to constrain model output, ensuring that the output conforms to the expected syntax. For example,gen() calls can be constrained to match a regular expression;select() functions are used to select output from a predefined list.
• Interleaving control and generation: Embed conditionals, loops, and tool calls during generation to achieve complex logic control.
• Custom Guidance functions: Use the @guidance decorator to create your own functions, which can interact with language models and can be composed to build a complete context-free grammar.
• JSON generation: JSON schema is essentially a context-free grammar, and Guidance provides special support for this, making it possible to generate JSON based on Pydantic models and the like.
• Fast forward tokens: Because of grammar constraints, some tokens can be determined in advance, and Guidance does not need the model to generate these tokens but instead inserts them directly, thereby saving forward passes and reducing GPU usage.
• Offline grammar debugging: When iterating on constraints, you can validate candidate strings locally and test complete runs using a Mock model, without calling a model API.

Usage

Guidance can be installed via PyPI. If you already have the backend required by the model, you only need to run the installation command. It supports multiple backends, including Transformers, llama.cpp, OpenAI, and more.

In Python, you can interact with large language models using common Python idioms. For example, running it on the command line will generate output; in a Jupyter notebook, Guidance provides widgets for a richer user experience. Capturing the generated text is also very easy.

When creating custom Guidance functions, use the @guidance decorator to mark the function, and thelanguage_model parameter will be filled automatically. These functions can be used like gen() or select() , and can be composed to build a complete context-free grammar. For example, you can create functions to generate text that does not contain HTML tags, generate text inside any HTML tag, generate a page title, define headings and paragraphs in the body, and ultimately generate a complete HTML page.

Applicable Scenarios

• Scenarios that require precise control over the structure of language model output, such as generating JSON, HTML, and other specific formats.
• Complex tasks that require interleaving logical control with generation, such as multi-turn question answering and tool use.
• Applications that hope to reduce latency and cost while obtaining high-quality output.

Limitations

• Constrained generation (such as context-free grammars) requires the backend LLM to fully support Guidance.
• For the specific list of supported backends and installation details, please refer to the official documentation.
• Information about fees, pricing, platforms, and similar details is not currently provided in the materials.