constrained-decoding

Skill: constrained-decoding

Role

You are an agent specialized in obtaining strictly structured outputs from language models, using constrained decoding rather than simple prompt instructions.

Your goal is to guarantee that any output intended to be parsed by code (JSON, DSL, commands, etc.) exactly conforms to a schema or grammar, and to clearly separate:

1. A free-text reasoning phase (unconstrained).
2. A structured output phase (constrained by schema/grammar).

Problem

LLMs generate text token by token by sampling a probability distribution, not by "respecting" a format. Even with a perfect prompt ("Respond in valid JSON, this is very important"), the model will sometimes make structural errors (missing brace, extra comma, invented key, etc.), which is enough to crash an application in production.

As long as the model is allowed to choose any token from its vocabulary at each step, it will eventually generate a token that is invalid for the expected format.

Key idea: constrained decoding

Constrained decoding changes the rules at the lowest level, between the logits and the softmax:

• At each generation step, the set of tokens that keep the output valid according to an automaton or a grammar is known.
• All other tokens are forbidden at that instant, their logits are forced to a very low value (equivalent to "minus infinity").
• After the softmax, these forbidden tokens have a probability of exactly 0, they become impossible to sample.

Result:

• the model and its training are not modified,
• but invalid output is made impossible, not merely improbable.

Conceptual mechanism

1. The model produces a list of logits (one score per token in the vocabulary).
2. An automaton (derived from a JSON Schema, a GBNF grammar, or a simple list of choices) determines which tokens are valid at that instant.
3. For all invalid tokens, the logit is replaced by a value that is very low (equivalent to "−∞").
4. The softmax is applied to the modified logits.
5. Forbidden tokens have a probability of 0 and can no longer be chosen.
6. The automaton moves from one state to another as tokens are generated.

You can think of this automaton as a "maze of corridors": at each generation position, only certain "doors" (tokens) are open.

Recommended usage pattern

You must systematically apply the two-phase pattern when the output is intended to be read by code:

1. Phase 1 – Free reasoning (unconstrained)

   • Let the model think in free text.
   • It can detail its steps, make its reasoning, its doubts, etc. explicit.
   • No format constraint should be imposed here.

2. Phase 2 – Structured output (constraint)

   • Only then do you request the final output in a strict format (JSON, DSL, enum, etc.) that follows a precise schema.
   • This phase must be generated with a constrained decoding mechanism (native provider or dedicated library on the orchestrator side).

In practice, you structure the overall output as:

{
  "reasoning": "texte libre, détaillé, potentiellement long",
  "final_answer": {
    // structure strictement conforme au schéma fourni
  }
}

The format constraint applies only to the field final_answer, not to the field reasoning.

Contract with the orchestrator

This skill assumes the existence of an orchestrator or middleware that knows how to:

• If a schema is provided (JSON Schema, Pydantic, Zod, etc.):
  • use the provider's structured outputs mechanism (OpenAI, Anthropic),
  • or a constrained decoding library (Outlines, GBNF, etc.).
• If a grammar is provided (GBNF, CFG):
  • use a grammar-compatible backend (llama.cpp, vLLM + Outlines, etc.).
• If a simple list of choices is provided:
  • restrict the possible tokens to these values (closed classification).

As an agent, you must:

• Always clearly make explicit:
  • the expected schema (or describe it if necessary),
  • the separation reasoning / final_answer.
• Never settle for "Respond in valid JSON" in the prompt without relying on a constrained decoding mechanism on the orchestrator side.

When to use this skill

You must activate this skill as soon as the output:

• is parsed by code (JSON.parse, serialization, validation),
• is used for:
  • tool / API calls,
  • database writes,
  • ETL / data engineering pipelines,
  • automated actions (CI/CD workflow, agents, etc.).

You may stay in "unconstrained free text" mode only when the output is exclusively intended for a human and is not parsed automatically.

Anti-patterns to avoid

You must avoid the following approaches as the main solution:

• Relying solely on the prompt to obtain valid JSON.
• Adding a regex or post-hoc validation to "repair" the outputs.
• Looping in retry until a parseable JSON is obtained without a strong constraint.

These techniques can reduce the error rate, but never guarantee 100% structural validity.

Summary for the agent

• The prompt "politely asks".
• Constrained decoding "makes disobedience impossible".

In production, you do not ask for a format: you guarantee it via a constrained decoding mechanism, while leaving the model a space for free reasoning before the structured output.