Guidance

Sweet spot: a team running local or open-source LLMs (llama.cpp, vLLM-served open models) that needs guaranteed structured output. OpenAI and Anthropic now ship first-class structured outputs, which took pressure off Guidance for hosted APIs — but for self-hosted models, Guidance and its siblings (Outlines, LMQL) remain the correct answer. Failure modes. If your stack is entirely hosted OpenAI / Anthropic, the providers' own structured-output features are simpler and just as reliable for most schemas. Template language requires investment — teams that expect a one-line decorator will find it heavier than that. And as a library, Guidance is a piece of a stack, not a full solution; you still need orchestration around it. What to pilot. Take one structured-extraction task on a local model where you currently post-process or retry. Rewrite it in Guidance. Measure: failure rate, wall-time per call, total tokens. If the guaranteed-schema property eliminates retries and speeds up generation, Guidance pays for its learning curve; if post-processing was already fast and rare, skip it.

Guidance is a Microsoft-developed Python library for constrained LLM generation. Instead of asking a model to "please return JSON" and hoping, you write a template with explicit structure — fixed text, typed slots, branches, loops — and Guidance constrains the model's token generation so the output is guaranteed to match the template. It works best with local and open-source models served via llama.cpp, Transformers, or vLLM, where Guidance can manipulate logits directly to prevent invalid tokens. It also supports OpenAI and Azure via best-effort constraint reflection (no logit access means weaker guarantees, but templates still help). Use cases include JSON/XML extraction where schema adherence matters, code generation where syntax must be valid, and multi-step reasoning where you want to force a specific format (chain-of-thought, then answer). Guidance is also notably faster than unconstrained sampling in many cases because it can skip generation for fixed template parts. Apache-2-licensed, originally released by Microsoft Research, now maintained by a mix of Microsoft and community contributors. It is the reference implementation of the "structured generation" approach that frameworks like Outlines and LMQL also explore.