Add support for context-free grammar-constrained generation for language models

[rbaks]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Would be great to have a way to feed context-free grammars to language models for constraining their token space pre-generation. Very much like arleady does parserllm with HuggingFace models.

Unlike with with_structured_output, this would allow to generate accurate, not only JSON output we cannot model from mere pydantic classes.

Motivation

Direct application would be Text-To-SQL where LLMs would consistently generate syntactically correct SQL, up to the column names, given the grammar as further explained in this medium article.

Other applications would be code generation for rare programming and configuration languages.

Proposal (If applicable)

No response