use llguidance library for constraints (including json schemas)

[mmoskal]

Fixes #754

This imports the llguidance library for dealing with constraints (regexp, json schema, CFGs in lark syntax) and removes the aici submodule which was implementing regexps and CFGs (in yacc syntax) before. The code for both is related (I wrote the original versions of both).

This is WIP - putting it out there for comments about how likely it is to be merged.

TODO

• remove testing scriptgs (scripst/*.sh)
• remove aici folder
• update python APIs to accept "json_schema"
• update examples

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 Happy                   1          442          369            0           73
 JSON                   12          105          104            0            1
 Python                 51         2205         1885           63          257
 Shell                   1           57           22           18           17
 TOML                   18          575          513            2           60
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          196          169            1           26
 (Total)                            273          201           32           40
-------------------------------------------------------------------------------
 Markdown               39         2805            0         2132          673
 |- BASH                 6          101           98            0            3
 |- JSON                 1           12           12            0            0
 |- Python               6          114          102            0           12
 |- Rust                 9          322          274            0           48
 |- TOML                 2           75           63            0           12
 (Total)                           3429          549         2132          748
-------------------------------------------------------------------------------
 Rust                  271        79755        71606         1690         6459
 |- Markdown           132         1359           25         1239           95
 (Total)                          81114        71631         2929         6554
===============================================================================
 Total                 403        86041        74568         3917         7556
===============================================================================
  

[EricLBuehler]

Hi @mmoskal! Thanks for the PR.

I took a look at the code and it looks great already. As you noted, adding some examples and updating the APIs + docs would be great (maybe some tests?). After that, this is something I would certainly be excited to merge.

[mmoskal]

Thanks for the review! I probably won't have time for it this week, but hopefully the one after that.

[EricLBuehler]

@mmoskal friendly ping :) Let me know if I can help!

[mmoskal]

It seems I won't have much time in the next two-three weeks. If anyone else is willing to take on the TODOs that would be great!

I'll be extending the llguidance APIs to be able to perform multithreaded mask computation in background (one of the things I will be busy with). Enabling this will get the constraint computation out of the hot path of sampling - but that should be a one-line change here.