Not able to reproduce the results provided at https://blog.dottxt.co/coalescence.html

[chsharma27]

Describe the issue as clearly as possible:

I need to reproduce the results provided at https://blog.dottxt.co/coalescence.html, but getting a lot of errors, which are due to changes in the code. Can you please provide the updated code to reproduce the results?

Steps/code to reproduce the bug:

import outlines.fsm as fsm
import json

regex_str = fsm.json_schema.build_regex_from_object(json.dumps(json_schema))
regex_str

Expected result:

'\\\\{"name":("John"|"Paul"),"age":(20|30)\\\\}'

Error message:

AttributeError: module 'outlines.fsm.json_schema' has no attribute 'build_regex_from_object'

Outlines/Python version information:

python -c "from outlines import _version; print(_version.version)"
0.1.1

python -c "import sys; print('Python', sys.version)"
Python 3.10.13 (main, Aug 25 2023, 13:20:03) [GCC 9.4.0]

pip freeze
absl-py==1.4.0
aiohttp==3.8.5
aiosignal==1.3.1
airportsdata==20241001
albumentations==1.3.1
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.2.0
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
artifactory==0.1.17
astroid==2.15.6
asttokens==2.4.1
astunparse==1.6.3
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.1.0
Babel==2.14.0
beautifulsoup4==4.12.2
bleach==6.1.0
boto3==1.28.35
botocore==1.31.35
braceexpand==0.1.7
Brotli==1.0.9
cachetools==5.3.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.2.0
clean-fid==0.1.6
click==8.1.7
clip @ git+https://github.com/openai/CLIP.git@a1d071733d7111c9c014f024669f959182114e33
cloudpickle==3.1.0
cmake==3.27.2
comm==0.2.1
contourpy==1.1.0
coverage==7.3.0
cryptography==41.0.7
cupy-cuda12x==12.2.0
cycler==0.11.0
Cython==3.0.1
datasets==3.1.0
debugpy==1.6.7.post1
decorator==5.1.1
deepspeed==0.10.1
defusedxml==0.7.1
diffusers @ git+https://github.com/huggingface/diffusers.git@2764db3194fc1b5069df7292fd938657d8568995
dill==0.3.8
diskcache==5.6.3
docker-pycreds==0.4.0
docutils==0.20.1
dohq-artifactory==0.9.0
dominate==2.8.0
einops==0.6.1
einops-exts==0.0.4
exceptiongroup==1.1.3
executing==2.0.1
fastapi==0.103.0
fastjsonschema==2.19.1
fastrlock==0.8.1
filelock==3.12.2
flake8==6.1.0
flatbuffers==23.5.26
fonttools==4.42.1
fqdn==1.5.1
frozenlist==1.4.0
fsspec==2023.6.0
ftfy==6.1.1
gast==0.5.4
gitdb==4.0.10
GitPython==3.1.32
google-auth==2.22.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
grpcio==1.57.0
h11==0.14.0
h5py==3.10.0
huggingface-hub==0.26.2
idna==3.4
imageio==2.31.1
importlib-metadata==6.8.0
iniconfig==2.0.0
interegular==0.3.3
ipykernel==6.28.0
ipython==8.20.0
ipywidgets==8.1.1
isoduration==20.11.0
isort==5.12.0
jaraco.classes==3.3.0
jedi==0.19.1
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.3.2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.20.0
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.9.0
jupyter-lsp==2.2.1
jupyter_client==8.6.0
jupyter_core==5.7.1
jupyter_server==2.12.4
jupyter_server_terminals==0.5.1
jupyterlab==4.0.10
jupyterlab-widgets==3.0.9
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.2
keras==2.15.0
keyring==24.3.0
kiwisolver==1.4.5
lark==1.2.2
lazy-object-proxy==1.9.0
lazy_loader==0.3
libclang==16.0.6
lightning-utilities==0.9.0
lit==16.0.6
lmdb==1.4.1
lz4==4.3.2
Markdown==3.4.4
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.7.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mdurl==0.1.2
mistune==3.0.2
ml-dtypes==0.2.0
more-itertools==10.2.0
mpld3==0.5.9
mpmath==1.3.0
multidict==6.0.4
multiprocess==0.70.16
natsort==8.2.0
nbclient==0.9.0
nbconvert==7.14.1
nbformat==5.9.2
nest-asyncio==1.5.9
networkx==3.1
nh3==0.2.15
notebook==7.0.6
notebook_shim==0.2.3
numpy==1.25.2
nvidia-cublas-cu12==12.2.5.6
nvidia-cuda-cupti-cu12==12.2.142
nvidia-cuda-nvrtc-cu12==12.2.140
nvidia-cuda-runtime-cu12==12.2.140
nvidia-cudnn-cu12==8.9.4.25
nvidia-cufft-cu12==11.0.8.103
nvidia-curand-cu12==10.3.3.141
nvidia-cusolver-cu12==11.5.2.141
nvidia-cusparse-cu12==12.1.2.141
nvidia-ml-py3==7.352.0
nvidia-nccl-cu12==2.18.3
nvidia-nvtx-cu12==12.2.140
oauthlib==3.2.2
omegaconf==2.3.0
opencv-contrib-python==4.8.0.76
opencv-python==4.8.0.76
opencv-python-headless==4.8.0.76
opt-einsum==3.3.0
outlines==0.1.1
outlines_core==0.1.14
overrides==7.4.0
packaging==23.1
pandas==2.0.3
pandocfilters==1.5.0
parso==0.8.3
pathlib==1.0.1
pathtools==0.1.2
pexpect==4.9.0
Pillow==10.0.0
Pillow-SIMD==9.0.0.post1
pkginfo==1.9.6
platformdirs==3.10.0
pluggy==1.2.0
prometheus-client==0.19.0
prompt-toolkit==3.0.43
protobuf==4.23.4
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==18.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycodestyle==2.11.0
pycountry==24.6.1
pycparser==2.21
pydantic==2.9.2
pydantic_core==2.23.4
pyDeprecate==0.3.2
pyflakes==3.1.0
Pygments==2.17.2
PyJWT==2.8.0
pylint==2.17.5
pynvml==11.5.0
pypandoc==1.11
pyparsing==3.0.9
pypiserver==2.0.1
pytest==7.4.0
python-dateutil==2.8.2
python-json-logger==2.0.7
pytorch-lightning==2.0.7
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0.1
pyzmq==25.1.1
qoi==0.3.1
qtconsole==5.5.1
QtPy==2.4.1
qudida==0.0.4
readme-renderer==42.0
referencing==0.32.1
regex==2023.8.8
requests==2.32.3
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986==2.0.0
rfc3986-validator==0.1.1
rich==13.7.0
rotary-embedding-torch==0.2.7
rpds-py==0.17.1
rsa==4.9
s3transfer==0.6.2
safetensors==0.4.5
scikit-image==0.21.0
scikit-learn==1.3.0
scipy==1.11.2
SecretStorage==3.3.3
Send2Trash==1.8.2
sentry-sdk==1.29.2
setproctitle==1.3.2
setuptools-git-versioning==1.13.5
six==1.16.0
slack-bolt==1.18.0
slack-sdk==3.21.3
smmap==5.0.0
snappy==3.1.1
sniffio==1.3.0
soupsieve==2.5
stack-data==0.6.3
sympy==1.12
tensorboard==2.15.1
tensorboard-data-server==0.7.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.15.0.post1
tensorflow-estimator==2.15.0
tensorflow-io-gcs-filesystem==0.35.0
termcolor==2.4.0
terminado==0.18.0
threadpoolctl==3.2.0
tifffile==2023.8.12
tinycss2==1.2.1
tokenizers==0.20.3
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.1
torch @ file:///opt/pytorch/dist/torch-2.1.0-cp310-cp310-linux_x86_64.whl
torch-tb-profiler==0.4.1
torchaudio @ file:///opt/audio/dist/torchaudio-2.1.0a0%2B47eaab4-cp310-cp310-linux_x86_64.whl
torchdata==0.6.1
torchmetrics==1.1.0
torchvision @ file:///opt/vision/dist/torchvision-0.16.0a0%2B224cbc8-cp310-cp310-linux_x86_64.whl
tornado==6.4
tqdm==4.67.0
traitlets==5.14.1
transformers==4.46.2
triton @ git+https://github.com/openai/triton@e6216047b8b0aef1fe8da6ca8667a3ad0a016411#subdirectory=python
twine==4.0.2
types-python-dateutil==2.8.19.20240106
typing_extensions==4.7.1
tzdata==2023.3
uri-template==1.3.0
urllib3==1.26.16
uvicorn==0.23.2
wandb==0.15.8
wcwidth==0.2.6
webcolors==1.13
webdataset==0.2.48
webencodings==0.5.1
websocket-client==1.7.0
Werkzeug==2.3.7
widgetsnbextension==4.0.9
wrapt==1.14.1
xformers==0.0.22+a24a6e3.d20240107
xxhash==3.3.0
yarl==1.9.2
zipp==3.16.2
zstandard==0.21.0
zstd==1.5.5.1

Context for the issue:

As I said before, I need to reproduce the results so that I can use Outline library's restricted decoding in our use case.

[gitlost-murali]

Here's the updated code:

from enum import Enum
from pydantic import BaseModel
from outlines_core.fsm.json_schema import build_regex_from_schema
import interegular
from outlines.fsm.json_schema import convert_json_schema_to_str

class Name(str, Enum):
    john = "John"
    paul = "Paul"

class Age(int, Enum):
    twenty = 20
    thirty = 30

class Character(BaseModel):
    name: Name
    age: Age


import json

json_schema = Character.model_json_schema()
json_schema_str = convert_json_schema_to_str(json_schema)
regex_str = build_regex_from_schema(json_schema_str)
pattern = interegular.parse_pattern(regex_str)
fsm = pattern.to_fsm()

[chsharma27]

Thanks for your reply, but can you please tell me where/how to install/get this "outlines_core" module

[gitlost-murali]

Outlines-core is installed as a part of outlines library. A minimal pip install outlines would work.

[chsharma27]

Thanks, it works.