llamacpp_example is broken

[bharathc346]

Describe the issue as clearly as possible:

examples/llamcpp_example.py is broken

It seems like the model is producing some garbage output (which shouldnt be allowed by the logit processor). Posted my debug below.

Steps/code to reproduce the bug:

from enum import Enum

from pydantic import BaseModel, constr

import outlines
import llama_cpp


class Weapon(str, Enum):
    sword = "sword"
    axe = "axe"
    mace = "mace"
    spear = "spear"
    bow = "bow"
    crossbow = "crossbow"


class Armor(str, Enum):
    leather = "leather"
    chainmail = "chainmail"
    plate = "plate"


class Character(BaseModel):
    name: constr(max_length=10)
    age: int
    armor: Armor
    weapon: Weapon
    strength: int


if __name__ == "__main__":
    # curl -L -o mistral-7b-instruct-v0.2.Q5_K_M.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.gguf
    model = outlines.models.LlamaCpp(
        model=llama_cpp.Llama(model_path="./mistral-7b-instruct-v0.2.Q5_K_M.gguf")
    )

    # Construct structured sequence generator
    generator = outlines.generate.json(model, Character)

    # Draw a sample
    seed = 789005

    prompt = "Instruct: You are a leading role play gamer. You have seen thousands of different characters and their attributes.\nPlease return a JSON object with common attributes of an RPG character. Give me a character description\nOutput:"

    sequence = generator(prompt, seed=seed, max_tokens=512)
    print(sequence)

Expected result:

Valid Character Type

Error message:

Traceback (most recent call last):
  File "/Users/bharathc/Projects/temp.py", line 48, in <module>
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/ap
i.py", line 512, in __call__
    return self._format(completions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/ap
i.py", line 488, in _format
    return self.format_sequence(sequences)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/outlines/generate/js
on.py", line 50, in <lambda>
    generator.format_sequence = lambda x: schema_object.parse_raw(x)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bharathc/Projects/.env/lib/python3.12/site-packages/pydantic/main.py", l
ine 1214, in parse_raw
    raise pydantic_core.ValidationError.from_exception_data(cls.__name__, [error])
pydantic_core._pydantic_core.ValidationError: 1 validation error for Character
__root__
  Expecting value: line 1 column 1 (char 0) [type=value_error.jsondecode, input_value='Abstract ERROR
 ERRORAbst...bstractAbstractAbstract', input_type=str]

Outlines/Python version information:

Outlines version
0.1.1

Python version
Python 3.12.5 (main, Aug 6 2024, 19:08:49) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Context for the issue:

I put a breakpoint here

and it seems the value of completions
is

(Pdb) completions
'Abstract ERROR ERRORAbstract ERROR ERRORAbstract ERRORAbstract ERRORAbstract ERRORAbstract ERRORAbst
ract ERRORAbstractAbstract ERRORAbstractAbstract ERRORAbstractAbstractAbstract ERRORAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstract
AbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstr
actAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAb
stractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstrac
tAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbst
ractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractA
bstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstra
ctAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbstractAbs
tractAbstractAbstractAbstractAbstractAbstractAbstractAbstract'

So not sure whats going on, I guess llama cpp is not using the logit processor correctly?

[axel-at-drom]

I'm having a similar experience trying the named entity extraction example from the cookbook using llama_cpp with a local model (zephyr-7b) on an M1 Mac.

Here is the interesting part of the error:
ValidationError: 1 validation error for Order __root__ Expecting value: line 1 column 1 (char 0) [type=value_error.jsondecode, input_value='AbstractAbstractAbstract...bstractAbstractAbstract', input_type=str]

Looks like outlines is making the model output lots of 'Abstract'.

Outlines version 0.1.3.
Python version 3.11.7
llama_cpp_python version 0.3.1