Ollama integration

[anakin87]

Ollama is a good and popular project to run LLMs easily.
It uses a Modelfile, a similar concept to Dockerfile.
The model, when launched, exposes a REST API (documented here).

Integrating it in Haystack 2.0 would be great, creating an OllamaGenerator.
It would also not require external dependencies!

If anyone wants to implement the integration, please comment below and we can discuss it together...

Minimal implementation

import logging
from typing import Optional, List, Dict, Any
import requests
from haystack import component

@component
class OllamaGenerator:
    def __init__(
        self,
        model_name: str = "orca-mini",
        url: str = "http://localhost:11434/api/generate",
        generation_kwargs: Optional[Dict[str, Any]] = None,
    ):
        self.model_name = model_name
        self.url = url
        self.generation_kwargs = generation_kwargs or {}


    @component.output_types(replies=List[str], metadata=List[Dict[str, Any]])
    def run(self, prompt: str, generation_kwargs: Optional[Dict[str, Any]] = None):

        generation_kwargs = {**self.generation_kwargs, **(generation_kwargs or {})}

        json_response = requests.post(
            url=self.url,
            json={
                "prompt": prompt,
                "model": self.model_name,
                "stream": False,
                "options": generation_kwargs,
            },
        ).json()

        return {
            "replies": json_response["response"]
        }

>>> from haystack.components.generators.ollama import OllamaGenerator
>>> og = OllamaGenerator()
>>> og.run(prompt="What can you tell me of capybaras?")
{'replies': ' Capybaras are large, semi-aquatic rodents native to South America. They have a distinctive appearance with their stocky bodies, large ears, and long tails. They are herbivores and feed on a variety of vegetation such as leaves, fruits, and grasses. Capybaras are known for their calm and peaceful nature, making them popular pets in some areas. They can grow up to 5 feet tall and weigh up to 160 pounds.'}

Design

A proper component should be similar to GPTGenerator and TGIGenerator.

I can imagine something similar to this:

@component
class OllamaGenerator:
    def __init__(
        self,
        model_name: str = "orca-mini",
        url: str = "http://localhost:11434/api/generate",
        generation_kwargs: Optional[Dict[str, Any]] = None,	 # Ollama options
        system_prompt: Optional[str] = None,  # Ollama system
        template: Optional[str] = None,  # Ollama template
        raw = False, # Ollama raw
        streaming_callback: Optional[Callable[[StreamingChunk], None]] = None, # A callback function that is called when a new token is received from the stream.
    ):
    ...

    @component.output_types(replies=List[str], metadata=List[Dict[str, Any]])
    def run(self, prompt: str, generation_kwargs: Optional[Dict[str, Any]] = None):
    ...
    # metadata should contain the generation data, shown in https://github.com/jmorganca/ollama/blob/main/docs/api.md#examples

[AlistairLR112]

Definitely something I think should be implemented as LangChain has this

[AlistairLR112]

I will have a go tomorrow morning

[anakin87]

@AlistairLR112 if you want to try, you are welcome and we can help you!

One thing I forgot: like all new external integrations for 2.x, this one should live on https://github.com/deepset-ai/haystack-core-integrations.

if something is not clear, let's talk about it here or on Discord...

[AlistairLR112]

Fantastic! Let's chat tomorrow! If I use LLMs too close to bed.. I will be dreaming in prompts!

[sachinsachdeva]

This would a useful feature , in case you haven't already started I can give it a go.

[anakin87]

@sachinsachdeva I think you can give it a try...

Please consider my minimal implementation and the fact that this integration should be created under https://github.com/deepset-ai/haystack-core-integrations.

[sachinsachdeva]

@anakin87 opened Draft pull request, you want continue scope discussion here or on the PR ?

[anakin87]

Thanks! Let's discuss on the PR...

[AlistairLR112]

Apologies all, Christmas family commitments got in the way!

[AlistairLR112]

deepset-ai/haystack-core-integrations#132 lets combine our attempts

[anakin87]

Sorry, I have been very busy...
I'll take a look at it after Christmas so we can make some progress!

[masci]

Closing in favor of deepset-ai/haystack-core-integrations#146 let's continue the conversation there