Issue running steps in Quick Start guide

[nathan-weinberg]

I am having an issue running the steps in the Quick Start guide found here: https://llama-stack.readthedocs.io/en/latest/getting_started/index.html

I have a local llama3.2 model running

$ ollama ps
NAME                         ID              SIZE      PROCESSOR    UNTIL               
llama3.2:3b-instruct-fp16    195a8c01d91e    8.0 GB    100% CPU     48 minutes from now 

But when I attempt to the run the subsequent Docker image, I get the following error:

$ docker run -it \                                                                                 
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \                                                                                              
  -v ~/.llama:/root/.llama \                                                                                                                                  
  llamastack/distribution-ollama \                                                                                                      
  --port $LLAMA_STACK_PORT \                                                                                                                                  
  --env INFERENCE_MODEL=$INFERENCE_MODEL \                                                                                              
  --env OLLAMA_URL=http://host.docker.internal:11434                                                                                                          
Setting CLI environment variable INFERENCE_MODEL => meta-llama/Llama-3.2-3B-Instruct                                                    
Setting CLI environment variable OLLAMA_URL => http://host.docker.internal:11434                                                                              
Using template ollama config file: /usr/local/lib/python3.10/site-packages/llama_stack/templates/ollama/run.yaml    
...
Traceback (most recent call last):                                                                                                      
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                                 
    return _run_code(code, main_globals, None,                                                                                          
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code  
    exec(code, run_globals)                                                                                                             
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/server/server.py", line 426, in <module>                                             
    main()                                                                                                                              
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/server/server.py", line 349, in main                                                 
    impls = asyncio.run(construct_stack(config))                                                                                        
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run                                                                                        
    return loop.run_until_complete(main)                                                                                                
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete                                                                    
    return future.result()                                                                                                              
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/stack.py", line 208, in construct_stack                                              
    impls = await resolve_impls(                                                                                                        
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/resolver.py", line 244, in resolve_impls                                             
    impl = await instantiate_provider(                                                                                                  
  File "/usr/local/lib/python3.10/site-packages/llama_stack/distribution/resolver.py", line 331, in instantiate_provider                                      
    impl = await fn(*args)                                                                                                              
  File "/usr/local/lib/python3.10/site-packages/llama_stack/providers/remote/inference/ollama/__init__.py", line 14, in get_adapter_impl                      
    await impl.initialize()                                                                                                             
  File "/usr/local/lib/python3.10/site-packages/llama_stack/providers/utils/telemetry/trace_protocol.py", line 101, in async_wrapper                          
    result = await method(self, *args, **kwargs)                                                                                        
  File "/usr/local/lib/python3.10/site-packages/llama_stack/providers/remote/inference/ollama/ollama.py", line 149, in initialize                             
    await self.client.ps()                                                                                                              
  File "/usr/local/lib/python3.10/site-packages/ollama/_client.py", line 1121, in ps                                                                          
    return await self._request(                                                                                                         
  File "/usr/local/lib/python3.10/site-packages/ollama/_client.py", line 682, in _request                                               
    return cls(**(await self._request_raw(*args, **kwargs)).json())                                                                     
  File "/usr/local/lib/python3.10/site-packages/ollama/_client.py", line 628, in _request_raw                                           
    raise ConnectionError(CONNECTION_ERROR_MESSAGE) from None                                                                           
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download                     

[mertpaker]

It works on Mac. What is your OS? Can any other container access the Ollama endpoint? You can try one of these to test.

Linux:

docker run --rm --network=host curlimages/curl:latest curl -X GET http://localhost:11434/api/ps | jq .

Mac or Windows

docker run --rm curlimages/curl:latest curl -X GET http://host.docker.internal:11434/api/ps | jq .

[nathan-weinberg]

I am on Fedora 40 - seems I can indeed hit that endpoint

$ docker run --rm --network=host curlimages/curl:latest curl -X GET http://localhost:11434/api/ps | jq .
{
  "models": [
    {
      "name": "llama3.2:3b-instruct-fp16",
      "model": "llama3.2:3b-instruct-fp16",
      "size": 8044877824,
      "digest": "195a8c01d91ec3cb1e0aad4624a51f2602c51fa7d96110f8ab5a20c84081804d",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": [
          "llama"
        ],
        "parameter_size": "3.2B",
        "quantization_level": "F16"
      },
      "expires_at": "2025-02-05T10:37:26.745569761-05:00",
      "size_vram": 0
    }
  ]
}

[mertpaker]

In that case, I think you need to update the connection parameters:

• add --network=host
• replace host.docker.internal with localhost.

This should work

docker run -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  --network=host \
  llamastack/distribution-ollama \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env OLLAMA_URL=http://localhost:11434

[nathan-weinberg]

Hmmm, it gave me a different error

$ docker run -it -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT -v ~/.llama:/root/.llama llamastack/distribution-ollama --network=host --port $LLAMA_STACK_PORT --env INFERENCE_MODEL=$INFERENCE_MODEL --env OLLAMA_URL=http://localhost:11434
usage: server.py [-h] [--yaml-config YAML_CONFIG] [--template TEMPLATE] [--port PORT] [--disable-ipv6] [--env ENV]
server.py: error: unrecognized arguments: --network=host

For reference, this is the image I'm using

REPOSITORY                                                TAG       IMAGE ID       CREATED         SIZE
llamastack/distribution-ollama                            latest    704a467c8585   3 days ago      1.75GB

[mertpaker]

Sorry, I edited my first command. --network=host should be before the image definition.

Can you please try this one?

docker run -it \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  --network=host \
  llamastack/distribution-ollama \
  --port $LLAMA_STACK_PORT \
  --env INFERENCE_MODEL=$INFERENCE_MODEL \
  --env OLLAMA_URL=http://localhost:11434

[nathan-weinberg]

This worked for me! Trying to understand the differences some more here - is this mainly a difference in Operating Systems? Worth creating an update to the docs for this?

[mertpaker]

It's not related to Llama-stack. It's a networking setting. 🤷‍♂️

By default, Docker containers run in their own isolated network namespace on Linux. To allow the container to communicate with services running on the host via localhost, you need --network=host. This makes the container use the host’s network directly so it can connect to Ollama running on localhost:11434. This is not required on Mac and Windows.

[nathan-weinberg]

Right, not specifically related to Llama Stack, but I figure any Linux user would hit this when trying to run through the Quick Start guide

[nathan-weinberg]

Opened #971

[booxter]

@nathan-weinberg does your ollama server serve on the same port? (11434) (probably depends how you run ollama)

[nathan-weinberg]

It should - I haven't modified the port env vars at all, and what I'm reading states that 11434 should be the default port. Any other way I could verify this?