OpenLLM cant connect to local server with error: "module 'openllm' has no attribute 'client'"

[SmolPandaDev]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

All the code for a minimum reproducible example is here: https://github.com/SmolPandaDev/openllm-issue

Description

I'm trying to use the langchain community library for openllm to connect to a locally running LLM and then ask the LLM a simple question.

System Info

Output of pip3 freeze | grep langchain:

langchain==0.2.13
langchain-community==0.2.12
langchain-core==0.2.30
langchain-openai==0.1.21
langchain-text-splitters==0.2.2
langchainhub==0.1.21

Platform: MacOS (14.5 (23F79))

Output of: python3 --version
Python 3.11.6

[dosubot[bot]]

Hey there, @SmolPandaDev! I'm a bot designed to help you solve bugs, answer questions, and become a contributor. Let me know how I can assist you with the OpenLLM issue. 🤖

To resolve the error "module 'openllm' has no attribute 'client'" and successfully connect to the local LLM using the LangChain community library for OpenLLM, you need to ensure that you are using the correct module and attribute names. Based on the provided code, it appears that the correct module to import is langchain_community.llms, and the attribute to use is OpenLLM.

Here is the relevant part of the code that shows the correct import and attribute usage:

from langchain_community.llms import OpenLLM

llm = OpenLLM(
    model_name='flan-t5',
    model_id='google/flan-t5-large',
)
llm.invoke("What is the difference between a duck and a goose?")

Make sure you have the langchain_community package installed:

pip install langchain_community

Then, ensure your code is using OpenLLM from the langchain_community.llms module. This should resolve the attribute error and allow you to connect to the local LLM using the LangChain community library for OpenLLM. The langchain_community.llms module dynamically looks up attributes, including OpenLLM, ensuring compatibility and proper import handling ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[SmolPandaDev]

No, this doesn't work it complains about a runner not being found which is same as this issue: #25243

[dosubot[bot]]

To resolve the AttributeError: module 'openllm' has no attribute 'Runner', ensure that you have the correct version of the openllm library installed. The Runner class is expected to be part of the openllm module as used in the OpenLLM class from the langchain_community.llms module.

Here is the relevant part of the OpenLLM class that uses openllm.Runner:

runner = openllm.Runner(
    model_name=model_name,
    model_id=model_id,
    init_local=embedded,
    ensure_available=True,
    **llm_kwargs,
)

Make sure you have installed the openllm library correctly:

pip install openllm

If the issue persists, verify that the openllm library version you are using includes the Runner class. You might need to update the library to the latest version:

pip install --upgrade openllm

If the Runner class is still not found, it could be a version mismatch or a missing feature in the openllm library. In that case, you may need to check the library's documentation or repository for further details.

Additionally, ensure that the OpenLLM class is being correctly imported from the langchain_community.llms module. The langchain library uses a dynamic import mechanism to handle deprecated imports, as shown below:

def _import_openllm() -> Any:
    from langchain_community.llms.openllm import OpenLLM

    return OpenLLM

def __getattr__(name: str) -> Any:
    from langchain_community import llms

    if name == "type_to_cls_dict":
        type_to_cls_dict: Dict[str, Type[BaseLLM]] = {
            k: v() for k, v in get_type_to_cls_dict().items()
        }
        return type_to_cls_dict
    else:
        return getattr(llms, name)

This mechanism ensures that the OpenLLM class is correctly imported even if it has been moved or renamed. Make sure your import statements align with this structure ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[SmolPandaDev]

I think maybe this dynamic import is failing

[sidm-23]

Hi, im also facing this issue in version 0.3.5

python3 -m pip show langchain-community
Name: langchain-community
Version: 0.3.5
Summary: Community contributed LangChain integrations.
Home-page: https://github.com/langchain-ai/langchain
Author:
Author-email:
License: MIT
Location: /root/llm/lib/python3.9/site-packages
Requires: aiohttp, dataclasses-json, httpx-sse, langchain, langchain-core, langsmith, numpy, pydantic-settings, PyYAML, requests, SQLAlchemy, tenacity
Required-by:

Error: