Langchain not supporting OCI Generative AI Chatting, Only text generation

[SherifBashar]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.llms import OCIGenAI
from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate


# use default authN method API-key
llm = OCIGenAI(
    model_id="ocid1.generativeaimodel.oc1.eu-frankfurt",
    service_endpoint="https://inference.generativeai.eu-frankfurt-1.oci.oraclecloud.com",
    compartment_id="ocid1.compartment.*********,
    provider="cohere"
)

prompt = PromptTemplate(input_variables=["query"], template="{query}")
llm_chain = LLMChain(llm=llm, prompt=prompt)
response = llm_chain.invoke("what is the capital of france?")
print(response["text"])

Description

Hi,

Oracle has announced that OCI Generative AI is now available in a new region, Frankfurt, with chat capabilities. When I attempted to run the attached code, I encountered the following error:

'message': 'Model cohere.command-r-16k does not have TextGeneration capability.'

This is accurate since the model only supports chatting, not text generation. Therefore, I am inquiring about when Langchain will support both TextGeneration and Chatting capabilities.

System Info

langchain==0.1.1
langchain-community==0.0.20
langchain-core==0.1.23

Mac

Python 3.12.0

[davidmonteroarin]

Hi, I'm exactly in the same page. Working with langchain and OpenAI in the last week without any problems but when trying to use with Generative AI we have this error:

"Model cohere.command-r-16k does not have TextGeneration capability"

I've printed every available and not expired models and none of them have TextGeneration capability

capabilities: ['CHAT']
Model ID: ocid1.generativeaimodel.oc1.eu-frankfurt-1.amaaaaaask7dceyazi3cpmptwa52f7dgwyskloughcxtjgrqre3pngwtig4q
Model Name: cohere.command-r-16k
vendor: cohere

capabilities: ['CHAT', 'FINE_TUNE']
Model ID: ocid1.generativeaimodel.oc1.eu-frankfurt-1.amaaaaaask7dceyauo27aj64jvt3iqt3ztuylst7nnosax63rudq3atpel5q
Model Name: meta.llama-3-70b-instruct
vendor: meta

capabilities: ['CHAT']
Model ID: ocid1.generativeaimodel.oc1.eu-frankfurt-1.amaaaaaask7dceya47muknh2iwh2oc6ojcszkgwv4qrsrcjg3hqmddzcsrba
Model Name: meta.llama-3-70b-instruct
vendor: meta

capabilities: ['TEXT_EMBEDDINGS']
Model ID: ocid1.generativeaimodel.oc1.eu-frankfurt-1.amaaaaaask7dceyawsl7mx35xcad5t2spbwlchf2nr6xpd3sajgbt2vtwemq
Model Name: cohere.embed-english-v3.0
vendor: cohere

capabilities: ['TEXT_EMBEDDINGS']
Model ID: ocid1.generativeaimodel.oc1.eu-frankfurt-1.amaaaaaask7dceyacat4d2awdzdbgzw5yqbv2hzpayrd45qyxp3cjxmjwwpq
Model Name: cohere.embed-multilingual-v3.0
vendor: cohere

[davidmonteroarin]

I am afraid I already got the answer. Text Generation models are only available in Chicago region

https://docs.oracle.com/en-us/iaas/Content/generative-ai/pretrained-models.htm

We will have to keep waiting

[SherifBashar]

Yes, it is true. Text generation models are available in Chicago and are supported by Langchain. However, while chat models are available in both Chicago and Frankfurt, they are not yet supported by Langchain.

Therefore, we need to wait for updates from Oracle and Langchain.

[davidmonteroarin]

I have an update.

The conclusion seems to be we need to wait for langchain to be updated to work with chat models as text generation models will be deprecated. So we will have to wait as well but for another things....

[SherifBashar]

Perfect, thank you so much for the update

[davidmonteroarin]

Hi @SherifBashar , we are still waiting. Do you have more updates about langchain and chat model compatibility? Actually, we have confirmed chat models work much better than text generation models so when we can work with them from chains we will have better performance

Regards,
David

[davidmonteroarin]

Finally I could make it work. I don't know if langchain launched a fix but if you use ChatOCIGenAI instead of OCIGenAI it works with chat models and you can use it in chains.

For example, when using chat models use this:

from langchain_community.chat_models import ChatOCIGenAI
llm_generation = ChatOCIGenAI(
model_id=modelId,
provider=provider,
service_endpoint=endpoint,
compartment_id=compartment_id,
model_kwargs={"temperature": 0.7, "top_p": 0.75, "max_tokens": 4000}
)

instead of this:

from langchain_community.llms import OCIGenAI
llm_generation = OCIGenAI(
model_id=modelId,
provider=provider,
service_endpoint=endpoint,
compartment_id=compartment_id,
model_kwargs={"temperature": 0.7, "top_p": 0.75, "max_tokens": 4000}
)

[ramjiiyengar]

This works. Thanks!