When I try to connect my LLM that is deployed on Azure AI studio using AzureMLOnlineEndpoint I get an 404 error.

[mmentese21]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import os
import dotenv
dotenv.load_dotenv()
from langchain_community.llms.azureml_endpoint import (
    AzureMLEndpointApiType,
    CustomOpenAIContentFormatter, AzureMLOnlineEndpoint,
)
from langchain_core.messages import HumanMessage
from langchain_core.prompts import ChatPromptTemplate

llm = AzureMLOnlineEndpoint(
    endpoint_url="https://Meta-Llama-3-1-8B-Instruct-*****.swedencentral.models.ai.azure.com/score",
    endpoint_api_type=AzureMLEndpointApiType.dedicated,
    endpoint_api_key=os.getenv("LLAMA3_API_KEY"),
    content_formatter=CustomOpenAIContentFormatter(),
    model_kwargs={"temperature": 0.8, "max_new_tokens": 400},
)
response = llm.invoke("Write me a song about sparkling water:")
print(response)

Description

I have changed the endpoint url to fit the expected format. When I don't change it it gives an error that says it should be in the provided format. What should i do

System Info

python 3

[dosubot[bot]]

@mmentese21 Hello there! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

To resolve the 404 error when trying to connect your LLM deployed on Azure AI Studio using AzureMLOnlineEndpoint, ensure that you have correctly set the endpoint_url, endpoint_api_type, and endpoint_api_key. Here is an example of how to set these parameters correctly:

from langchain_community.llms.azureml_endpoint import (
    AzureMLEndpointApiType,
    CustomOpenAIContentFormatter,
    AzureMLOnlineEndpoint,
)

llm = AzureMLOnlineEndpoint(
    endpoint_url="https://<your-endpoint>.<your_region>.inference.ml.azure.com/score",  # Ensure this URL is correct
    endpoint_api_type=AzureMLEndpointApiType.dedicated,  # Ensure this matches your deployment type
    endpoint_api_key="my-api-key",  # Ensure this API key is correct
    content_formatter=CustomOpenAIContentFormatter(),
    model_kwargs={"temperature": 0.8, "max_new_tokens": 400},
)
response = llm.invoke("Write me a song about sparkling water:")
print(response)

Make sure that:

1. The endpoint_url is correctly formatted and points to the right endpoint.
2. The endpoint_api_type matches the type of your deployment (dedicated or serverless).
3. The endpoint_api_key is valid and has the necessary permissions.

If these parameters are correctly set and you still encounter a 404 error, verify that the endpoint is active and accessible. Additionally, ensure that the deployment_name is correctly specified if required by your endpoint configuration ^[1][2].

To continue talking to Dosu, mention @dosu.