Error using custom imported Llama3.1 model on Bedrock

[mgaionWalit]

• Lanchain V 0.3.2
• Lanchain AWS v 0.2.2

We are using a fine-tuned version of Llama 3.1-instruct, uploaded to Bedrock. Since we are using an ARN model ID (which does not contain any information about the specific Foundation Model used), we encountered an issue.

In the code chat_models/bedrock.py at line 349, there is an if statement evaluating the model string to choose between Llama2 and Llama3 for prompt conversion.

In our case, we need to use convert_messages_to_prompt_llama3, but the logic falls into the else statement, which uses convert_messages_to_prompt_llama.

Is there any solution to ensure the correct conversion function is used?

Thank you!

[3coins]

@mgaionWalit
The best solution here would be to and a new attribute to supply the base model used to fine tune the custom model.

[mgaionWalit]

Thanks for the reply.

There is a temporary workaround?

[3coins]

@mgaionWalit
Don't think there is a way to handle this without a code change. I don't have any custom model setup to verify, but does the Bedrock API provide any info regarding the model that can help us infer the base provider/model?

[3coins]

@mgaionWalit
The converse API now supports imported models for Meta and Mistral models, can you verify if this works as expected with the ChatBedrockConverse, by passing in a provider value.

[mgaionWalit]

Hi @3coins

thanks for you support.

I've updated langchain-aws to the latest available version (0.2.4) and tried to init the custom model with ChatBedrockConverse :

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime"
)
model = ChatBedrockConverse(
    client=bedrock_runtime,
    model_id='arn:aws:bedrock:us-east-1:XXXXXX:imported-model/XXXXXX', # here I set the arn of the imported model
    provider='meta',
    temperature=0.15,
    max_tokens=100
)

but I still get an error when trying to start a chat with stream that says:
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the ConverseStream operation: This action doesn't support the model that you provided. Try again with a supported text or chat model

Am I doing something wrong? Do you have some suggestions on how to make it work?