fix: refactor adapter weight loading and mapping

[drbh]

This PR refactors the loading and mapping of lora adapters and avoids the need for model specific changes for adapters. The goal of this PR is to simplify the loading flow and avoid modify modeling code for loras to work.

[drbh]

this PR has been updated to include the ability to load lora adapters from a local directory.

the path to the lora adapter can be specified in the following way:

LORA_ADAPTERS=adapter_id=/dir/path

note it's possible to mix adapter_ids with adapter_id=adapter_path e.g.

LORA_ADAPTERS=predibase/dbpedia,myadapter=/path/to/dir/

[OlivierDehaene]

Isn't this super brittle? It only works for llama, no?

[drbh]

its a bit brittle but should work in most cases, currently this will correctly load in weight for llama, mistral and gemma type models (just added some updates and tests too). The intention of moving this code here is to avoid mapping the weights inside of each models code.. currently this is the best approach I have, however I'm happy to make any changes!

[danieldk]

I guess we could have a mapping table here per model type? But probably better for another PR.

[danieldk]

Looks like a really nice improvement 🎉. Had two questions.

[tensimixt]

This PR refactors the loading and mapping of lora adapters and avoids the need for model specific changes for adapters. The goal of this PR is to simplify the loading flow and avoid modify modeling code for loras to work.

Does this also apply to lora adapters generated by using the mistral-finetune repo also? As the weights in these adapters use keys that are in the consolidated.safetensors file typically found in recent mistral huggingface repos, and these differ from conventional key names. The best example is Mistral 7b instruct v0.3.

For example have tried to use a lora adapter generated from mistral-finetune but it did not work with TGI. It also did not work with vllm. Also to get it in a state where it might work with vllm had mapped the weights keys to what is conventionally used but found lm_head, embed_tokens and layernorm weight keys that vllm could not handle. Will this PR address these issues also?

Thank you!

[drbh]

Hi @tensimixt, thank you for your question. If mistral-finetune loads LoRA with different adapter names than expected, this may not be resolved in this PR. The goal of this PR is to simplify the LoRA logic. Improvements, including support for mistral-finetune, will be explored in future PRs.

[danieldk]

Added a comment about _get_model/get_model, maybe the naming can be improved?

Looks good to me otherwise.

[danieldk]

I guess we could have a mapping table here per model type? But probably better for another PR.

[mhou7712]

Any update about this PR? We will have this PR included in the next release build? Thanks.

[ErikKaum]

Hi @mhou7712 👋

So this PR (#2193) has been merged and should be in the next release 👍

[mhou7712]

Hi @ErikKaum , cool and thanks a lots!!