[Feature] TurbomindEngineConfig supports lora

[seanxuu]

Motivation

The existing HF is supporting inference without merging weights.

This is very important for a large number of tests! Merging weights consumes a lot of resources and time.

Related resources

No response

Additional context

No response

[lvhan028]

Turbomind has no plan to support lora recently.
But the other engine of LMDeploy TorchEngine has supported it. May give it a try.
@grimoire please provide some guide

[grimoire]

Hi, you can try out PyTorch Engine.

You can build engine with a dictionary with adapter_name:adapter_path pair.

from lmdeploy.messages import PytorchEngineConfig
from lmdeploy.pytorch.engine.engine import Engine
adapters = {'default':'/path/to/adapter'}
engine_config = PytorchEngineConfig(adapters=adapters)
engine = Engine.from_pretrained(model_path,
                                engine_config=engine_config,
                                trust_remote_code=True)

And perform inference with engine instance:

generator = engine.create_instance()

for outputs in generator.stream_infer(session_id=session_id,
                                      input_ids=input_ids,
                                      gen_config=gen_config,
                                      adapter_name=adapter_name):
    # read outputs

# close session and release caches
generator.end(session_id)

[Cloopen-ReLiNK]

Hi, you can try out PyTorch Engine.

You can build engine with a dictionary with adapter_name:adapter_path pair.

from lmdeploy.messages import PytorchEngineConfig
from lmdeploy.pytorch.engine.engine import Engine
adapters = {'default':'/path/to/adapter'}
engine_config = PytorchEngineConfig(adapters=adapters)
engine = Engine.from_pretrained(model_path,
                                engine_config=engine_config,
                                trust_remote_code=True)

And perform inference with engine instance:

generator = engine.create_instance()

for outputs in generator.stream_infer(session_id=session_id,
                                      input_ids=input_ids,
                                      gen_config=gen_config,
                                      adapter_name=adapter_name):
    # read outputs

# close session and release caches
generator.end(session_id)

Hi, you can try out PyTorch Engine.

You can build engine with a dictionary with adapter_name:adapter_path pair.

from lmdeploy.messages import PytorchEngineConfig
from lmdeploy.pytorch.engine.engine import Engine
adapters = {'default':'/path/to/adapter'}
engine_config = PytorchEngineConfig(adapters=adapters)
engine = Engine.from_pretrained(model_path,
                                engine_config=engine_config,
                                trust_remote_code=True)

And perform inference with engine instance:

generator = engine.create_instance()

for outputs in generator.stream_infer(session_id=session_id,
                                      input_ids=input_ids,
                                      gen_config=gen_config,
                                      adapter_name=adapter_name):
    # read outputs

# close session and release caches
generator.end(session_id)

python: /project/lib/Analysis/Allocation.cpp:40: std::pair<llvm::SmallVector, llvm::SmallVector > mlir::triton::getCvtOrder(mlir::Attribute, mlir::Attribute): Assertion `!(srcMmaLayout && dstMmaLayout) && "Unexpected mma -> mma layout conversion"' failed.
Aborted，

chatglm2-6b加载lora预测报错 V100 cuda12 pytorch:2.1.1 triton:2.1.0，按指导下载的whl文件安装的lmdeploy
使用cuda11机器也是同样的报错

[grimoire]

@Cloopen-ReLiNK I can not reproduce with this adapter. Would you mind provide the adapter?

[Cloopen-ReLiNK]

you mind provide the adapter

Thank you, I will test load the adapter you are using first. If there is any problem, I will provide the adapter.

produced the same error。

[Cloopen-ReLiNK]

torch:2.1.1 triton:2.1.0 lmdeploy:0.2.1 cuda:11.7 V100-32G

@grimoire
What is the machine environment you use?

from lmdeploy.messages import PytorchEngineConfig
from lmdeploy.pytorch.engine.engine import Engine
adapters = {'default':'./lora-chatglm2-6b-guodegang'}
engine_config = PytorchEngineConfig(adapters=adapters)
model_path = '/home/pretrained_model/chatglm2-6b'

engine = Engine.from_pretrained(model_path,
engine_config=engine_config,
trust_remote_code=True)

generator = engine.create_instance()

session_id = '1'
input_ids = engine.tokenizer.encode("你好")
adapter_name = 'default'
from lmdeploy import GenerationConfig
gen_config = GenerationConfig(top_p=0.8,
top_k=40,
temperature=0.8,
max_new_tokens=1024
)

for outputs in generator.stream_infer(session_id=session_id,
input_ids=input_ids,
gen_config=gen_config,
adapter_name=adapter_name):
pass

[grimoire]

I can replicate the error on V100 now. It seems that the flash-attention implementation does not fit devices with sm<80.

[Cloopen-ReLiNK]

3090 with cuda11.6, both adapters can run normally.


But the generated results are inconsistent, the question is "你好"

[Cloopen-ReLiNK]

Is there any solution for using V100? flash attention v1？

[grimoire]

But the generated results are inconsistent

This might caused by random sampling.

Is there any solution for using V100?

I am trying to find a fix, please allow me some time.

[grimoire]

@Cloopen-ReLiNK #1027 Please have a try.

[Cloopen-ReLiNK]

@Cloopen-ReLiNK #1027 Please have a try.

solved

[seanxuu]

Hi, you can try out PyTorch Engine.

You can build engine with a dictionary with adapter_name:adapter_path pair.

from lmdeploy.messages import PytorchEngineConfig
from lmdeploy.pytorch.engine.engine import Engine
adapters = {'default':'/path/to/adapter'}
engine_config = PytorchEngineConfig(adapters=adapters)
engine = Engine.from_pretrained(model_path,
                                engine_config=engine_config,
                                trust_remote_code=True)

And perform inference with engine instance:

generator = engine.create_instance()

for outputs in generator.stream_infer(session_id=session_id,
                                      input_ids=input_ids,
                                      gen_config=gen_config,
                                      adapter_name=adapter_name):
    # read outputs

# close session and release caches
generator.end(session_id)

Can it achieve the same abilities as the TurbomindEngine? Because I find TurbomindEngine is better than PytorchEngine.

[grimoire]

Can it achieve the same abilities as the TurbomindEngine?

No, PytorchEngine is slower.

[seanxuu]

Can it achieve the same abilities as the TurbomindEngine?

No, PytorchEngine is slower.

Got it. Thank you.