关于运行run.py 执行卡死的情况

[GOOD-N-LCM]

我在运行run.py 测试本地HF模型 ，日志卡在这里很久一动不动。 请问这是什么原因造成的？

from opencompass.models import HuggingFaceCausalLM

models = [
dict(
type=HuggingFaceCausalLM,
abbr='qwen2-7b_test002',
path="/data/LLM_model/models/Qwen2-7B-Instruct/", # 模型路径
max_out_len=100,
max_seq_len=2048,
batch_size=16,
run_cfg=dict(num_gpus=2),
)
]

[liushz]

你用huggingface能直接load这个模型吗？

[liushz]

先确定你能跑通下面的Qwen2官方demo，表示你的本地模型没有问题：

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model_name_or_path = "data/LLM_model/models/Qwen2-7B-Instruct/"

model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

[GOOD-N-LCM]

谢谢您的回复。是的，这套代码我已经使用过的。确保他可以完成我的需求才开是opencompass的评测工程的。

[liushz]

你的output文件夹有prediction输出吗？

[liushz]

你的output文件夹有prediction输出吗？

尝试加上--debug

[GOOD-N-LCM]

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00, 1.82it/s]07/08 19:17:08 - OpenCompass - INFO - Start inferencing [_hf/ceval-computer_network]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 498073.60it/s][2024-07-08 19:17:08,533] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
0%| | 0/5 [00:00<?, ?it/s]/home/aaa/anaconda3/envs/opencompass/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:540: UserWarning: do_sample is set to False. However, temperature is set to 0.7 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature.
warnings.warn(
/home/aaa/anaconda3/envs/opencompass/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:545: UserWarning: do_sample is set to False. However, top_p is set to 0.8 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p.
warnings.warn(
/home/aaa/anaconda3/envs/opencompass/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:562: UserWarning: do_sample is set to False. However, top_k is set to 20 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_k.
warnings.warn(

是的，我运行的时候已经加上了。 但是他会卡在这里 ，这个waring不会影响工程的进行吧 是因为我算力不足但是了他很慢?

[liushz]

看起来推理是正常的，只是推理比较慢，任务很多但看起来你只有一张卡，另外--debug会一条一条推理，正常推理时候请关闭--debug

[GOOD-N-LCM]

所以目前从日志推测是我算力不足导致的推理慢。是的我只能用一张卡。
感谢您的解答！

[qizheyanger]

您好，我也出现了您的问题，推理的速度非常慢。但是后面我通过nvidia-smi观察发现，我的代码完全是在cpu上运行，并没有在gpu上运行，不知道您是否解决了该问题。