Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build the Mistral TRT-LLM int4 AWQ Engine not working #77

Open
kzpromo opened this issue Jun 24, 2024 · 1 comment
Open

Build the Mistral TRT-LLM int4 AWQ Engine not working #77

kzpromo opened this issue Jun 24, 2024 · 1 comment

Comments

@kzpromo
Copy link

kzpromo commented Jun 24, 2024

(chatrtx) C:\chatrtx>trtllm-build --checkpoint_dir .\model\mistral_model\model_checkpoints --output_dir .\model\mistral_model\engine --gpt_attention_plugin float16 --gemm_plugin float16 --max_batch_size 1 --max_input_len 7168 --max_output_len 1024 --context_fmha=enable --paged_kv_cache=disable --remove_input_padding=disable
Traceback (most recent call last):
File "C:\Users***\AppData\Local\anaconda3\envs\chatrtx\lib\site-packages\tensorrt_llm_init_.py", line 39, in
import tensorrt_llm.bindings # NOQA
ImportError: DLL load failed while importing bindings: Не найден указанный модуль.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users*\AppData\Local\anaconda3\envs\chatrtx\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users*
\AppData\Local\anaconda3\envs\chatrtx\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users***\AppData\Local\anaconda3\envs\chatrtx\Scripts\trtllm-build.exe_main
.py", line 4, in
File "C:\Users***\AppData\Local\anaconda3\envs\chatrtx\lib\site-packages\tensorrt_llm_init_.py", line 41, in
raise ImportError(
ImportError: Import of the bindings module failed. Please check the package integrity. If you are attempting to use the pip development mode (editable installation), please execute build_wheels.py first, and then run pip install -e .

@anujj
Copy link
Collaborator

anujj commented Jun 24, 2024

This usually means that a dynamic dependency of the bindings library is missing on PATH. Can u please try with these command

python -m pip uninstall -y tensorrt
pip uninstall -y tensorrt tensorrt_libs tensorrt_bindings
python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.3.0.post12.dev1 --no-cache-dir
Check with: python -c "import tensorrt_llm; print(tensorrt_llm._utils.trt_version())"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants