[ascend]feat: support kv int8 #103
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yao-fengchen
force-pushed
the
ascend_kv_int8
branch
8 times, most recently
from
0908362
to
1c73e98
Compare
yao-fengchen
force-pushed
the
ascend_kv_int8
branch
from
e266508
to
3608fd7
Compare
yao-fengchen
force-pushed
the
ascend_kv_int8
branch
from
3608fd7
to
83b567c
Compare
yao-fengchen
force-pushed
the
ascend_kv_int8
branch
from
83b567c
to
b8fc1b3
Compare
CyCle1024
reviewed
Nov 13, 2024
There was a problem hiding this comment.
建议做成python -m dlinfer.framework.lmdeploy_ext.quants.ascend_kv --model_path xxx etc. 加上参数可以直接执行。。。
| "dlinfer::fill_kv_cache", | ||
| ["key_cache", "value_cache"], | ||
| default_value={ | ||
| "quant_bits": 0, |
There was a problem hiding this comment.
是否可以添加默认值：
"k_scales_zeros": tuple()
"v_scales_zeros": tuple()
| @@ -205,6 +220,9 @@ def paged_decode_attention( | |||
| softmax_scale: Optional[float], | |||
| alibi_slopes: Optional[Sequence[float]], | |||
| attn_output: Optional[Tensor], | |||
| kv_scales: Optional[Tensor], | |||
| kv_zeros: Optional[Tensor], | |||
There was a problem hiding this comment.
这里 kv_zeros 的类型为啥和 fill_kv_cache 中的 k_scales_zeros，v_scales_zeros 类型不一致？
There was a problem hiding this comment.
fill_kv_cache中是为了避免slice
dlinfer/ops/llm.py
Outdated
| k_scales_zeros: Sequence[Optional[Tensor]], | ||
| v_scales_zeros: Sequence[Optional[Tensor]], | ||
| quant_bits: int, | ||
| ) -> Tuple[Tensor, Tensor, Tensor, Tensor]: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.