multi-image support for llama3.2 #705
Conversation
yma11
requested review from
kzawora-intel,
madamczykhabana,
michalkuligowski,
mgawarkiewicz,
vivekgoe and
afierka-intel
as code owners
yma11
force-pushed
the
multi-image
branch
3 times, most recently
from
8477d17 to
08541ff
Compare
|
@kdamaszk @michalkuligowski @kzawora-intel, can you help review on this PR? Already tested on 11B and 90B. |
| sampling_params = SamplingParams(top_p=0.99, top_k=self.vocab_size - 1) | ||
| max_num_batched_tokens = self.scheduler_config.max_num_batched_tokens | ||
| # Workaround to avoid unexpeced OOM failure during profile run | ||
| max_num_seqs = int(self.scheduler_config.max_num_seqs/2) |
There was a problem hiding this comment.
I don't like this. Max prefill batch size is limited by self.max_num_prefill_seqs. I believe the role of profile_run is to generate with maximum possible batch size. If it is too big than user should limit it. Such hacks will not prevent from actual OOM in the middle of benchmark.
There was a problem hiding this comment.
Understand your point. This is a temporary work around. For multi-modal models, this configurations not only affect the generated prefill prompts but also the encode dummy data, which will occupy a lot of memory during process before actually computed together with prefill and cause OOM. Current encode data process has some flaws and will be our next step to see how to improve it. Also, I have tested different configurations on benchmarks and not observed actual OOM.
There was a problem hiding this comment.
@kdamaszk I revert back to use previous logic but add some fixes for image tokens. Can you help merge this PR if there is no other big issue? QA need this PR merged to launch test. I can address in my later PR if there are some minor comments.
yma11
force-pushed
the
multi-image
branch
3 times, most recently
from
c9bfe70 to
7223fb8
Compare
Signed-off-by: yan ma <[email protected]>
Signed-off-by: yan ma <[email protected]>
Signed-off-by: yan ma <[email protected]>
No description provided.