More Inference Endpoints features and fixes #68
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will raise an error, signaling there was a problem. Before the root thread was getting stuck waiting for the agent that was dead. This way it should exit.
It is possible to disable it by setting JSON_OUTPUT_DISABLE. It is now possible also to play with more batch sizes.
This will further simplify the implementation of prefill bucketing.
tengomucho
force-pushed
the
debug-tgi-ie-pt2
branch
from
8738cd2
to
890d402
Compare
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Truncation was sub-optimal, and it was done on the wrong side.
This will prevent XLA compilation at inference time. Note that I had to disable dynamo compilation though, otherwise the model was not generating correct results. This leads to slower generation, but at least generation seems stable now.
This will allow for testing IE before release.
tengomucho
force-pushed
the
debug-tgi-ie-pt2
branch
from
890d402
to
3145343
Compare
mfuntowicz
approved these changes
Jul 5, 2024
| @@ -3,6 +3,9 @@ name: Release | |||
| on: | |||
| release: | |||
| types: [published] | |||
| push: | |||
| branches: | |||
| - debug-tgi-ie-pt2 | |||
There was a problem hiding this comment.
Nope, I can remove it
| for _ in reversed(range(batch_size)): | ||
| # Prefill with different truncate sizes to test all prefill lengths. List is reversed so first longest | ||
| # sequences are tested and, if there is a memory failure, that will appear sooner. | ||
| for l in reversed(PREFILL_LENGTHS): |
There was a problem hiding this comment.
We will certainly need to cap the PREFILL_LENGTHS w.r.t the model's config max_position_embedding otherwise it might generate too much and also unsupported lengths
There was a problem hiding this comment.
good point. For now it will stop at the sequence length passed as parameter, but we can also make this check. I will do it on a future PR.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Several changes to improve Inference Endpoints and TGI overall.