Added benchmarking results to main README.md #176
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
olson-ibm
requested review from
alex-jw-brooks,
gkumbhat,
evaline-ju and
gabe-l-hart
as code owners
olson-ibm
force-pushed
the
139/fine-tuning-a100
branch
from
bebf62e to
48bc9f4
Compare
gkumbhat
reviewed
Sep 7, 2023
Comment on lines
+9
to
+10
| | [2023-09-05](./logs/llama2-7b/20230905_183655.output) | 1 x A100 80GB | [Glue / RTE](https://huggingface.co/datasets/glue) | 1 | bfloat16 | 6 | 4096 | 350 | 21.325 | 0.22 | 1.65 | 4096 is the context size for Llama2 | | ||
| | [2023-09-05](./logs/llama2-7b/20230905_184809.output) | 1 x A100 80GB | [Glue / RTE](https://huggingface.co/datasets/glue) | 1 | bfloat16 | 6 | 1024 | 350 | 21.333 | 0.22 | 1.65 | batch size of 7 fails CUDA OOM | |
There was a problem hiding this comment.
I think RTE might have text long enough for 4096 context length, but definitely good to note and then we can iterate later on with a different dataset..
gkumbhat
reviewed
Sep 7, 2023
README.md
Outdated
| @@ -41,6 +41,10 @@ Prompt tuning - learning soft prompts. This is different from prompt engineering | |||
|
|
|||
| The important difference between fine tuning and capabilities like prompt tuning/multi-taskprompt tuning is that the latter doesn't change the base model's weights at all. So when you run inference for prompt tuned models, you can have n prompts to 1 base model, and just inject the prompt tensors you need when they're requested instead of having _n_ separate fine-tuned models. | |||
|
|
|||
| ### Benchmarking | |||
|
|
|||
| [Benchmarks](./benchmarks/README.md) for tuning various models. | |||
There was a problem hiding this comment.
nit: Can we add a small description here too (similar to benchmarks readme) so that people are not looking for quality metrics when they look at this one.
There was a problem hiding this comment.
Rephrase as "Performance Benchmarking"?
There was a problem hiding this comment.
Yep. Rephrasing as Runtime Performance Benchmarking sounds good!
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
Signed-off-by: Joe Olson <[email protected]>
olson-ibm
force-pushed
the
139/fine-tuning-a100
branch
from
904b5e7 to
b3ba86a
Compare
gkumbhat
approved these changes
Sep 12, 2023
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a section to the main README.md for benchmarking. This links to a second README.md in ./benchmarking that contains a poorly formatted table summarizing the benchmarking results on Llama2-7b using 1 x A100 80GB. on the CCC.
Also included under ./logs is the output from the individual benchmarks "showing our work" including all the parameters used for training, in the event someone wants to reproduce this.
The poor formatting on the table seems to be with how GitHub is rendering the markdown. I spent 2 hours messing with the formatting / reading docs / wasting time trying to get the column widths to adjust, but failed in every attempt. Several artifacts of this attempt are still left in the table formatting to prove that nothing works to set the column width so the table looks clean. The fact that this is a systematic problem with Github rendering the markdown is also demonstrated with the table in the main README.md that was not changed for this PR.