Skip to content

Issues: aws-samples/foundation-model-benchmarking-tool

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

33 Open 29 Closed
33 Open 29 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add benchmarking for training models
#270 opened Jan 17, 2025 by madhurprash
@madhurprash
The pricing fetching logic only works for EC2
#268 opened Jan 14, 2025 by Yiwen-Zhang
Experiment config file's model name doesn't match with pricing_fallback.yml file's model name
#267 opened Jan 14, 2025 by Yiwen-Zhang
Integrate Ollama deployment into FMBench
#262 opened Jan 9, 2025 by dheerajoruganty
parameterize configuration files
#261 opened Jan 8, 2025 by madhurprash
@madhurprash
Fails when testing Haiku on AWS Bedrock
#247 opened Nov 28, 2024 by NidPlays
Allow backup/ failover regions to be specified
#232 opened Nov 7, 2024 by jimburtoft
Add support for Bedrock Custom Import Models
#224 opened Oct 26, 2024 by dickren123
@madhurprash
1
miss warm up phase
#213 opened Oct 8, 2024 by lxning
Add support for non AWS models to be benchmarked
#203 opened Sep 28, 2024 by madhurprash
1
@madhurprash
add hallucination metric to model evals on FMBench
#170 opened Aug 14, 2024 by madhurprash
Add retry handling for throttling errors during inference
#157 opened Aug 1, 2024 by madhurprash
UI Support for FMBench
#155 opened Jul 31, 2024 by madhurprash
@madhurprash
TypeError: unsupported format string passed to NoneType.__format__
#154 opened Jul 24, 2024 by prasad-nair-amd
Unable to locate credential issue
#153 opened Jul 23, 2024 by prasadwrites
1
Remove the config_filepath.txt as a config reference for developer workflow
#150 opened Jul 18, 2024 by madhurprash
@madhurprash
Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker.
#120 opened Jun 14, 2024 by prasad-nair-amd
@aarora79
5
Reports only show information per instance instead of per endpoint
#114 opened Jun 7, 2024 by jimburtoft
Loosen dependencies for more flexible installation
#105 opened May 31, 2024 by athewsey
@aarora79
1
FMBench to support benchmarking for embedding models enhancement New feature or request
#33 opened Mar 25, 2024 by madhurprash
@madhurprash
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
#32 opened Mar 16, 2024 by akramIOT
Compare different models for the same dataset enhancement New feature or request
#30 opened Feb 17, 2024 by aarora79
@aarora79
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute enhancement New feature or request
#29 opened Feb 17, 2024 by aarora79
Provide config file for FLAN-T5 out of the box enhancement New feature or request
#28 opened Feb 15, 2024 by aarora79
@madhurprash
Allow different experiments in the same config file to use different regions good first issue Good for newcomers
#27 opened Feb 15, 2024 by aarora79
@4mp
ProTip! Mix and match filters to narrow down what you’re looking for.