Add wandb
logging to BootstrapFewShot
's metric_val
#544
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Weights & Biases (wandb) are one of the industry leaders in experiment tracking software for Machine Learning.
I think
runs
andsweeps
both offer significant value to monitoring DSPy optimization runs.This PR will add wandb logging to
BootstrapFewShot
to log the values ofmetric_val
as the teacher model is bootstrapping traces through the program.Intended use:
I think this will be especially effective when say looping through program configurations (in terms of multi-model design or the composition of modules) or teacherLMs.
As a result, DSPy users will have a nice visualization of how effectively the teacher LM is able to bootstrap traces through the program that surpass the
metric_threshold
.I think this is a nice building block, but generally the wandb integration will be more effective with higher-level optimizers such as
BootstrapFewShotRandomSearch
,BootstrapFewShotWithOptuna
, or theBayesianSignatureOptimizer
.