Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wandb logging to BootstrapFewShot's metric_val #544

Closed
wants to merge 3 commits into from

Conversation

CShorten
Copy link
Collaborator

@CShorten CShorten commented Mar 4, 2024

Weights & Biases (wandb) are one of the industry leaders in experiment tracking software for Machine Learning.

I think runs and sweeps both offer significant value to monitoring DSPy optimization runs.

This PR will add wandb logging to BootstrapFewShot to log the values of metric_val as the teacher model is bootstrapping traces through the program.

Intended use:

# connect to wandb project
wandb.init(
  project: "my-dspy-program",
  config = {
    "program": "Perspective-Guided-STORM",
    "teacherLM": "GPT4"
  }
)

# wandb logging is internal to the `teleprompter.compile` call
# ^ you just add values to the project with wandb.log(...)

compiled_progarm = teleprompter.compile(PG_STORM, trainset=trainset, ...)

I think this will be especially effective when say looping through program configurations (in terms of multi-model design or the composition of modules) or teacherLMs.

As a result, DSPy users will have a nice visualization of how effectively the teacher LM is able to bootstrap traces through the program that surpass the metric_threshold.

I think this is a nice building block, but generally the wandb integration will be more effective with higher-level optimizers such as BootstrapFewShotRandomSearch, BootstrapFewShotWithOptuna, or the BayesianSignatureOptimizer.

@CShorten CShorten changed the title Add wandb logging to BootstrapFewShot's metric_val Add wandb logging to BootstrapFewShot's metric_val Mar 4, 2024
@isaacbmiller
Copy link
Collaborator

Maybe better to implement this in config so we don't have to pass it as an individual parameter to each class and child?

@CShorten
Copy link
Collaborator Author

Just in case anyone had seen this --> moved here: #849

@CShorten CShorten closed this Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants