diff --git a/README.md b/README.md index c10ad1cc..78c06065 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,12 @@ -# OptunaHub Registry +OptunaHub Registry +================== + +![OptunaHub](https://github.com/user-attachments/assets/ee24b6eb-a431-4e02-ae52-c2538ffe01ee) + +:link: [**OptunaHub**](https://hub.optuna.org/) +| :page_with_curl: [**Docs**](https://optuna.github.io/optunahub/) +| :page_with_curl: [**Tutorials**](https://optuna.github.io/optunahub-registry/) +| [**Optuna.org**](https://optuna.org/) OptunaHub Registry is a registry service for sharing and discovering user-defined Optuna packages. It provides a platform for users to share their Optuna packages with others and discover useful packages created by other users. @@ -6,6 +14,18 @@ See the [OptunaHub Website](https://hub.optuna.org/) for registered packages. See also the [OptunaHub API documentation](https://optuna.github.io/optunahub/) for the API to use the registry, and the [OptunaHub tutorial](https://optuna.github.io/optunahub-registry/) for how to register and discover packages. +## Contribution + +Any contributions to OptunaHub are more than welcome! + +OptunaHub is composed of the following three related repositories. Please contribute to the appropriate repository for your purposes. +- [optunahub](https://github.com/optuna/optunahub) + - The python library to use OptunaHub. If you find issues and/or bugs in the optunahub library, please report it here via [Github issues](https://github.com/optuna/optunahub/issues). +- [optunahub-registry](https://github.com/optuna/optunahub-registry/) (*this repository*) + - The registry of the OptunaHub packages. If you are interested in registering your package with OptunaHub, please contribute to this repository. For general guidelines on how to contribute to the repository, take a look at [CONTRIBUTING.md](https://github.com/optuna/optunahub-registry/blob/main/CONTRIBUTING.md). +- [optunahub-web](https://github.com/optuna/optunahub-web/) + - The web frontend for OptunaHub. If you find issues and/or bugs on the website, please report it here via [GitHub issues](https://github.com/optuna/optunahub-web/issues). + ## Quick TODO List towards Contribution When creating your package, please check the following TODO list: diff --git a/docs/source/index.rst b/docs/source/index.rst index c92589e7..7ddbeece 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -14,6 +14,8 @@ Recipes - :doc:`recipes/003_pruner` - :doc:`recipes/004_visualization` - :doc:`recipes/005_debugging` +- :doc:`recipes/006_benchmarks_basic` +- :doc:`recipes/007_benchmarks_advanced` License diff --git a/package/samplers/cmamae/README.md b/package/samplers/cmamae/README.md index 7801df9f..b14a5d17 100644 --- a/package/samplers/cmamae/README.md +++ b/package/samplers/cmamae/README.md @@ -27,6 +27,10 @@ with improvement ranking, all wrapped up in a However, it is possible to implement many variations of CMA-MAE and other quality diversity algorithms using pyribs. +For visualizing the results of the `CmaMaeSampler`, note that we use the +`plot_grid_archive_heatmap` function from the +[`plot_pyribs`](https://hub.optuna.org/visualization/plot_pyribs/) plugin. + ## Class or Function Names - CmaMaeSampler @@ -46,12 +50,17 @@ $ pip install ribs ## Example ```python +import matplotlib.pyplot as plt import optuna import optunahub + module = optunahub.load_module("samplers/cmamae") CmaMaeSampler = module.CmaMaeSampler +plot_pyribs = optunahub.load_module(package="visualization/plot_pyribs") +plot_grid_archive_heatmap = plot_pyribs.plot_grid_archive_heatmap + def objective(trial: optuna.trial.Trial) -> float: """Returns an objective followed by two measures.""" @@ -80,6 +89,11 @@ if __name__ == "__main__": ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10000) + + fig, ax = plt.subplots(figsize=(8, 6)) + plot_grid_archive_heatmap(study, ax=ax) + plt.savefig("archive.png") + plt.show() ``` ## Others diff --git a/package/samplers/cmamae/example.py b/package/samplers/cmamae/example.py index 2903a81a..97b88743 100644 --- a/package/samplers/cmamae/example.py +++ b/package/samplers/cmamae/example.py @@ -1,3 +1,4 @@ +import matplotlib.pyplot as plt import optuna import optunahub @@ -5,6 +6,9 @@ module = optunahub.load_module("samplers/cmamae") CmaMaeSampler = module.CmaMaeSampler +plot_pyribs = optunahub.load_module(package="visualization/plot_pyribs") +plot_grid_archive_heatmap = plot_pyribs.plot_grid_archive_heatmap + def objective(trial: optuna.trial.Trial) -> float: """Returns an objective followed by two measures.""" @@ -33,3 +37,8 @@ def objective(trial: optuna.trial.Trial) -> float: ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=10000) + + fig, ax = plt.subplots(figsize=(8, 6)) + plot_grid_archive_heatmap(study, ax=ax) + plt.savefig("archive.png") + plt.show() diff --git a/package/samplers/hebo/README.md b/package/samplers/hebo/README.md index c60ea858..11f1d1a2 100644 --- a/package/samplers/hebo/README.md +++ b/package/samplers/hebo/README.md @@ -14,10 +14,12 @@ license: MIT License ## Installation ```bash -pip install -r https://hub.optuna.org/samplers/hebo/requirements.txt -git clone git@github.com:huawei-noah/HEBO.git -cd HEBO/HEBO -pip install -e . +# Install the dependencies. +pip install optunahub hebo + +# NOTE: Below is optional, but pymoo must be installed after NumPy for faster HEBOSampler, +# we run the following command to make sure that the compiled version is installed. +pip install --upgrade pymoo ``` ## APIs @@ -59,11 +61,7 @@ def objective(trial: optuna.trial.Trial) -> float: module = optunahub.load_module("samplers/hebo") -sampler = module.HEBOSampler(search_space={ - "x": optuna.distributions.FloatDistribution(-10, 10), - "y": optuna.distributions.IntDistribution(-10, 10), -}) -# sampler = module.HEBOSampler() # Note: `search_space` is not required, and thus it works too. +sampler = module.HEBOSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=100) @@ -73,6 +71,19 @@ print(study.best_trial.params, study.best_trial.value) See [`example.py`](https://github.com/optuna/optunahub-registry/blob/main/package/samplers/hebo/example.py) for a full example. ![History Plot](images/hebo_optimization_history.png "History Plot") +Note that it may slightly speed up the sampling routine by giving the search space directly to `HEBOSampler` since Optuna can skip the search space inference. +For example, the instantiation of `HEBOSampler` above can be modified as follows: + +```python +search_space={ + "x": optuna.distributions.FloatDistribution(-10, 10), + "y": optuna.distributions.IntDistribution(-10, 10), +} +sampler = module.HEBOSampler(search_space=search_space) +``` + +However, users need to make sure that the provided search space and the search space defined in the objective function must be consistent. + ## Others HEBO is the winning submission to the [NeurIPS 2020 Black-Box Optimisation Challenge](https://bbochallenge.com/leaderboard). diff --git a/package/samplers/hebo/requirements.txt b/package/samplers/hebo/requirements.txt index 2ba8ec1a..2edc8955 100644 --- a/package/samplers/hebo/requirements.txt +++ b/package/samplers/hebo/requirements.txt @@ -1,3 +1,3 @@ optuna optunahub -hebo@git+https://github.com/huawei-noah/HEBO.git@v0.3.6#subdirectory=HEBO +hebo diff --git a/package/samplers/hebo/sampler.py b/package/samplers/hebo/sampler.py index f887d64f..45bb9369 100644 --- a/package/samplers/hebo/sampler.py +++ b/package/samplers/hebo/sampler.py @@ -2,7 +2,6 @@ from collections.abc import Sequence from typing import Any -import warnings import numpy as np import optuna @@ -10,6 +9,7 @@ from optuna.distributions import CategoricalDistribution from optuna.distributions import FloatDistribution from optuna.distributions import IntDistribution +from optuna.logging import get_logger from optuna.samplers import BaseSampler from optuna.search_space import IntersectionSearchSpace from optuna.study import Study @@ -23,6 +23,9 @@ from hebo.optimizers.hebo import HEBO +_logger = get_logger(f"optuna.{__name__}") + + class HEBOSampler(optunahub.samplers.SimpleBaseSampler): """A sampler using `HEBO __` as the backend. @@ -85,7 +88,6 @@ def __init__( self._hebo = None self._intersection_search_space = IntersectionSearchSpace() self._independent_sampler = independent_sampler or optuna.samplers.RandomSampler(seed=seed) - self._is_independent_sample_necessary = False self._constant_liar = constant_liar self._rng = np.random.default_rng(seed) @@ -113,12 +115,12 @@ def _suggest_and_transform_to_dict( def _sample_relative_define_and_run( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] - ) -> dict[str, float]: + ) -> dict[str, Any]: return self._suggest_and_transform_to_dict(self._hebo, search_space) def _sample_relative_stateless( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] - ) -> dict[str, float]: + ) -> dict[str, Any]: if self._constant_liar: target_states = [TrialState.COMPLETE, TrialState.RUNNING] else: @@ -131,10 +133,8 @@ def _sample_relative_stateless( # note: The backend HEBO implementation uses Sobol sampling here. # This sampler does not call `hebo.suggest()` here because # Optuna needs to know search space by running the first trial in Define-by-Run. - self._is_independent_sample_necessary = True return {} - else: - self._is_independent_sample_necessary = False + trials = [t for t in trials if set(search_space.keys()) <= set(t.params.keys())] # Assume that the back-end HEBO implementation aims to minimize. @@ -149,12 +149,12 @@ def _sample_relative_stateless( params = pd.DataFrame([t.params for t in trials]) values[np.isnan(values)] = worst_value values *= sign - hebo.observe(params, values) + hebo.observe(params, values[:, np.newaxis]) return self._suggest_and_transform_to_dict(hebo, search_space) def sample_relative( self, study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution] - ) -> dict[str, float]: + ) -> dict[str, Any]: if study._is_multi_objective(): raise ValueError( f"{self.__class__.__name__} has not supported multi-objective optimization." @@ -226,10 +226,10 @@ def sample_independent( param_name: str, param_distribution: BaseDistribution, ) -> Any: - if not self._is_independent_sample_necessary: - warnings.warn( - "`HEBOSampler` falls back to `RandomSampler` due to dynamic search space." - ) + states = (TrialState.COMPLETE,) + trials = study._get_trials(deepcopy=False, states=states, use_cache=True) + if any(param_name in trial.params for trial in trials): + _logger.warn(f"Use `RandomSampler` for {param_name} due to dynamic search space.") return self._independent_sampler.sample_independent( study, trial, param_name, param_distribution diff --git a/package/samplers/smac_sampler/README.md b/package/samplers/smac_sampler/README.md index 460b9caa..50882ec9 100644 --- a/package/samplers/smac_sampler/README.md +++ b/package/samplers/smac_sampler/README.md @@ -7,9 +7,30 @@ optuna_versions: [3.6.1] license: MIT License --- -## Class or Function Names +## APIs -- SAMCSampler +A sampler that uses SMAC3 v2.2.0. + +Please check the API reference for more details: + +- https://automl.github.io/SMAC3/main/5_api.html + +### `SMACSampler(search_space: dict[str, BaseDistribution], n_trials: int = 100, seed: int | None = None, *, surrogate_model_type: str = "rf", acq_func_type: str = "ei_log", init_design_type: str = "sobol", surrogate_model_rf_num_trees: int = 10, surrogate_model_rf_ratio_features: float = 1.0, surrogate_model_rf_min_samples_split: int = 2, surrogate_model_rf_min_samples_leaf: int = 1, init_design_n_configs: int | None = None, init_design_n_configs_per_hyperparameter: int = 10, init_design_max_ratio: float = 0.25, output_directory: str = "smac3_output")` + +- `search_space`: A dictionary of Optuna distributions. +- `n_trials`: Number of trials to be evaluated in a study. This argument is used to determine the number of initial configurations by SMAC3. Use at most `n_trials * init_design_max_ratio` number of configurations in the initial design. This argument does not have to be precise, but it is better to be exact for better performance. +- `seed`: Seed for random number generator. If `None` is given, seed is generated randomly. +- `surrogate_model_type`: What model to use for the probabilistic model. Either `"gp"` (Gaussian process), `"gp_mcmc"` (Gaussian process with MCMC), or `"rf"` (random forest). Default to `"rf"` (random forest). +- `acq_func_type`: What acquisition function to use. Either `"ei"` (expected improvement), `"ei_log"` (expected improvement with log-scaled function), `"pi"` (probability of improvement), or `"lcb"` (lower confidence bound). Default to `"ei_log"`. +- `init_design_type`: What initialization sampler to use. Either `"sobol"` (Sobol sequence), `"lhd"` (Latin hypercube), or `"random"`. Default to `"sobol"`. +- `surrogate_model_rf_num_trees`: The number of trees used for random forest. Equivalent to `n_estimators` in `RandomForestRegressor` in sklearn. +- `surrogate_model_rf_ratio_features`: The ratio of features to use for each tree training in random forest. Equivalent to `max_features` in `RandomForestRegressor` in sklearn. +- `surrogate_model_rf_min_samples_split`: The minimum number of samples required to split an internal node: Equivalent to `min_samples_split` in `RandomForestRegressor` in sklearn. +- `surrogate_model_rf_min_samples_leaf`: The minimum number of samples required to be at a leaf node. A split point at any depth will only be considered if it leaves at least `min_samples_leaf` training samples in each of the left and right branches. This may have the effect of smoothing the model, especially in regression. Equivalent to `min_samples_leaf` in `RandomForestRegressor` in sklearn. +- `init_design_n_configs`: Number of initial configurations. +- `init_design_n_configs_per_hyperparameter`: Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and `n_configs_per_hyperparameter` is set to 10, then 50 initial configurations will be sampled. +- `init_design_max_ratio`: Use at most `n_trials * init_design_max_ratio` number of configurations in the initial design. Additional configurations are not affected by this parameter. +- `output_directy`: Output directory path, defaults to `"smac3_output"`. The directory in which to save the output. The files are saved in `./output_directory/name/seed`. ## Installation @@ -41,6 +62,7 @@ sampler = SMACSampler( "y": optuna.distributions.IntDistribution(-10, 10), }, n_trials=n_trials, + output_directory="smac3_output", ) study = optuna.create_study(sampler=sampler) study.optimize(objective, n_trials=n_trials) diff --git a/package/samplers/smac_sampler/sampler.py b/package/samplers/smac_sampler/sampler.py index 8c3f412a..a92179b2 100644 --- a/package/samplers/smac_sampler/sampler.py +++ b/package/samplers/smac_sampler/sampler.py @@ -1,6 +1,7 @@ from __future__ import annotations from collections.abc import Sequence +from pathlib import Path from ConfigSpace import Categorical from ConfigSpace import Configuration @@ -96,6 +97,10 @@ class SMACSampler(optunahub.samplers.SimpleBaseSampler): init_design_max_ratio: Use at most ``n_trials * init_design_max_ratio`` number of configurations in the initial design. Additional configurations are not affected by this parameter. + output_directory: + Output directory path, defaults to "smac3_output". + The directory in which to save the output. + The files are saved in `./output_directory/name/seed`. """ def __init__( @@ -114,11 +119,16 @@ def __init__( init_design_n_configs: int | None = None, init_design_n_configs_per_hyperparameter: int = 10, init_design_max_ratio: float = 0.25, + output_directory: str = "smac3_output", ) -> None: super().__init__(search_space) self._cs, self._hp_scale_value = self._convert_to_config_space_design_space(search_space) scenario = Scenario( - configspace=self._cs, deterministic=True, n_trials=n_trials, seed=seed or -1 + configspace=self._cs, + deterministic=True, + n_trials=n_trials, + seed=seed or -1, + output_directory=Path(output_directory), ) surrogate_model = self._get_surrogate_model( scenario, diff --git a/package/visualization/plot_pyribs/LICENSE b/package/visualization/plot_pyribs/LICENSE new file mode 100644 index 00000000..51c2bdfd --- /dev/null +++ b/package/visualization/plot_pyribs/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 Bryon Tjanaka + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/package/visualization/plot_pyribs/README.md b/package/visualization/plot_pyribs/README.md new file mode 100644 index 00000000..979524e3 --- /dev/null +++ b/package/visualization/plot_pyribs/README.md @@ -0,0 +1,74 @@ +--- +author: Bryon Tjanaka +title: Pyribs Visualization Wrappers +description: This visualizaton module provides wrappers around the visualization functions from pyribs, which is useful for plotting results from [CmaMaeSampler](https://hub.optuna.org/samplers/cmamae/). +tags: [visualization, quality diversity, pyribs] +optuna_versions: [4.0.0] +license: MIT License +--- + +## Class or Function Names + +- `plot_grid_archive_heatmap(study: optuna.Study, ax: plt.Axes, **kwargs)` + + - `study`: Optuna study with a sampler that uses pyribs. This function will plot the result archive from the sampler's scheduler. + - `ax`: Axes on which to plot the heatmap. If None, we retrieve the current axes. + - `**kwargs`: All remaining kwargs will be passed to [`grid_archive_heatmap`](https://docs.pyribs.org/en/stable/api/ribs.visualize.grid_archive_heatmap.html). + +## Installation + +```shell +$ pip install ribs[visualize] +``` + +## Example + +A minimal example would be the following: + +```python +import matplotlib.pyplot as plt +import optuna +import optunahub + +module = optunahub.load_module("samplers/cmamae") +CmaMaeSampler = module.CmaMaeSampler + +plot_pyribs = optunahub.load_module(package="visualization/plot_pyribs") +plot_grid_archive_heatmap = plot_pyribs.plot_grid_archive_heatmap + + +def objective(trial: optuna.trial.Trial) -> float: + """Returns an objective followed by two measures.""" + x = trial.suggest_float("x", -10, 10) + y = trial.suggest_float("y", -10, 10) + trial.set_user_attr("m0", 2 * x) + trial.set_user_attr("m1", x + y) + return x**2 + y**2 + + +if __name__ == "__main__": + sampler = CmaMaeSampler( + param_names=["x", "y"], + measure_names=["m0", "m1"], + archive_dims=[20, 20], + archive_ranges=[(-1, 1), (-1, 1)], + archive_learning_rate=0.1, + archive_threshold_min=-10, + n_emitters=1, + emitter_x0={ + "x": 0, + "y": 0, + }, + emitter_sigma0=0.1, + emitter_batch_size=20, + ) + study = optuna.create_study(sampler=sampler) + study.optimize(objective, n_trials=10000) + + fig, ax = plt.subplots(figsize=(8, 6)) + plot_grid_archive_heatmap(study, ax=ax) + plt.savefig("archive.png") + plt.show() +``` + +![Example of this Plot](images/archive.png) diff --git a/package/visualization/plot_pyribs/__init__.py b/package/visualization/plot_pyribs/__init__.py new file mode 100644 index 00000000..6d6630eb --- /dev/null +++ b/package/visualization/plot_pyribs/__init__.py @@ -0,0 +1,44 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import matplotlib.pyplot as plt +import optuna +from ribs.visualize import grid_archive_heatmap + + +if TYPE_CHECKING: + from matplotlib.axes._axes import Axes + + +def plot_grid_archive_heatmap( # type: ignore + study: optuna.Study, + ax: Axes | None = None, + **kwargs, +) -> Axes: + """Wrapper around pyribs grid_archive_heatmap. + + Refer to the `grid_archive_heatmap + `_ + function from pyribs for information. + + Args: + study: Optuna study with a sampler that uses pyribs. This function will + plot the result archive from the sampler's scheduler. + ax: Axes on which to plot the heatmap. If None, we retrieve the current + axes. + kwargs: All remaining kwargs will be passed to `grid_archive_heatmap + `_. + Returns: + The axes on which the plot was created. + """ + if ax is None: + ax = plt.gca() + + archive = study.sampler.scheduler.result_archive + grid_archive_heatmap(archive, ax=ax, **kwargs) + + return ax + + +__all__ = ["plot_grid_archive_heatmap"] diff --git a/package/visualization/plot_pyribs/example.py b/package/visualization/plot_pyribs/example.py new file mode 100644 index 00000000..97b88743 --- /dev/null +++ b/package/visualization/plot_pyribs/example.py @@ -0,0 +1,44 @@ +import matplotlib.pyplot as plt +import optuna +import optunahub + + +module = optunahub.load_module("samplers/cmamae") +CmaMaeSampler = module.CmaMaeSampler + +plot_pyribs = optunahub.load_module(package="visualization/plot_pyribs") +plot_grid_archive_heatmap = plot_pyribs.plot_grid_archive_heatmap + + +def objective(trial: optuna.trial.Trial) -> float: + """Returns an objective followed by two measures.""" + x = trial.suggest_float("x", -10, 10) + y = trial.suggest_float("y", -10, 10) + trial.set_user_attr("m0", 2 * x) + trial.set_user_attr("m1", x + y) + return x**2 + y**2 + + +if __name__ == "__main__": + sampler = CmaMaeSampler( + param_names=["x", "y"], + measure_names=["m0", "m1"], + archive_dims=[20, 20], + archive_ranges=[(-1, 1), (-1, 1)], + archive_learning_rate=0.1, + archive_threshold_min=-10, + n_emitters=1, + emitter_x0={ + "x": 0, + "y": 0, + }, + emitter_sigma0=0.1, + emitter_batch_size=20, + ) + study = optuna.create_study(sampler=sampler) + study.optimize(objective, n_trials=10000) + + fig, ax = plt.subplots(figsize=(8, 6)) + plot_grid_archive_heatmap(study, ax=ax) + plt.savefig("archive.png") + plt.show() diff --git a/package/visualization/plot_pyribs/images/archive.png b/package/visualization/plot_pyribs/images/archive.png new file mode 100644 index 00000000..96a615e0 Binary files /dev/null and b/package/visualization/plot_pyribs/images/archive.png differ diff --git a/recipes/002_registration.py b/recipes/002_registration.py index 17ec7abc..3e554714 100644 --- a/recipes/002_registration.py +++ b/recipes/002_registration.py @@ -1,10 +1,10 @@ """ .. _registration: -How to Register Your Algorithm with OptunaHub +How to Register Your Package with OptunaHub =========================================================== -After implementing your own algorithm, you can register the algorithm as a package with OptunaHub. +After implementing your own feature, you can register it as a package with OptunaHub. To add your package to the `optunahub-registry `__ repository, you need to create a pull request from your fork. Your pull request must be aligned with `the contribution guidelines `__. @@ -12,9 +12,9 @@ See the `template directory `__ for an example of the directory structure. | `package `__ -| └── category (e.g. samplers, pruners, and visualization) +| └── category (samplers, pruners, visualization, or benchmarks) | └── YOUR_PACKAGE_NAME (you need to create this directory and its contents) -| ├── YOUR_ALGORITHM_NAME.py +| ├── YOUR_FEATURE_NAME.py | ├── __init__.py | ├── README.md | ├── LICENSE @@ -24,20 +24,20 @@ | ├── (figure1.png) | └── (numerical_results.png) -An implemented algorithm should be put in the corresponding directory, e.g., a sampler should be put in the ``samplers`` directory. -In the ``samplers`` directory, you should create a directory with a unique identifier. -This unique identifier is the name of your algorithm package, is used to load the package, and is unable to change once it is registered. -The package name must be a valid Python module name, preferably one that is easily searchable. +An implemented feature should be put in the corresponding directory, e.g., a sampler should be put in the ``samplers`` directory. +A newly created directory must be named uniquely in the corresponding directory. +This unique identifier is the name of your package, is used to load the package, and is unable to change once it is registered. +The package name must be a valid Python module name (e.g., please use "_" instead of "-"), preferably one that is easily searchable. Abbreviations are not prohibited in package names, but their abuse should be avoided. The created directory should include the following files: -- ``YOUR_ALGORITHM_NAME.py``: The implementation of your algorithm. -- ``__init__.py``: An initialization file. This file must implement your algorithm or import its implementation from another file, e.g., ``YOUR_ALGORITHM_NAME.py``. -- ``README.md``: A description of your algorithm. This file is used to create an `web page of OptunaHub `_. Let me explain the format of the ``README.md`` file later. -- ``LICENSE``: A license file. This file must contain the license of your algorithm. It should be the MIT license in the alpha version of OptunaHub. -- ``example.py``, ``example.ipynb``: This is optional. This file should contain a simple example of how to use your algorithm (Example: `example.py for Simulated Annealing Sampler `_). You can provide examples in both formats. -- ``requirements.txt``: This is optional. A file that contains the additional dependencies of your algorithm. If there are no additional dependencies other than Optuna and OptunaHub, you do not need to create this file. +- ``YOUR_FEATURE_NAME.py``: The implementation of your feature. +- ``__init__.py``: An initialization file. This file must implement your feature or import its implementation from another file, e.g., ``YOUR_FEATURE_NAME.py``. +- ``README.md``: A description of your package. This file is used to create an `web page of OptunaHub `_. Let me explain the format of the ``README.md`` file later. +- ``LICENSE``: A license file. This file must contain the license of your package. It should be the MIT license in the alpha version of OptunaHub. +- ``example.py``, ``example.ipynb``: This is optional. This file should contain a simple example of how to use your package (Example: `example.py for Simulated Annealing Sampler `_). You can provide examples in both formats. +- ``requirements.txt``: This is optional. A file that contains the additional dependencies of your package. If there are no additional dependencies other than Optuna and OptunaHub, you do not need to create this file. - ``images``: This is optional. A directory that contains images. Only relative references to images in this directory are allowed in README.md, e.g., ``![Numrical Results](images/numerical_results.png)``, and absolute paths to images are not allowed. The first image that appears in README.md will be used as the thumbnail. All files must pass linter and formetter checks to be merged to the optunahub-registry repository. @@ -74,7 +74,7 @@ - ``author`` (string): The author of the package. It can be your name or your organization name. - ``title`` (string): The package title. It should not be a class/function name but a human-readable name. For example, `Demo Sampler` is a good title, but `DemoSampler` is not. - ``description`` (string): A brief description of the package. It should be a one-sentence summary of the package. - - ``tags`` (list[string]): The package tags. It should be a list of strings. The tags must include ``sampler``, ``visualization``, or ``pruner`` depending on the type of the package. You can add other tags as needed. For example, "['sampler', 'LLM']". + - ``tags`` (list[string]): The package tags. It should be a list of strings. The tags must include ``sampler``, ``visualization``, ``pruner``, or ``benchmark`` depending on the type of the package. You can add other tags as needed. For example, "['sampler', 'LLM']". - ``optuna_versions`` (list[string]): A list of Optuna versions that the package supports. It should be a list of strings. You can find your Optuna version with ``python -c 'import optuna; print(optuna.__version__)'``. - ``license`` (string): The license of the package. It should be a string. For example, `MIT License`. The license must be `MIT License` in the current version of OptunaHub. diff --git a/recipes/006_benchmarks_basic.py b/recipes/006_benchmarks_basic.py new file mode 100644 index 00000000..b443383c --- /dev/null +++ b/recipes/006_benchmarks_basic.py @@ -0,0 +1,81 @@ +""" +.. _benchmarks_basic: + +How to Implement Your Benchmark Problems with OptunaHub (Basic) +=============================================================== + +OptunaHub provides the ``optunahub.benchmarks`` module for implementing benchmark problems. +In this tutorial, we will explain how to implement your own benchmark problems using ``optunahub.benchmarks``. +""" + +################################################################################################### +# First of all, import `optuna` and other required modules. +from __future__ import annotations + +import optuna +from optunahub.benchmarks import BaseProblem + + +################################################################################################### +# Next, define your own problem class by inheriting ``BaseProblem`` class. +# Here, let's implement a simple 2-dimensional sphere function. +# +# You need to implement the following methods defined in the ``BaseProblem`` class.: +# +# - ``search_space``: This method returns the dictionary of search space of the problem. Each dictionary element consists of the parameter name and distribution (see `optuna.distributions `__). +# - ``directions``: This method returns the directions (minimize or maximize) of the problem. The return type is the list of `optuna.study.direction `__. +# - ``evaluate``: This method evaluates the objective function given a dictionary of input parameters. +class Sphere2D(BaseProblem): + @property + def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]: + return { + "x0": optuna.distributions.FloatDistribution(low=-5, high=5), + "x1": optuna.distributions.FloatDistribution(low=-5, high=5), + } + + @property + def directions(self) -> list[optuna.study.StudyDirection]: + return [optuna.study.StudyDirection.MINIMIZE] + + def evaluate(self, params: dict[str, float]) -> float: + return params["x0"] ** 2 + params["x1"] ** 2 + + +################################################################################################### +# Since ``BaseProblem`` provides the default implementation of ``__call__(self, trial: optuna.Trial)`` that calls the ``evaluate`` method internally, the problem instance can be directly used as an objective function for ``study.optimize``. +sphere2d = Sphere2D() +study = optuna.create_study(directions=sphere2d.directions) +study.optimize(sphere2d, n_trials=20) + + +################################################################################################### +# The constructor of the problem class can be customized to introduce additional attributes. +# To give an illustration, we show an example with a dynamic dimensional sphere function. +class SphereND(BaseProblem): + def __init__(self, dim: int) -> None: + self.dim = dim + + @property + def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]: + return { + f"x{i}": optuna.distributions.FloatDistribution(low=-5, high=5) + for i in range(self.dim) + } + + @property + def directions(self) -> list[optuna.study.StudyDirection]: + return [optuna.study.StudyDirection.MINIMIZE] + + def evaluate(self, params: dict[str, float]) -> float: + return sum(params[f"x{i}"] ** 2 for i in range(self.dim)) + + +sphere3d = SphereND(dim=3) +study = optuna.create_study(directions=sphere3d.directions) +study.optimize(sphere3d, n_trials=20) + +################################################################################################### +# After implementing your own benchmark problem, you can register it with OptunaHub. +# See :doc:`002_registration` for how to register your benchmark problem with OptunaHub. +# +# In :ref:`benchmarks_advanced`, we explain how to implement more complex benchmark problems such as a problem with dynamic search space. diff --git a/recipes/007_benchmarks_advanced.py b/recipes/007_benchmarks_advanced.py new file mode 100644 index 00000000..464cb9bd --- /dev/null +++ b/recipes/007_benchmarks_advanced.py @@ -0,0 +1,68 @@ +""" +.. _benchmarks_advanced: + +How to Implement Your Benchmark Problems with OptunaHub (Advanced) +================================================================== + +OptunaHub provides the ``optunahub.benchmarks`` module for implementing benchmark problems. +In this tutorial, we will explain how to implement complex benchmark problems such as a problem with dynamic search space using ``optunahub.benchmarks``. + +For the implementation of simple benchmark problems, please refer to :ref:`benchmarks_basic`. +""" + +################################################################################################### +# Implementing a Problem with Dynamic Search Space +# ------------------------------------------------- +# +# Here, let's implement a problem with a dynamic search space. +# +# First of all, import `optuna` and other required modules. +from __future__ import annotations + +import optuna +from optunahub.benchmarks import BaseProblem + + +################################################################################################### +# Next, define your own problem class by inheriting ``BaseProblem`` class. +# To implement a problem with a dynamic search space, ``__call__(self, trial: optuna.Trial)`` method must be overridden so that we can define a dynamic search space in the define-by-run manner. +# Please note that ``direcitons`` property must also be implemented. +class DynamicProblem(BaseProblem): + def __call__(self, trial: optuna.Trial) -> float: + x = trial.suggest_float("x", -5, 5) + if x < 0: + # Parameter `y` exists only when `x` is negative. + y = trial.suggest_float("y", -5, 5) + return x**2 + y + else: + return x**2 + + @property + def directions(self) -> list[optuna.study.StudyDirection]: + return [optuna.study.StudyDirection.MINIMIZE] + + @property + def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]: + # You can implement this property as you like, or leave it unimplemented (``BaseProblem`` provides this default behavior). + raise NotImplementedError + + def evaluate(self, params: dict[str, float]) -> float: + # You can implement this method as you like, or leave it unimplemented (``BaseProblem`` provides this default behavior). + raise NotImplementedError + + +################################################################################################### +# The implementations of the ``search_space`` and ``evaluate`` are non-trivial when the search space is dynamic. +# However, since ``__call__(self, trial: optuna.Trial)`` does not have to depend on both the ``evaluate`` method and the ``search_space`` attribute internally, their implementations are up to users. +# If possible, you could provide their implementations, but this is not necessary to make your benchmark problem work. +# Please note that calling them will result in ``NotImplementedError`` if you leave them unimplemented. + +################################################################################################### +# Then, you can optimize the problem with Optuna as usual. +dynamic_problem = DynamicProblem() +study = optuna.create_study(directions=dynamic_problem.directions) +study.optimize(dynamic_problem, n_trials=20) + +################################################################################################### +# After implementing your own benchmark problem, you can register it with OptunaHub. +# See :doc:`002_registration` for how to register your benchmark problem with OptunaHub. diff --git a/recipes/README.rst b/recipes/README.rst index 59ef6c9d..96a7a667 100644 --- a/recipes/README.rst +++ b/recipes/README.rst @@ -1,4 +1,4 @@ Tutorial ======== -If you are new to Optuna or want a general introduction, we highly recommend the below video. +Through the following tutorials, you can learn how to develop and register features for OptunaHub.