Merge pull request #202 from y0z/feature/benchmarks-tutorial

Add benchmarks tutorials

docs/source/index.rst

@@ -14,6 +14,8 @@ Recipes
 - :doc:`recipes/003_pruner`
 - :doc:`recipes/004_visualization`
 - :doc:`recipes/005_debugging`
+- :doc:`recipes/006_benchmarks_basic`
+- :doc:`recipes/007_benchmarks_advanced`
 
 
 License

recipes/002_registration.py

@@ -1,20 +1,20 @@
 """
 .. _registration:
 
-How to Register Your Algorithm with OptunaHub
+How to Register Your Package with OptunaHub
 ===========================================================
 
-After implementing your own algorithm, you can register the algorithm as a package with OptunaHub.
+After implementing your own feature, you can register it as a package with OptunaHub.
 To add your package to the `optunahub-registry <https://github.com/optuna/optunahub-registry>`__ repository, you need to create a pull request from your fork.
 Your pull request must be aligned with `the contribution guidelines <https://github.com/optuna/optunahub-registry/blob/main/CONTRIBUTING.md>`__.
 
 The following is an example of the directory structure of a package.
 See the `template directory <https://github.com/optuna/optunahub-registry/tree/main/template>`__ for an example of the directory structure.
 
 | `package <https://github.com/optuna/optunahub-registry/tree/main/package>`__
-| └── category (e.g. samplers, pruners, and visualization)
+| └── category (samplers, pruners, visualization, or benchmarks)
 |     └── YOUR_PACKAGE_NAME (you need to create this directory and its contents)
-|         ├── YOUR_ALGORITHM_NAME.py
+|         ├── YOUR_FEATURE_NAME.py
 |         ├── __init__.py
 |         ├── README.md
 |         ├── LICENSE
@@ -24,20 +24,20 @@
 |             ├──  (figure1.png)
 |             └──  (numerical_results.png)
 
-An implemented algorithm should be put in the corresponding directory, e.g., a sampler should be put in the ``samplers`` directory.
-In the ``samplers`` directory, you should create a directory with a unique identifier.
-This unique identifier is the name of your algorithm package, is used to load the package, and is unable to change once it is registered.
-The package name must be a valid Python module name, preferably one that is easily searchable.
+An implemented feature should be put in the corresponding directory, e.g., a sampler should be put in the ``samplers`` directory.
+A newly created directory must be named uniquely in the corresponding directory.
+This unique identifier is the name of your package, is used to load the package, and is unable to change once it is registered.
+The package name must be a valid Python module name (e.g., please use "_" instead of "-"), preferably one that is easily searchable.
 Abbreviations are not prohibited in package names, but their abuse should be avoided.
 
 The created directory should include the following files:
 
-- ``YOUR_ALGORITHM_NAME.py``: The implementation of your algorithm.
-- ``__init__.py``: An initialization file. This file must implement your algorithm or import its implementation from another file, e.g., ``YOUR_ALGORITHM_NAME.py``.
-- ``README.md``: A description of your algorithm. This file is used to create an `web page of OptunaHub <https://hub.optuna.org/>`_. Let me explain the format of the ``README.md`` file later.
-- ``LICENSE``: A license file. This file must contain the license of your algorithm. It should be the MIT license in the alpha version of OptunaHub.
-- ``example.py``, ``example.ipynb``: This is optional. This file should contain a simple example of how to use your algorithm (Example: `example.py for Simulated Annealing Sampler <https://github.com/optuna/optunahub-registry/blob/main/package/samplers/simulated_annealing/example.py>`_). You can provide examples in both formats.
-- ``requirements.txt``: This is optional. A file that contains the additional dependencies of your algorithm. If there are no additional dependencies other than Optuna and OptunaHub, you do not need to create this file.
+- ``YOUR_FEATURE_NAME.py``: The implementation of your feature.
+- ``__init__.py``: An initialization file. This file must implement your feature or import its implementation from another file, e.g., ``YOUR_FEATURE_NAME.py``.
+- ``README.md``: A description of your package. This file is used to create an `web page of OptunaHub <https://hub.optuna.org/>`_. Let me explain the format of the ``README.md`` file later.
+- ``LICENSE``: A license file. This file must contain the license of your package. It should be the MIT license in the alpha version of OptunaHub.
+- ``example.py``, ``example.ipynb``: This is optional. This file should contain a simple example of how to use your package (Example: `example.py for Simulated Annealing Sampler <https://github.com/optuna/optunahub-registry/blob/main/package/samplers/simulated_annealing/example.py>`_). You can provide examples in both formats.
+- ``requirements.txt``: This is optional. A file that contains the additional dependencies of your package. If there are no additional dependencies other than Optuna and OptunaHub, you do not need to create this file.
 - ``images``: This is optional. A directory that contains images. Only relative references to images in this directory are allowed in README.md, e.g., ``![Numrical Results](images/numerical_results.png)``, and absolute paths to images are not allowed. The first image that appears in README.md will be used as the thumbnail.
 
 All files must pass linter and formetter checks to be merged to the optunahub-registry repository.
@@ -74,7 +74,7 @@
   - ``author`` (string): The author of the package. It can be your name or your organization name.
   - ``title`` (string): The package title. It should not be a class/function name but a human-readable name. For example, `Demo Sampler` is a good title, but `DemoSampler` is not.
   - ``description`` (string): A brief description of the package. It should be a one-sentence summary of the package.
-  - ``tags`` (list[string]): The package tags. It should be a list of strings. The tags must include ``sampler``, ``visualization``, or ``pruner`` depending on the type of the package. You can add other tags as needed. For example, "['sampler', 'LLM']".
+  - ``tags`` (list[string]): The package tags. It should be a list of strings. The tags must include ``sampler``, ``visualization``, ``pruner``, or ``benchmark`` depending on the type of the package. You can add other tags as needed. For example, "['sampler', 'LLM']".
   - ``optuna_versions`` (list[string]): A list of Optuna versions that the package supports. It should be a list of strings. You can find your Optuna version with ``python -c 'import optuna; print(optuna.__version__)'``.
   - ``license`` (string): The license of the package. It should be a string. For example, `MIT License`. The license must be `MIT License` in the current version of OptunaHub.
 

recipes/006_benchmarks_basic.py

@@ -0,0 +1,81 @@
+"""
+.. _benchmarks_basic:
+
+How to Implement Your Benchmark Problems with OptunaHub (Basic)
+===============================================================
+
+OptunaHub provides the ``optunahub.benchmarks`` module for implementing benchmark problems.
+In this tutorial, we will explain how to implement your own benchmark problems using ``optunahub.benchmarks``.
+"""
+
+###################################################################################################
+# First of all, import `optuna` and other required modules.
+from __future__ import annotations
+
+import optuna
+from optunahub.benchmarks import BaseProblem
+
+
+###################################################################################################
+# Next, define your own problem class by inheriting ``BaseProblem`` class.
+# Here, let's implement a simple 2-dimensional sphere function.
+#
+# You need to implement the following methods defined in the ``BaseProblem`` class.:
+#
+# - ``search_space``: This method returns the dictionary of search space of the problem. Each dictionary element consists of the parameter name and distribution (see `optuna.distributions <https://optuna.readthedocs.io/en/stable/reference/distributions.html>`__).
+# - ``directions``: This method returns the directions (minimize or maximize) of the problem. The return type is the list of `optuna.study.direction <https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.StudyDirection.html>`__.
+# - ``evaluate``: This method evaluates the objective function given a dictionary of input parameters.
+class Sphere2D(BaseProblem):
+    @property
+    def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]:
+        return {
+            "x0": optuna.distributions.FloatDistribution(low=-5, high=5),
+            "x1": optuna.distributions.FloatDistribution(low=-5, high=5),
+        }
+
+    @property
+    def directions(self) -> list[optuna.study.StudyDirection]:
+        return [optuna.study.StudyDirection.MINIMIZE]
+
+    def evaluate(self, params: dict[str, float]) -> float:
+        return params["x0"] ** 2 + params["x1"] ** 2
+
+
+###################################################################################################
+# Since ``BaseProblem`` provides the default implementation of ``__call__(self, trial: optuna.Trial)`` that calls the ``evaluate`` method internally, the problem instance can be directly used as an objective function for ``study.optimize``.
+sphere2d = Sphere2D()
+study = optuna.create_study(directions=sphere2d.directions)
+study.optimize(sphere2d, n_trials=20)
+
+
+###################################################################################################
+# The constructor of the problem class can be customized to introduce additional attributes.
+# To give an illustration, we show an example with a dynamic dimensional sphere function.
+class SphereND(BaseProblem):
+    def __init__(self, dim: int) -> None:
+        self.dim = dim
+
+    @property
+    def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]:
+        return {
+            f"x{i}": optuna.distributions.FloatDistribution(low=-5, high=5)
+            for i in range(self.dim)
+        }
+
+    @property
+    def directions(self) -> list[optuna.study.StudyDirection]:
+        return [optuna.study.StudyDirection.MINIMIZE]
+
+    def evaluate(self, params: dict[str, float]) -> float:
+        return sum(params[f"x{i}"] ** 2 for i in range(self.dim))
+
+
+sphere3d = SphereND(dim=3)
+study = optuna.create_study(directions=sphere3d.directions)
+study.optimize(sphere3d, n_trials=20)
+
+###################################################################################################
+# After implementing your own benchmark problem, you can register it with OptunaHub.
+# See :doc:`002_registration` for how to register your benchmark problem with OptunaHub.
+#
+# In :ref:`benchmarks_advanced`, we explain how to implement more complex benchmark problems such as a problem with dynamic search space.

recipes/007_benchmarks_advanced.py

@@ -0,0 +1,68 @@
+"""
+.. _benchmarks_advanced:
+
+How to Implement Your Benchmark Problems with OptunaHub (Advanced)
+==================================================================
+
+OptunaHub provides the ``optunahub.benchmarks`` module for implementing benchmark problems.
+In this tutorial, we will explain how to implement complex benchmark problems such as a problem with dynamic search space using ``optunahub.benchmarks``.
+
+For the implementation of simple benchmark problems, please refer to :ref:`benchmarks_basic`.
+"""
+
+###################################################################################################
+# Implementing a Problem with Dynamic Search Space
+# -------------------------------------------------
+#
+# Here, let's implement a problem with a dynamic search space.
+#
+# First of all, import `optuna` and other required modules.
+from __future__ import annotations
+
+import optuna
+from optunahub.benchmarks import BaseProblem
+
+
+###################################################################################################
+# Next, define your own problem class by inheriting ``BaseProblem`` class.
+# To implement a problem with a dynamic search space, ``__call__(self, trial: optuna.Trial)`` method must be overridden so that we can define a dynamic search space in the define-by-run manner.
+# Please note that ``direcitons`` property must also be implemented.
+class DynamicProblem(BaseProblem):
+    def __call__(self, trial: optuna.Trial) -> float:
+        x = trial.suggest_float("x", -5, 5)
+        if x < 0:
+            # Parameter `y` exists only when `x` is negative.
+            y = trial.suggest_float("y", -5, 5)
+            return x**2 + y
+        else:
+            return x**2
+
+    @property
+    def directions(self) -> list[optuna.study.StudyDirection]:
+        return [optuna.study.StudyDirection.MINIMIZE]
+
+    @property
+    def search_space(self) -> dict[str, optuna.distributions.BaseDistribution]:
+        # You can implement this property as you like, or leave it unimplemented (``BaseProblem`` provides this default behavior).
+        raise NotImplementedError
+
+    def evaluate(self, params: dict[str, float]) -> float:
+        # You can implement this method as you like, or leave it unimplemented (``BaseProblem`` provides this default behavior).
+        raise NotImplementedError
+
+
+###################################################################################################
+# The implementations of the ``search_space`` and ``evaluate`` are non-trivial when the search space is dynamic.
+# However, since ``__call__(self, trial: optuna.Trial)`` does not have to depend on both the ``evaluate`` method and the ``search_space`` attribute internally, their implementations are up to users.
+# If possible, you could provide their implementations, but this is not necessary to make your benchmark problem work.
+# Please note that calling them will result in ``NotImplementedError`` if you leave them unimplemented.
+
+###################################################################################################
+# Then, you can optimize the problem with Optuna as usual.
+dynamic_problem = DynamicProblem()
+study = optuna.create_study(directions=dynamic_problem.directions)
+study.optimize(dynamic_problem, n_trials=20)
+
+###################################################################################################
+# After implementing your own benchmark problem, you can register it with OptunaHub.
+# See :doc:`002_registration` for how to register your benchmark problem with OptunaHub.

recipes/README.rst

@@ -1,4 +1,4 @@
 Tutorial
 ========
 
-If you are new to Optuna or want a general introduction, we highly recommend the below video.
+Through the following tutorials, you can learn how to develop and register features for OptunaHub.