Add a sampler that automatically picks a sampler based on developer recommendations #168

nabenabe0928 · 2024-10-22T09:22:33Z

Contributor Agreements

Please read the contributor agreements and if you agree, please click the checkbox below.

I agree to the contributor agreements.

Tip

Please follow the Quick TODO list to smoothly merge your PR.

Motivation

In this PR, we provide a more general sampler for a wider range of users.
By using this sampler, users can use the recommended sampler based on the provided search space without the deep knowledge of black-box optimization.

TODO List towards PR Merge

Please remove this section if this PR is not an addition of a new package.
Otherwise, please check the following TODO list:

Copy ./template/ to create your package
Replace <COPYRIGHT HOLDER> in LICENSE of your package with your name
Fill out README.md in your package
Add import statements of your function or class names to be used in __init__.py
(Optional) Add from __future__ import annotations at the head of any Python files that include typing to support older Python versions
Apply the formatter based on the tips in README.md
Check whether your module works as intended based on the tips in README.md

…h-fallback Support multi-objective and constrained optimization.

nabenabe0928 · 2024-10-22T09:25:11Z

@not522 @toshihikoyanase
Could you review this PR?

toshihikoyanase · 2024-10-22T10:01:46Z

I'm sorry, but I'm unavailable at the moment. @c-bata, could you please review this PR? Thank you!

package/samplers/auto_sampler/tests/test_sampler.py

package/samplers/auto_sampler/_sampler.py

package/samplers/auto_sampler/tests/test_auto_sampler.py

package/samplers/auto_sampler/README.md

package/samplers/auto_sampler/tests/test_sampler.py

package/samplers/auto_sampler/_sampler.py

nabenabe0928 · 2024-10-30T05:26:55Z

@not522 @c-bata
I addressed all your comments:)
But if you come up with any additional suggestions, please feel free to add them!

c-bata

Changes look almost good to me. However, it looks that there is a concurrency issue probably caused by GPSampler. Could you take a look please?

import math
import numpy as np
import optuna
import optunahub

optuna.logging.set_verbosity(optuna.logging.DEBUG)


def objective(trial: optuna.Trial) -> float:
    x = 10 ** trial.suggest_float("log10_x", -5, 0)
    y = 2 ** trial.suggest_float("log2_y", 1, 10)
    return x**2 + y**2


if __name__ == "__main__":
    module = optunahub.load_local_module(package="samplers/auto_sampler", registry_root="./package")
    sampler = module.AutoSampler()
    study = optuna.create_study(sampler=sampler)
    study.optimize(objective, n_trials=260, n_jobs=10)
    print(study.best_trial.value, study.best_trial.params)

error traces

$ python example_auto_sampler.py
:
[I 2024-10-30 17:55:38,829] Trial 9 finished with value: 5.756463197970189 and parameters: {'log10_x': -0.9301663717569495, 'log2_y': 1.2608608210121335}. Best is trial 9 with value: 5.756463197970189.
[I 2024-10-30 17:55:38,831] Trial 11 finished with value: 1293.2027666641377 and parameters: {'log10_x': -1.5116363820199834, 'log2_y': 5.168365863169032}. Best is trial 9 with value: 5.756463197970189.
[D 2024-10-30 17:55:38,831] Sample trial#12 with GPSampler.
[I 2024-10-30 17:55:38,832] Trial 10 finished with value: 77.03732990068637 and parameters: {'log10_x': -0.8844302887115854, 'log2_y': 3.133583445415238}. Best is trial 9 with value: 5.756463197970189.
[D 2024-10-30 17:55:38,833] Sample trial#21 with GPSampler.
[I 2024-10-30 17:55:38,834] Trial 12 finished with value: 10.630071734643709 and parameters: {'log10_x': -3.288115982382003, 'log2_y': 1.705039695741501}. Best is trial 9 with value: 5.756463197970189.
[D 2024-10-30 17:55:38,835] Sample trial#22 with GPSampler.
[D 2024-10-30 17:55:38,836] Sample trial#23 with GPSampler.
[D 2024-10-30 17:55:38,837] Sample trial#24 with GPSampler.
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
ValueError: got differing extents in dimension 0 (got 29 and 17)
Traceback (most recent call last):
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
ValueError: got differing extents in dimension 0 (got 29 and 17)
Traceback (most recent call last):
ValueError: got differing extents in dimension 0 (got 29 and 17)
Exception ignored in: 'scipy.stats._sobol._initialize_v'
Traceback (most recent call last):
Exception ignored in: 'scipy.stats._sobol._initialize_v'
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
Traceback (most recent call last):
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
Exception ignored in: 'scipy.stats._sobol._initialize_v'
Traceback (most recent call last):
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
ValueError: got differing extents in dimension 0 (got 29 and 17)
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
ValueError: got differing extents in dimension 0 (got 29 and 17)
  File "<stringsource>", line 1249, in View.MemoryView._err_extents
ValueError: got differing extents in dimension 0 (got 29 and 17)
ValueError: got differing extents in dimension 0 (got 29 and 17)
Exception ignored in: 'scipy.stats._sobol._initialize_v'
Traceback (most recent call last):
ValueError: got differing extents in dimension 0 (got 29 and 17)
ValueError: got differing extents in dimension 0 (got 29 and 17)
Exception ignored in: 'scipy.stats._sobol._initialize_v'
Traceback (most recent call last):
  File "<stringsource>", line 1294, in View.MemoryView.memoryview_copy_contents

:
[D 2024-10-30 17:56:43,980] Sample trial#259 with CmaEsSampler.
[I 2024-10-30 17:56:43,998] Trial 254 finished with value: 22.78847730915046 and parameters: {'log10_x': -5.0, 'log2_y': 2.2551163120253923}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,002] Trial 259 finished with value: 34.58067968677882 and parameters: {'log10_x': -1.4001358515209974, 'log2_y': 2.555914119530404}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,166] Trial 250 finished with value: 23.150778844607334 and parameters: {'log10_x': -5.0, 'log2_y': 2.266494412355062}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,277] Trial 253 finished with value: 23.16306513095228 and parameters: {'log10_x': -5.0, 'log2_y': 2.26687713517129}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,305] Trial 251 finished with value: 23.171298911344138 and parameters: {'log10_x': -5.0, 'log2_y': 2.2671335071860943}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,345] Trial 249 finished with value: 11.990664074969528 and parameters: {'log10_x': -5.0, 'log2_y': 1.791919828070403}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,453] Trial 255 finished with value: 23.06595549414097 and parameters: {'log10_x': -5.0, 'log2_y': 2.2638465756737034}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,520] Trial 256 finished with value: 64.4372072980525 and parameters: {'log10_x': 0.0, 'log2_y': 2.993628677716093}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,635] Trial 257 finished with value: 23.27703533210791 and parameters: {'log10_x': -5.0, 'log2_y': 2.270417708349971}. Best is trial 24 with value: 4.0000000001.
[I 2024-10-30 17:56:44,742] Trial 258 finished with value: 64.47728862564898 and parameters: {'log10_x': 0.0, 'log2_y': 2.994084300488396}. Best is trial 24 with value: 4.0000000001.
4.0000000001 {'log10_x': -5.0, 'log2_y': 1.0}

package/samplers/auto_sampler/_sampler.py

package/samplers/auto_sampler/tests/test_sampler.py

c-bata · 2024-10-31T06:39:01Z

Changes look almost good to me. However, it looks that there is a concurrency issue probably caused by GPSampler. Could you take a look please?

import math
import numpy as np
import optuna
import optunahub

optuna.logging.set_verbosity(optuna.logging.DEBUG)


def objective(trial: optuna.Trial) -> float:
    x = 10 ** trial.suggest_float("log10_x", -5, 0)
    y = 2 ** trial.suggest_float("log2_y", 1, 10)
    return x**2 + y**2


if __name__ == "__main__":
    module = optunahub.load_local_module(package="samplers/auto_sampler", registry_root="./package")
    sampler = module.AutoSampler()
    study = optuna.create_study(sampler=sampler)
    study.optimize(objective, n_trials=260, n_jobs=10)
    print(study.best_trial.value, study.best_trial.params)

I opened a pull request to address this concurrency issue.
optuna/optuna#5737

c-bata

LGTM!

…t from requirements

not522

LGTM!

nabenabe0928 and others added 23 commits October 4, 2024 10:57

Add AutoSampler

0545dbe

Enhance README

e47861a

Add LICENSE

a8edf5e

Update the random seed handling

f101e0f

Make WS-CMA-ES distributed setup robust

6df7a95

Support multi-objective and constrained optimization.

f71caeb

Merge pull request #1 from toshihikoyanase/prototype-auto-sampler-wit…

fe3da2e

…h-fallback Support multi-objective and constrained optimization.

Update auto sampler

2a7d4bb

Support constraints_func

b5b204d

Replace the random seed with the max int32

bf07aaf

Add the switching algorithm for multi-objective optimization

c037248

Refactor the determination algo

ee0d97b

Add NSGAIIISampler

b3af183

Make magical numbers to class attributes

317369c

Verify the implementation by a simple example

370d522

Merge remote-tracking branch 'upstream/main' into add-auto-sampler

0debbf5

Add unittests for MO

45e4e35

Add unittests for single objective

0949605

Enhance README

8204608

Fix the test path in README

baf4f7e

Add thumbnail

fdd1558

Fix README

0c1ed85

Fix README and sampler

133ad55

nabenabe0928 added the new-package New packages label Oct 22, 2024

nabenabe0928 assigned not522 and toshihikoyanase Oct 22, 2024

toshihikoyanase assigned c-bata and unassigned toshihikoyanase Oct 22, 2024

nabenabe0928 added 3 commits October 29, 2024 05:06

Apply c-bata's comment

7bd35d1

Apply c-bata's comment

c2be135

Fix local thread and its tests

9a4d0d9

nabenabe0928 force-pushed the add-auto-sampler branch from ad5d30b to 9a4d0d9 Compare October 29, 2024 06:09

Remove algo description

b8c4588

not522 reviewed Oct 29, 2024

View reviewed changes

package/samplers/auto_sampler/tests/test_sampler.py Show resolved Hide resolved

Add ref of test

a2a1896

nabenabe0928 commented Oct 29, 2024

View reviewed changes

package/samplers/auto_sampler/tests/test_sampler.py Show resolved Hide resolved

not522 reviewed Oct 29, 2024

View reviewed changes

nabenabe0928 added 2 commits October 29, 2024 08:55

Apply not's comment

d56cbe3

Apply not's comment

90193ff

c-bata reviewed Oct 30, 2024

View reviewed changes

package/samplers/auto_sampler/README.md Outdated Show resolved Hide resolved

package/samplers/auto_sampler/tests/test_sampler.py Outdated Show resolved Hide resolved

c-bata reviewed Oct 30, 2024

View reviewed changes

package/samplers/auto_sampler/_sampler.py Outdated Show resolved Hide resolved

nabenabe0928 added 2 commits October 30, 2024 05:44

Apply c-bata's comments

d37f6c9

Apply c-bata's comments

13e63a6

c-bata reviewed Oct 30, 2024

View reviewed changes

not522 reviewed Oct 30, 2024

View reviewed changes

package/samplers/auto_sampler/_sampler.py Outdated Show resolved Hide resolved

y0z reviewed Oct 31, 2024

View reviewed changes

package/samplers/auto_sampler/tests/test_sampler.py Outdated Show resolved Hide resolved

nabenabe0928 added 2 commits October 31, 2024 05:29

Apply ozaki's comment

396b703

Apply not's comment

a5bc25e

c-bata mentioned this pull request Oct 31, 2024

Fix a concurrency issue in GPSampler optuna/optuna#5737

Merged

nabenabe0928 added 2 commits October 31, 2024 08:44

Apply the feedback from the mob review

ea89dd1

Add an Optuna version constraint for GPSampler with n_jobs>1

5660f62

nabenabe0928 force-pushed the add-auto-sampler branch from 92bbfa0 to 5660f62 Compare October 31, 2024 07:49

c-bata approved these changes Oct 31, 2024

View reviewed changes

Add the note about the Optuna version constraint and remove constrain…

b0da194

…t from requirements

not522 approved these changes Oct 31, 2024

View reviewed changes

not522 merged commit 6024f49 into optuna:main Oct 31, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a sampler that automatically picks a sampler based on developer recommendations #168

Add a sampler that automatically picks a sampler based on developer recommendations #168

nabenabe0928 commented Oct 22, 2024

nabenabe0928 commented Oct 22, 2024

toshihikoyanase commented Oct 22, 2024

nabenabe0928 commented Oct 30, 2024

c-bata left a comment

c-bata commented Oct 31, 2024

c-bata left a comment

not522 left a comment

Add a sampler that automatically picks a sampler based on developer recommendations #168

Add a sampler that automatically picks a sampler based on developer recommendations #168

Conversation

nabenabe0928 commented Oct 22, 2024

Contributor Agreements

Motivation

TODO List towards PR Merge

nabenabe0928 commented Oct 22, 2024

toshihikoyanase commented Oct 22, 2024

nabenabe0928 commented Oct 30, 2024

c-bata left a comment

Choose a reason for hiding this comment

c-bata commented Oct 31, 2024

c-bata left a comment

Choose a reason for hiding this comment

not522 left a comment

Choose a reason for hiding this comment