Add GP-based Bayesian optimization algorithms

[HideakiImamura]

Motivation

Gaussian processes(GP) is a well-known nonparametric model for black-box optimization. Optuna supports GP based algorithms as optuna.samplers.GPSampler and optuna.integration.BoTorchSampler, but the former only supports the expected improvement (EI) as an acquisition function and the latter is too slow to use in production. I suggest that how about customizing the implementation of GPSampler and introduce the probability improvement (PI), upper confidence bound (UCB), and Thompson sampling (TS) as acquisition functions.

Description

It would be great to customize the implementation of GPSampler to introduce PI, UCB, and TS based GP sampler in OptunaHub. The existing implementation is here and here. See the textbook for more details of the acquisition functions.

• GPEISampler: This should be an alias to GPSampler.
• GPPISampler
• GPUCBSampler
• GPTSSampler

Of course it's welcome to add more advanced acquisition functions in OptunaHub, e.g., entropy search, predictive entropy search, knowledge gradient, and so on.

Alternatives (optional)

No response

Additional context (optional)

No response