diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 000000000..88263a450
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,2 @@
+*.ipynb linguist-detectable=false
+.pages linguist-language=YAML
diff --git a/docs/coding/machine-learning/.pages b/docs/coding/machine-learning/.pages
index f7086f0f0..f3070fe99 100644
--- a/docs/coding/machine-learning/.pages
+++ b/docs/coding/machine-learning/.pages
@@ -1,3 +1,4 @@
nav:
- 线性模型: linear-models.ipynb
- - 决策树: decision-tree.ipynb
\ No newline at end of file
+ - 决策树: decision-tree.ipynb
+ - 贝叶斯优化: bayesian-optimization.ipynb
diff --git a/docs/coding/machine-learning/bayesian-optimization.ipynb b/docs/coding/machine-learning/bayesian-optimization.ipynb
new file mode 100644
index 000000000..be40ecba3
--- /dev/null
+++ b/docs/coding/machine-learning/bayesian-optimization.ipynb
@@ -0,0 +1,7869 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# 贝叶斯优化\n",
+ "\n",
+ "贝叶斯优化是一类可以用于优化黑盒函数的优化方法。对于黑盒函数来说，我们无法得知其显式的表达式，只能通过输入一个数据，观测其输出的方法寻找它的最大值。借助于贝叶斯统计的思想，贝叶斯优化最初认为函数是一个完全随机的函数，可以通过不断地观测来更新对函数的估计（后验）。在每一步迭代中，算法会根据当前的后验，通过采集函数来选择下一个观测的点。贝叶斯优化的优点是可以在较少的迭代次数内找到全局最优解，因此在实际应用中被广泛使用。\n",
+ "\n",
+ "假设目标函数的形式如下："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "from scipy.stats import norm\n",
+ "\n",
+ "def objective(x):\n",
+ " return norm.pdf(x, 3, 2) * 1.5 + norm.pdf(x, 7, 1) + norm.pdf(x, 11, 2)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "初步对该函数进行可视化"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from matplotlib import pyplot as plt\n",
+ "%config InlineBackend.figure_format = 'svg'\n",
+ "\n",
+ "x = np.linspace(-5, 20, 50)\n",
+ "plt.plot(x, objective(x))\n",
+ "plt.xlabel('x')\n",
+ "plt.ylabel('f(x)')\n",
+ "plt.title('Objective function')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "采集函数的目标是选出后验中最优的点。在此处，我们使用期望改进（Expected Improvement, EI）来选择下一个采集点。EI的定义如下：\n",
+ "\n",
+ "$$\n",
+ "\\begin{aligned}\n",
+ "\\text{EI}(x) &= \\mathbb{E}[\\max(0, \\tilde f(x) - f(x^*) - \\xi)] \\\\\n",
+ "\\end{aligned}\n",
+ "$$\n",
+ "\n",
+ "其中$\\tilde f(x)$是目标函数在$x$处的预测值，$f(x^*)$是当前最优的函数值，$\\xi$是一个控制探索和开发的参数，当$\\xi$较大时，算法更倾向于探索，当$\\xi$较小时，算法更倾向于开发。对于高斯分布，对于给定$x$, $\\tilde f(x)$的分布是一个高斯分布，其均值为$\\mu(x)$，方差为$\\sigma^2(x)$，令\n",
+ "\n",
+ "$$\n",
+ "z(x) = \\frac{\\mu(x) - f(x^*) - \\xi}{\\sigma(x)}\n",
+ "$$\n",
+ "\n",
+ "则期望改进为\n",
+ "\n",
+ "$$\n",
+ "\\text{EI}(x) = \\sigma(x)[z(x)\\Phi(z(x)) + \\phi(z(x))]\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from typing import Callable\n",
+ "from sklearn.gaussian_process import GaussianProcessRegressor\n",
+ "\n",
+ "GP = GaussianProcessRegressor\n",
+ "Acquisition = Callable[[np.ndarray, GP, float], np.ndarray]\n",
+ "\n",
+ "def expected_improvement(x: np.ndarray, gp: GP, y_max: float, xi: float = 0.01):\n",
+ " x = np.array(x).reshape(-1, 1)\n",
+ " mu, sigma = gp.predict(x, return_std=True) # type: ignore\n",
+ " sigma = sigma.reshape(-1, 1)\n",
+ " mu = mu.reshape(-1, 1)\n",
+ "\n",
+ " with np.errstate(divide='warn'):\n",
+ " Z = (mu - y_max - xi) / sigma # Normalized Z-score\n",
+ " ei = sigma * (Z * norm.cdf(Z) + norm.pdf(Z))\n",
+ "\n",
+ " return ei"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "根据采集函数，我们可以在范围中选择一个点使得EI最大，然后观测该点的函数值，更新后验，重复这个过程直到达到迭代次数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def next_point(\n",
+ " x_min: float, x_max: float, n_samples: int,\n",
+ " acquisition: Acquisition, gp: GP, y_max: float\n",
+ "):\n",
+ " x = np.random.uniform(x_min, x_max, n_samples).reshape(-1, 1)\n",
+ " ei = acquisition(x, gp, y_max)\n",
+ " return x[np.argmax(ei)]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "此后，我们可以使用高斯过程来拟合目标函数。`sklearn`中提供了高斯过程的实现。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/wg/gb8y92_d43j60wvfw6bs_cz00000gn/T/ipykernel_54343/3771838051.py:19: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n",
+ " fig.show()\n"
+ ]
+ },
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from sklearn.gaussian_process.kernels import RBF\n",
+ "\n",
+ "kernel = RBF(length_scale=1.0, length_scale_bounds=(1e-2, 1e2))\n",
+ "gp = GP(kernel=kernel, alpha=1e-5, n_restarts_optimizer=10)\n",
+ "\n",
+ "def plot_gp(ax, gp: GP, x_min: float, x_max: float, ci: float = 0.95):\n",
+ " x = np.linspace(x_min, x_max, 50).reshape(-1, 1)\n",
+ " y, sigma = gp.predict(x, return_std=True)\n",
+ "\n",
+ " ci_coef = norm.ppf((1 + ci) / 2)\n",
+ "\n",
+ " ax.plot(x, y, linewidth=1)\n",
+ " ax.fill_between(x.flatten(), y - ci_coef * sigma, y + ci_coef * sigma, alpha=0.2)\n",
+ " ax.set_xlabel('x')\n",
+ " ax.set_ylabel('f(x)')\n",
+ "\n",
+ "fig, ax = plt.subplots(1, 1)\n",
+ "plot_gp(ax, gp, -5, 20)\n",
+ "fig.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "此后，我们可以迭代地优化目标函数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/var/folders/wg/gb8y92_d43j60wvfw6bs_cz00000gn/T/ipykernel_54343/3753732084.py:63: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown\n",
+ " fig.show()\n"
+ ]
+ },
+ {
+ "data": {
+ "image/svg+xml": [
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "fig, ax = plt.subplots(4, 4, sharex=False, sharey=False, figsize=(9, 9))\n",
+ "ax_flatten = ax.flatten()\n",
+ "\n",
+ "x_min = -5\n",
+ "x_max = 20\n",
+ "x_samples = np.random.uniform(x_min, x_max, 3).reshape(-1, 1)\n",
+ "y_samples = objective(x_samples)\n",
+ "y_max = y_samples.max()\n",
+ "gp = gp.fit(x_samples, y_samples)\n",
+ "\n",
+ "for i, current_ax in enumerate(ax_flatten):\n",
+ " # Setup plot\n",
+ " current_ax.set_xlim(x_min, x_max)\n",
+ " current_ax.set_ylim(-1, 1)\n",
+ " if i // 4 != 3:\n",
+ " current_ax.get_xaxis().set_visible(False)\n",
+ " if i % 4 != 0:\n",
+ " current_ax.get_yaxis().set_visible(False)\n",
+ "\n",
+ " # Plot prediction\n",
+ " plot_gp(current_ax, gp, x_min, x_max)\n",
+ "\n",
+ " # Plot function\n",
+ " x_grid = np.linspace(x_min, x_max, 50).reshape(-1, 1)\n",
+ "\n",
+ " # Plot samples\n",
+ " current_ax.plot(\n",
+ " x_grid, objective(x_grid), '--', color='gray', linewidth=0.5\n",
+ " )\n",
+ " current_ax.scatter(x_samples, y_samples, color='red', s=4, zorder=10)\n",
+ "\n",
+ " # Plot optimal point\n",
+ " x_optimal = x_samples[np.argmax(y_samples)].item()\n",
+ " current_ax.scatter(x_optimal, y_max, color='green', s=10, zorder=20)\n",
+ " current_ax.text(\n",
+ " 0.5, 0.9,\n",
+ " f'Best: x={x_optimal:.2f}, y={y_max:.2f}',\n",
+ " ha='center', transform=current_ax.transAxes\n",
+ " )\n",
+ "\n",
+ " point = next_point(x_min, x_max, 100, expected_improvement, gp, y_max)\n",
+ "\n",
+ " # Plot next point\n",
+ " current_ax.plot(\n",
+ " [point, point], [-0.8, 0.8], '--', color='red', linewidth=0.5\n",
+ " )\n",
+ " current_ax.text(\n",
+ " 0.5, 0.05,\n",
+ " f'Next x: {point[0]:.2f}',\n",
+ " ha='center', transform=current_ax.transAxes\n",
+ " )\n",
+ "\n",
+ " # Update samples\n",
+ " y = objective(point).item()\n",
+ " x_samples = np.vstack([x_samples, point])\n",
+ " y_samples = np.append(y_samples, y)\n",
+ "\n",
+ " y_max = max(y_max, y)\n",
+ "\n",
+ " # Re-fit model\n",
+ " gp = gp.fit(x_samples, y_samples)\n",
+ "\n",
+ "fig.show()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.10"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/docs/coding/machine-learning/index.md b/docs/coding/machine-learning/index.md
index 38e63699b..8bfd8ba20 100644
--- a/docs/coding/machine-learning/index.md
+++ b/docs/coding/machine-learning/index.md
@@ -2,5 +2,6 @@
部分机器学习算法的Python实现
-1. [线性模型](linear-models.ipynb)
-2. [决策树](decision-tree.ipynb)
\ No newline at end of file
+* [线性模型](linear-models.ipynb)
+* [决策树](decision-tree.ipynb)
+* [贝叶斯优化](bayesian-optimization.ipynb)
diff --git a/docs/js/jupyter.js b/docs/js/jupyter.js
new file mode 100644
index 000000000..3e99916c7
--- /dev/null
+++ b/docs/js/jupyter.js
@@ -0,0 +1,8 @@
+// Using the document$ observable from mkdocs-material to get notified of page "reload" also if using `navigation.instant` (SSA)
+document$.subscribe(function () {
+ // First check if the page contains a notebook-related class
+ if (document.querySelector('.jp-Notebook')) {
+ // Remove table of contents
+ document.querySelector('.md-sidebar--secondary').remove();
+ }
+});
diff --git a/mkdocs.yml b/mkdocs.yml
index ccf4b85a1..556cdf826 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -140,6 +140,7 @@ extra_css:
extra_javascript:
- js/loadColor.js
- js/love.js
+ - js/jupyter.js
- js/katex.js
- https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/katex.min.js
- https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.7/contrib/auto-render.min.js
diff --git a/overrides/main.html b/overrides/main.html
index 0340905d2..7c254e141 100644
--- a/overrides/main.html
+++ b/overrides/main.html
@@ -18,12 +18,12 @@
{% set hidden = "hidden" if "toc" in page.meta.hide %}
{% endif %}
- -->
+
{% endif %}
{% endblock %}