From f990236c951a93cc7639099ac53989e54849c8c6 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 10:52:37 +0200 Subject: [PATCH 1/9] rewrite ensemble learning --- .../ensemble_learning_models/ensemble_learning.tex | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index d9432409..9ab587f1 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -1,8 +1,7 @@ \subsubsection{Ensemble Learning} -Ensemble learning is a technique in machine learning where multiple models, known as \textit{weak learners}, are combined to produce more accurate predictions. -Mathematically, ensemble learning can be defined as combining the predictions of $M$ weak learners to form a final prediction $\hat{y}$, such that: -\begin{equation} - \hat{y} = \sum_{m=1}^{M} \alpha_m \hat{y}_m, -\end{equation} -where $\hat{y}_m$ is the prediction of the $m$-th weak learner and $\alpha_m$ is the weight assigned to the $m$-th weak learner. -While there are various choices for weak learners, decision trees are a common choice\cite{James2023AnIS}. \ No newline at end of file +Ensemble learning is a machine learning technique that combines multiple models, referred to as weak learners, to generate more accurate predictions. +Individually, weak learners might have limited performance, but their combination can lead to a more precise and robust model. +Decision trees are often chosen as weak learners due to their simplicity and effectiveness. +Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS}. + +In this section, we provide an overview of the ensemble learning methods used in this work. \ No newline at end of file From c0eb17e80f10fb227ce173117e72f46637ea9776 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 10:55:26 +0200 Subject: [PATCH 2/9] add stacking source --- .../background/ensemble_learning_models/ensemble_learning.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index 9ab587f1..aaa0aff6 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -2,6 +2,6 @@ \subsubsection{Ensemble Learning} Ensemble learning is a machine learning technique that combines multiple models, referred to as weak learners, to generate more accurate predictions. Individually, weak learners might have limited performance, but their combination can lead to a more precise and robust model. Decision trees are often chosen as weak learners due to their simplicity and effectiveness. -Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS}. +Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} In this section, we provide an overview of the ensemble learning methods used in this work. \ No newline at end of file From e5f87625874db620102f1d05fcdd40bc03321b31 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 11:02:07 +0200 Subject: [PATCH 3/9] stacking now in ensemble section --- .../ensemble_learning.tex | 3 ++- .../ensemble_learning_models/index.tex | 3 ++- .../stacked_generalization.tex | 25 +++++++++++++++++++ .../src/sections/background/index.tex | 3 +-- 4 files changed, 30 insertions(+), 4 deletions(-) create mode 100644 report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index aaa0aff6..57913c65 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -4,4 +4,5 @@ \subsubsection{Ensemble Learning} Decision trees are often chosen as weak learners due to their simplicity and effectiveness. Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} -In this section, we provide an overview of the ensemble learning methods used in this work. \ No newline at end of file +In this section, we provide an overview of the ensemble learning methods used in this work. +We begin by introducing decision trees, the most common weak learner used in ensemble methods. diff --git a/report_thesis/src/sections/background/ensemble_learning_models/index.tex b/report_thesis/src/sections/background/ensemble_learning_models/index.tex index 9dbfa5ee..310b4135 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/index.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/index.tex @@ -7,4 +7,5 @@ \subsection{Ensemble Learning Models} \input{sections/background/ensemble_learning_models/etr.tex} \input{sections/background/ensemble_learning_models/gbr.tex} \input{sections/background/ensemble_learning_models/ngboost.tex} -\input{sections/background/ensemble_learning_models/xgboost.tex} \ No newline at end of file +\input{sections/background/ensemble_learning_models/xgboost.tex} +\input{sections/background/ensemble_learning_models/stacked_generalization.tex} \ No newline at end of file diff --git a/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex b/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex new file mode 100644 index 00000000..87129b35 --- /dev/null +++ b/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex @@ -0,0 +1,25 @@ +\subsubsection{Stacked Generalization} +Stacked generalization, introduced by \citet{wolpertstacked_1992}, is an ensemble method specifically designed to improve predictive performance by combining multiple models in a hierarchical manner, where a meta-learner integrates the outputs of base models to make final predictions. + +In this technique, multiple base models are trained on the original dataset. +The outputs of these base models serve as inputs to a meta-model, which is trained to make the final prediction. +This strategy enables the meta-model to learn the optimal way to combine the outputs of the base models to minimize the generalization error. + +Formally, let $\mathbf{X}$ denote the input data and $\mathbf{y}$ the target variable. +Initially, $N$ base models $G_1, G_2, \ldots, G_N$ are trained on the dataset $\mathbf{X}$. +Each base model generates a set of predictions $\hat{\mathbf{y}}_i = G_i(\mathbf{X})$. + +The predictions from the base models are then compiled into a new dataset $\mathbf{Z}$, where each column $\mathbf{z}_i \in \mathbf{Z}$ represents the predictions from the $i$-th base model: + +$$ +\mathbf{Z} = [\hat{\mathbf{y}}_1, \hat{\mathbf{y}}_2, \ldots, \hat{\mathbf{y}}_N] +$$ + +A meta-model $F$ is subsequently trained on this new dataset $\mathbf{Z}$ to predict the target variable $\mathbf{y}$: + +$$ +\mathbf{\hat{y}} = F(\mathbf{Z}) +$$ + +The final prediction is provided by the meta-model $F$, which effectively integrates the outputs of the base models to enhance overall performance. +The effectiveness of stacked generalization stems from its ability to leverage the unique strengths of different base models while mitigating their individual weaknesses, thereby producing a more accurate and generalizable ensemble model. \ No newline at end of file diff --git a/report_thesis/src/sections/background/index.tex b/report_thesis/src/sections/background/index.tex index 80883dce..b85edbe6 100644 --- a/report_thesis/src/sections/background/index.tex +++ b/report_thesis/src/sections/background/index.tex @@ -5,5 +5,4 @@ \section{Background}\label{sec:background} \input{sections/background/preprocessing/index.tex} \input{sections/background/ensemble_learning_models/index.tex} -\input{sections/background/linear_and_regularization_models/index.tex} -\input{sections/background/stacked_generalization.tex} \ No newline at end of file +\input{sections/background/linear_and_regularization_models/index.tex} \ No newline at end of file From fae2cc17daba0ab972929350efa82c4643750a34 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 11:02:13 +0200 Subject: [PATCH 4/9] whoos --- .../background/stacked_generalization.tex | 25 ------------------- 1 file changed, 25 deletions(-) delete mode 100644 report_thesis/src/sections/background/stacked_generalization.tex diff --git a/report_thesis/src/sections/background/stacked_generalization.tex b/report_thesis/src/sections/background/stacked_generalization.tex deleted file mode 100644 index f17cc2a4..00000000 --- a/report_thesis/src/sections/background/stacked_generalization.tex +++ /dev/null @@ -1,25 +0,0 @@ -\subsection{Stacked Generalization} -Stacked generalization, introduced by \citet{wolpertstacked_1992}, is a method designed to improve the predictive performance of machine learning models by leveraging the strengths of multiple models. - -In this technique, multiple base models are trained on the original dataset. -The outputs of these base models serve as inputs to a meta-model, which is trained to make the final prediction. -This strategy enables the meta-model to learn the optimal way to combine the outputs of the base models to minimize the generalization error. - -Formally, let $\mathbf{X}$ denote the input data and $\mathbf{y}$ the target variable. -Initially, $N$ base models $G_1, G_2, \ldots, G_N$ are trained on the dataset $\mathbf{X}$. -Each base model generates a set of predictions $\hat{\mathbf{y}}_i = G_i(\mathbf{X})$. - -The predictions from the base models are then compiled into a new dataset $\mathbf{Z}$, where each column $\mathbf{z}_i \in \mathbf{Z}$ represents the predictions from the $i$-th base model: - -$$ -\mathbf{Z} = [\hat{\mathbf{y}}_1, \hat{\mathbf{y}}_2, \ldots, \hat{\mathbf{y}}_N] -$$ - -A meta-model $F$ is subsequently trained on this new dataset $\mathbf{Z}$ to predict the target variable $\mathbf{y}$: - -$$ -\mathbf{\hat{y}} = F(\mathbf{Z}) -$$ - -The final prediction is provided by the meta-model $F$, which effectively integrates the outputs of the base models to enhance overall performance. -The effectiveness of stacked generalization stems from its ability to leverage the unique strengths of different base models while mitigating their individual weaknesses, thereby producing a more accurate and generalizable ensemble model. \ No newline at end of file From 31cfb273c2c64eec92d6ae2d0d4af28efbe3888a Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 11:11:05 +0200 Subject: [PATCH 5/9] rewrite stacking intro --- .../ensemble_learning_models/stacked_generalization.tex | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex b/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex index 87129b35..2dd74108 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/stacked_generalization.tex @@ -1,9 +1,7 @@ \subsubsection{Stacked Generalization} -Stacked generalization, introduced by \citet{wolpertstacked_1992}, is an ensemble method specifically designed to improve predictive performance by combining multiple models in a hierarchical manner, where a meta-learner integrates the outputs of base models to make final predictions. - -In this technique, multiple base models are trained on the original dataset. -The outputs of these base models serve as inputs to a meta-model, which is trained to make the final prediction. -This strategy enables the meta-model to learn the optimal way to combine the outputs of the base models to minimize the generalization error. +Stacked generalization, introduced by \citet{wolpertstacked_1992}, is an ensemble method that combines the predictions of multiple base models, which are trained on the original dataset, as input to a meta-model. +This meta-model is trained to make the final prediction. +The strategy allows the meta-model to learn the optimal way to combine the predictions of the base models to minimize the generalization error. Formally, let $\mathbf{X}$ denote the input data and $\mathbf{y}$ the target variable. Initially, $N$ base models $G_1, G_2, \ldots, G_N$ are trained on the dataset $\mathbf{X}$. From 7b599b99f20c3ead7075d978477e96f0c4627871 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 14:51:36 +0200 Subject: [PATCH 6/9] remove double mention --- .../background/ensemble_learning_models/ensemble_learning.tex | 1 - 1 file changed, 1 deletion(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index 57913c65..8c26679e 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -1,7 +1,6 @@ \subsubsection{Ensemble Learning} Ensemble learning is a machine learning technique that combines multiple models, referred to as weak learners, to generate more accurate predictions. Individually, weak learners might have limited performance, but their combination can lead to a more precise and robust model. -Decision trees are often chosen as weak learners due to their simplicity and effectiveness. Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} In this section, we provide an overview of the ensemble learning methods used in this work. From 21ca9c52ec37032ca0c45189474983ae4398bcf7 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 14:52:59 +0200 Subject: [PATCH 7/9] commonly used --- .../background/ensemble_learning_models/ensemble_learning.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index 8c26679e..ade9306f 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -4,4 +4,4 @@ \subsubsection{Ensemble Learning} Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} In this section, we provide an overview of the ensemble learning methods used in this work. -We begin by introducing decision trees, the most common weak learner used in ensemble methods. +We begin by introducing decision trees, a commonly used weak learner in ensemble methods. From 26ec60b98b118e1a8588016f2ebafee8a28063a2 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Tue, 4 Jun 2024 14:54:20 +0200 Subject: [PATCH 8/9] Update report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex Co-authored-by: Pattrigue <57709490+Pattrigue@users.noreply.github.com> --- .../background/ensemble_learning_models/ensemble_learning.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index ade9306f..c39c3c7a 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -1,6 +1,6 @@ \subsubsection{Ensemble Learning} Ensemble learning is a machine learning technique that combines multiple models, referred to as weak learners, to generate more accurate predictions. -Individually, weak learners might have limited performance, but their combination can lead to a more precise and robust model. +Individually, the predictions of weak learners might have limited performance, but their combination can lead to a more precise and robust model. Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} In this section, we provide an overview of the ensemble learning methods used in this work. From fdd6c44dec10ef951f283377b7c5fa26b82254ef Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Tue, 4 Jun 2024 14:55:25 +0200 Subject: [PATCH 9/9] le fix --- .../background/ensemble_learning_models/ensemble_learning.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex index c39c3c7a..0f2cb311 100644 --- a/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex +++ b/report_thesis/src/sections/background/ensemble_learning_models/ensemble_learning.tex @@ -1,6 +1,6 @@ \subsubsection{Ensemble Learning} Ensemble learning is a machine learning technique that combines multiple models, referred to as weak learners, to generate more accurate predictions. -Individually, the predictions of weak learners might have limited performance, but their combination can lead to a more precise and robust model. +Individually, the predictive ability of weak learners may be limited, but when combined, they can produce a more precise and robust model. Ensemble learning encompasses several methods, including bagging, boosting, and stacking\cite{James2023AnIS, pavlyshenko2018stacking} In this section, we provide an overview of the ensemble learning methods used in this work.