From e5eb50ab1affe5a586970ce2d643dbdc1afead35 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Sat, 8 Jun 2024 08:17:49 +0200 Subject: [PATCH 01/18] Only need hyperparameters --- .../background/preprocessing/norm3.tex | 2 +- .../experiment_design/data_preparation.tex | 71 ++++++++++++++++ .../experiment_design/experimental_setup.tex | 6 ++ .../src/sections/experiment_design/index.tex | 7 ++ .../experiment_design/initial_experiment.tex | 16 ++++ report_thesis/src/sections/methodology.tex | 81 ------------------- 6 files changed, 101 insertions(+), 82 deletions(-) create mode 100644 report_thesis/src/sections/experiment_design/data_preparation.tex create mode 100644 report_thesis/src/sections/experiment_design/experimental_setup.tex create mode 100644 report_thesis/src/sections/experiment_design/index.tex create mode 100644 report_thesis/src/sections/experiment_design/initial_experiment.tex diff --git a/report_thesis/src/sections/background/preprocessing/norm3.tex b/report_thesis/src/sections/background/preprocessing/norm3.tex index ea5406c7..a8eec08d 100644 --- a/report_thesis/src/sections/background/preprocessing/norm3.tex +++ b/report_thesis/src/sections/background/preprocessing/norm3.tex @@ -1,4 +1,4 @@ -\subsubsection{Norm 3} +\subsubsection{Norm 3}\label{sec:norm3} As previously mentioned, the \gls{chemcam} instrument consists of three spectrometers, each producing 2048 channels. For data normalization, we follow the approach taken by the ChemCam team and normalize across individual spectrometers' wavelength ranges, a process known as \textit{Norm 3}~\cite{cleggRecalibrationMarsScience2017, andersonImprovedAccuracyQuantitative2017}. This method ensures that the wavelength intensities captured by each spectrometer are normalized independently, thus preserving the relative intensities within each spectrometer. diff --git a/report_thesis/src/sections/experiment_design/data_preparation.tex b/report_thesis/src/sections/experiment_design/data_preparation.tex new file mode 100644 index 00000000..9cfe3568 --- /dev/null +++ b/report_thesis/src/sections/experiment_design/data_preparation.tex @@ -0,0 +1,71 @@ +\subsection{Data Preparation}\label{sec:data-preparation} +The first step in our methodology is to prepare the datasets for model training and evaluation. +As mentioned in Section~\ref{sec:data-overview}, the data used in this study was obtained from NASA's \gls{pds} and consists of \gls{ccs} data and major oxide compositions for various samples. + +The initial five shots from each sample are excluded because they are usually contaminated by dust covering the sample, which is cleared away by the shock waves produced by the laser \cite{cleggRecalibrationMarsScience2017}. +The remaining 45 shots from each location are then averaged, yielding a single spectrum $s$ per location $l$ in the Averaged Intensity Tensor\ref{matrix:averaged_intensity}, resulting in a total of five spectra for each sample. + +At this stage, the data still contains noise at the edges of the spectrometers. +These edges correspond to the boundaries of the three spectrometers, which collectively cover the \gls{uv}, \gls{vio}, and \gls{vnir} light spectra. +The noisy edge ranges are as follows: 240.811-246.635 nm, 338.457-340.797 nm, 382.138-387.859 nm, 473.184-492.427 nm, and 849-905.574 nm. +In addition to being noisy regions, these regions do not contain any useful information related to each of the major oxides. +Consequently, these regions are masked by zeroing out the values, rather than removing them, as they represent meaningful variation in the data~\cite{cleggRecalibrationMarsScience2017}. + +Additionally, as a result of the aforementioned preprocessing applied to the raw \gls{libs} data, negative values are present in the \gls{ccs} data. +These negative values are not physically meaningful, since you cannot have negative light intensity \cite{p9_paper}. +Similar to the noisy edges, these negative values are also masked by zeroing out the values. + +We transpose the data so that each row represents a location and each column represents a wavelength feature. +Each location is now represented as a vector of wavelengths, with the corresponding average intensity values for each wavelength. +These vectors are then concatenated to form a tensor, giving us the full Averaged Intensity Tensor. + +For each sample, we have a corresponding set of major oxide compositions in weight percentage (wt\%). +These compositions are used as the target labels for the machine learning models. +An excerpt of this data is shown in Table \ref{tab:composition_data_example}. +While the \textit{Target}, \textit{Spectrum Name}, and \textit{Sample Names} are part of the dataset, our analysis focuses primarily on the \textit{Sample Names}. +The concentrations of the eight oxides \ce{SiO2}, \ce{TiO2}, \ce{Al2O3}, \ce{FeO_T}, \ce{MnO}, \ce{MgO}, \ce{CaO}, \ce{Na2O}, and \ce{K2O} represent the expected values for these oxides in the sample, serving as our ground truth. The \textit{MOC total} is not utilized in this study. + +\begin{table*}[h] +\centering +\begin{tabular}{lllllllllllll} +\toprule + Target & Spectrum Name & Sample Name & \ce{SiO2} & \ce{TiO2} & \ce{Al2O3} & \ce{FeO_T} & \ce{MnO} & \ce{MgO} & \ce{CaO} & \ce{Na2O} & \ce{K2O} & \ce{MOC total} \\ +\midrule +AGV2 & AGV2 & AGV2 & 59.3 & 1.05 & 16.91 & 6.02 & 0.099 & 1.79 & 5.2 & 4.19 & 2.88 & 97.44 \\ +BCR-2 & BCR2 & BCR2 & 54.1 & 2.26 & 13.5 & 12.42 & 0.2 & 3.59 & 7.12 & 3.16 & 1.79 & 98.14 \\ +$\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ \\ +TB & --- & --- & 60.23 & 0.93 & 20.64 & 11.6387 & 0.052 & 1.93 & 0.000031 & 1.32 & 3.87 & 100.610731 \\ + TB2 & --- & --- & 60.4 & 0.93 & 20.5 & 11.6536 & 0.047 & 1.86 & 0.2 & 1.29 & 3.86 & 100.7406 \\ +\bottomrule +\end{tabular} +\caption{Excerpt from the composition dataset (from \citet{p9_paper})} +\label{tab:composition_data_example} +\end{table*} + +The major oxide weight percentages are appended to the matrix of spectral data, forming the final dataset. +This dataset is shown in Table~\ref{tab:final_dataset_example}. +The \textit{Target} column corresponds to the sample name, while the \textit{ID} column contains the unique identifier for each location. + +\begin{table*}[h] +\centering +\footnotesize +\begin{tabular}{llllllllllllllllllllll} +\toprule + 240.81 & $\cdots$ & 425.82 & 425.87 & $\cdots$ & 905.57 & \ce{SiO2} & \ce{TiO2} & \ce{Al2O3} & \ce{FeO_T} & \ce{MgO} & \ce{CaO} & \ce{Na2O} & \ce{K2O} & Target & ID \\ +\midrule + 0 & $\cdots$ & 1.53e+10 & 1.62e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_211002\_ccs \\ + 0 & $\cdots$ & 1.28e+10 & 1.30e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_211143\_ccs \\ + 0 & $\cdots$ & 1.87e+10 & 1.83e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210628\_ccs \\ + 0 & $\cdots$ & 1.77e+10 & 1.78e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210415\_ccs \\ + 0 & $\cdots$ & 1.75e+10 & 1.79e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210811\_ccs \\ + 0 & $\cdots$ & 5.52e+10 & 3.74e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161903\_ccs \\ + 0 & $\cdots$ & 5.09e+10 & 3.41e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_162038\_ccs \\ + 0 & $\cdots$ & 5.99e+10 & 3.97e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161422\_ccs \\ + 0 & $\cdots$ & 5.22e+10 & 3.47e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161735\_ccs \\ + 0 & $\cdots$ & 5.29e+10 & 3.62e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161552\_ccs \\ + $\vdots$ & $\cdots$ & $\vdots$ & $\vdots$ & $\cdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ \\ +\midrule +\end{tabular} +\caption{Excerpt from the final dataset (values have been rounded to two decimal places for brevity).} +\label{tab:final_dataset_example} +\end{table*} \ No newline at end of file diff --git a/report_thesis/src/sections/experiment_design/experimental_setup.tex b/report_thesis/src/sections/experiment_design/experimental_setup.tex new file mode 100644 index 00000000..12c30f60 --- /dev/null +++ b/report_thesis/src/sections/experiment_design/experimental_setup.tex @@ -0,0 +1,6 @@ +\subsection{Experimental Setup} +Experiments were conducted on a machine equipped with an Intel Xeon Gold 6242 CPU, featuring 16 cores and 32 threads. +The CPU has a base clock speed of 2.80 GHz and a maximum turbo frequency of 3.90 GHz. +The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. +Models were implemented using Python 3.10.11. +The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. \ No newline at end of file diff --git a/report_thesis/src/sections/experiment_design/index.tex b/report_thesis/src/sections/experiment_design/index.tex new file mode 100644 index 00000000..3520f166 --- /dev/null +++ b/report_thesis/src/sections/experiment_design/index.tex @@ -0,0 +1,7 @@ +\section{Experimental Design}\label{sec:methodology} +In this section, we outline the methodology used in this study to address the challenges identified in Section~\ref{sec:problem_definition}. Our objective was to identify the most promising machine learning models and preprocessing techniques proposed from the literature, as outlined in Section~\ref{sec:related-work}, for predicting major oxide compositions from \gls{libs} data. +Then, using this knowledge, develop a pipeline that utilizes the strengths of these models and preprocessing techniques to improve prediction accuracy and robustness of the predictions. +We first describe the datasets used, including their preparation and the method of splitting for model training. Next, we outline the preprocessing steps and the model selection process, followed by a detailed explanation of the experimental setup and evaluation metrics. Finally, we discuss our validation testing procedures and the approach taken to ensure unbiased final model evaluations. + +\include{sections/experiment_design/data_preparation} +\include{sections/experiment_design/experimental_setup} \ No newline at end of file diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex new file mode 100644 index 00000000..82576884 --- /dev/null +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -0,0 +1,16 @@ +\subsection{Design for Initial Experiment}\label{sec:initial-experiment} +The goal of the initial experiment is to identify the most promising models for further evaluation. +This experiment was designed to give an initial indication of the performance of the models on the \gls{ccs} dataset. + +To conduct the initial experiment and evaluate the twelve selected models shown in Table~\ref{tab:preprocessing-models} we wanted to ensure that the models were evaluated on a consistent basis. +We achieve this by setting the following conditions for the experiment. +The only preprocessor used for all models was \texttt{Norm 3} normalization. +This normalization technique was developed specifically by \citet{cleggRecalibrationMarsScience2017} for accomodating the characteristics of the \gls{libs} data, whereby the data represents the readings of three different spectrometers, as outlined in Section~\ref{sec:norm3}. +Therefore, \texttt{Norm 3} represents the most \gls{libs}-specific normalization technique available in from the pool of selected preprocessors and is the most appropriate choice for this experiment. +All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. +To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. +However, due to the nature of the neural network models architecture, some extra time was spent on tuning the models to ensure a fair comparison. +Finally, all the models ran only one trial. +Although running multiple trials would allow for an average performance evaluation, we argue that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. +Additionally, as stated, the goal of this experiment is merely to get an initial indication of the performance of the models. +Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. \ No newline at end of file diff --git a/report_thesis/src/sections/methodology.tex b/report_thesis/src/sections/methodology.tex index 8eb5d63d..b28b04f6 100644 --- a/report_thesis/src/sections/methodology.tex +++ b/report_thesis/src/sections/methodology.tex @@ -1,84 +1,3 @@ -\section{Experimental Design}\label{sec:methodology} -In this section, we outline the methodology used in this study to address the challenges identified in Section~\ref{sec:problem_definition}. Our objective was to identify the most promising machine learning models and preprocessing techniques proposed from the literature, as outlined in Section~\ref{sec:related-work}, for predicting major oxide compositions from \gls{libs} data. -Then, using this knowledge, develop a pipeline that utilizes the strengths of these models and preprocessing techniques to improve prediction accuracy and robustness of the predictions. -We first describe the datasets used, including their preparation and the method of splitting for model training. Next, we outline the preprocessing steps and the model selection process, followed by a detailed explanation of the experimental setup and evaluation metrics. Finally, we discuss our validation testing procedures and the approach taken to ensure unbiased final model evaluations. -\subsection{Data Preparation}\label{sec:data-preparation} -The first step in our methodology is to prepare the datasets for model training and evaluation. -As mentioned in Section~\ref{sec:data-overview}, the data used in this study was obtained from NASA's \gls{pds} and consists of \gls{ccs} data and major oxide compositions for various samples. -The initial five shots from each sample are excluded because they are usually contaminated by dust covering the sample, which is cleared away by the shock waves produced by the laser \cite{cleggRecalibrationMarsScience2017}. -The remaining 45 shots from each location are then averaged, yielding a single spectrum $s$ per location $l$ in the Averaged Intensity Tensor\ref{matrix:averaged_intensity}, resulting in a total of five spectra for each sample. - -At this stage, the data still contains noise at the edges of the spectrometers. -These edges correspond to the boundaries of the three spectrometers, which collectively cover the \gls{uv}, \gls{vio}, and \gls{vnir} light spectra. -The noisy edge ranges are as follows: 240.811-246.635 nm, 338.457-340.797 nm, 382.138-387.859 nm, 473.184-492.427 nm, and 849-905.574 nm. -In addition to being noisy regions, these regions do not contain any useful information related to each of the major oxides. -Consequently, these regions are masked by zeroing out the values, rather than removing them, as they represent meaningful variation in the data~\cite{cleggRecalibrationMarsScience2017}. - -Additionally, as a result of the aforementioned preprocessing applied to the raw \gls{libs} data, negative values are present in the \gls{ccs} data. -These negative values are not physically meaningful, since you cannot have negative light intensity \cite{p9_paper}. -Similar to the noisy edges, these negative values are also masked by zeroing out the values. - -We transpose the data so that each row represents a location and each column represents a wavelength feature. -Each location is now represented as a vector of wavelengths, with the corresponding average intensity values for each wavelength. -These vectors are then concatenated to form a tensor, giving us the full Averaged Intensity Tensor. - -For each sample, we have a corresponding set of major oxide compositions in weight percentage (wt\%). -These compositions are used as the target labels for the machine learning models. -An excerpt of this data is shown in Table \ref{tab:composition_data_example}. -While the \textit{Target}, \textit{Spectrum Name}, and \textit{Sample Names} are part of the dataset, our analysis focuses primarily on the \textit{Sample Names}. -The concentrations of the eight oxides \ce{SiO2}, \ce{TiO2}, \ce{Al2O3}, \ce{FeO_T}, \ce{MnO}, \ce{MgO}, \ce{CaO}, \ce{Na2O}, and \ce{K2O} represent the expected values for these oxides in the sample, serving as our ground truth. The \textit{MOC total} is not utilized in this study. - -\begin{table*}[h] -\centering -\begin{tabular}{lllllllllllll} -\toprule - Target & Spectrum Name & Sample Name & \ce{SiO2} & \ce{TiO2} & \ce{Al2O3} & \ce{FeO_T} & \ce{MnO} & \ce{MgO} & \ce{CaO} & \ce{Na2O} & \ce{K2O} & \ce{MOC total} \\ -\midrule -AGV2 & AGV2 & AGV2 & 59.3 & 1.05 & 16.91 & 6.02 & 0.099 & 1.79 & 5.2 & 4.19 & 2.88 & 97.44 \\ -BCR-2 & BCR2 & BCR2 & 54.1 & 2.26 & 13.5 & 12.42 & 0.2 & 3.59 & 7.12 & 3.16 & 1.79 & 98.14 \\ -$\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ \\ -TB & --- & --- & 60.23 & 0.93 & 20.64 & 11.6387 & 0.052 & 1.93 & 0.000031 & 1.32 & 3.87 & 100.610731 \\ - TB2 & --- & --- & 60.4 & 0.93 & 20.5 & 11.6536 & 0.047 & 1.86 & 0.2 & 1.29 & 3.86 & 100.7406 \\ -\bottomrule -\end{tabular} -\caption{Excerpt from the composition dataset (from \citet{p9_paper})} -\label{tab:composition_data_example} -\end{table*} - -The major oxide weight percentages are appended to the matrix of spectral data, forming the final dataset. -This dataset is shown in Table~\ref{tab:final_dataset_example}. -The \textit{Target} column corresponds to the sample name, while the \textit{ID} column contains the unique identifier for each location. - -\begin{table*}[h] -\centering -\footnotesize -\begin{tabular}{llllllllllllllllllllll} -\toprule - 240.81 & $\cdots$ & 425.82 & 425.87 & $\cdots$ & 905.57 & \ce{SiO2} & \ce{TiO2} & \ce{Al2O3} & \ce{FeO_T} & \ce{MgO} & \ce{CaO} & \ce{Na2O} & \ce{K2O} & Target & ID \\ -\midrule - 0 & $\cdots$ & 1.53e+10 & 1.62e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_211002\_ccs \\ - 0 & $\cdots$ & 1.28e+10 & 1.30e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_211143\_ccs \\ - 0 & $\cdots$ & 1.87e+10 & 1.83e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210628\_ccs \\ - 0 & $\cdots$ & 1.77e+10 & 1.78e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210415\_ccs \\ - 0 & $\cdots$ & 1.75e+10 & 1.79e+10 & $\cdots$ & 0 & 56.13 & 0.69 & 17.69 & 5.86 & 3.85 & 7.07 & 3.32 & 1.44 & jsc1421 & jsc1421\_2013\_09\_12\_210811\_ccs \\ - 0 & $\cdots$ & 5.52e+10 & 3.74e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161903\_ccs \\ - 0 & $\cdots$ & 5.09e+10 & 3.41e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_162038\_ccs \\ - 0 & $\cdots$ & 5.99e+10 & 3.97e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161422\_ccs \\ - 0 & $\cdots$ & 5.22e+10 & 3.47e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161735\_ccs \\ - 0 & $\cdots$ & 5.29e+10 & 3.62e+10 & $\cdots$ & 0 & 57.60 & 0.78 & 26.60 & 2.73 & 0.70 & 0.01 & 0.38 & 7.10 & pg7 & pg7\_2013\_11\_07\_161552\_ccs \\ - $\vdots$ & $\cdots$ & $\vdots$ & $\vdots$ & $\cdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ & $\vdots$ \\ -\midrule -\end{tabular} -\caption{Excerpt from the final dataset (values have been rounded to two decimal places for brevity).} -\label{tab:final_dataset_example} -\end{table*} - -\subsection{Experimental Setup} -Experiments were conducted on a machine equipped with an Intel Xeon Gold 6242 CPU, featuring 16 cores and 32 threads. -The CPU has a base clock speed of 2.80 GHz and a maximum turbo frequency of 3.90 GHz. -The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. -Models were implemented using Python 3.10.11. -The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. From 9136a31d6e12091869295fe955ff0d3ed40efcb6 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Sat, 8 Jun 2024 20:31:58 +0200 Subject: [PATCH 02/18] ready for review --- report_thesis/src/_preamble.tex | 1 + report_thesis/src/sections/appendix.tex | 249 +++++++++++++++++- .../experiment_design/experimental_setup.tex | 3 +- .../experiment_design/initial_experiment.tex | 6 +- 4 files changed, 255 insertions(+), 4 deletions(-) diff --git a/report_thesis/src/_preamble.tex b/report_thesis/src/_preamble.tex index 7f681129..f26d9e1e 100644 --- a/report_thesis/src/_preamble.tex +++ b/report_thesis/src/_preamble.tex @@ -17,6 +17,7 @@ \usepackage{algorithm} \usepackage{algpseudocode} \usepackage{enumitem} +\usepackage{placeins} \setmainfont{texgyrepagella}[ Extension = .otf, UprightFont = *-regular, diff --git a/report_thesis/src/sections/appendix.tex b/report_thesis/src/sections/appendix.tex index d70df27a..6f9c3bc3 100644 --- a/report_thesis/src/sections/appendix.tex +++ b/report_thesis/src/sections/appendix.tex @@ -43,4 +43,251 @@ \subsection{Cross-Validation Plots for Major Oxides}\label{subsec:cv_plots} \caption{Distribution of \ce{\oxide} concentrations across cross-validation folds, training set, test set, and the entire dataset. The mean and standard deviation statistics for each partition are indicated figure.} \label{fig:distribution_plot_\oxide} \end{figure*} -} \ No newline at end of file +} + +\subsection{Initial Experiment: Model Hyperparameters}\label{subsec:initial_experiment_hyperparameters} +This section provides a detailed overview of the hyperparameters used for the models in the initial experiment. + +\begin{table}[H] +\centering +\caption{Hyperparameters for Partial Least Squares} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Number of Components & 34 \\ + Scale & True \\ + Max Iter & 500 \\ + \bottomrule +\end{tabular} +\label{tab:pls_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Support Vector Regression} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Kernel & poly \\ + C & 100 \\ + Epsilon & 0.1 \\ + Gamma & scale \\ + Degree & 2 \\ + Coef0 & 1.0 \\ + \bottomrule +\end{tabular} +\label{tab:svr_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Elastic Nets} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ + L1 Ratio & [0.1, 0.5, 0.7, 0.9, 1.0] \\ + Max Iter & 1000 \\ + Tolerance & 1e-4 \\ + \bottomrule +\end{tabular} +\label{tab:elasticnet_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Lasso} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ + Max Iter & 1000 \\ + Tolerance & 1e-4 \\ + \bottomrule +\end{tabular} +\label{tab:lasso_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Ridge Regression} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ + Max Iter & 1000 \\ + Tolerance & 1e-4 \\ + \bottomrule +\end{tabular} +\label{tab:ridge_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Random Forest} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Number of Estimators & 100 \\ + Max Depth & 10 \\ + Min Samples Split & 2 \\ + Min Samples Leaf & 1 \\ + Max Features & sqrt \\ + Random State & 42 \\ + \bottomrule +\end{tabular} +\label{tab:randomforest_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Gradient Boosting Regression} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Number of Estimators & 100 \\ + Max Depth & 3 \\ + Min Samples Split & 2 \\ + Min Samples Leaf & 1 \\ + Max Features & None \\ + Loss & squared\_error \\ + Learning Rate & 0.1 \\ + Subsample & 1.0 \\ + Criterion & friedman\_mse \\ + Random State & 42 \\ + Verbose & 0 \\ + Validation Fraction & 0.1 \\ + Number of Iterations No Change & None \\ + Tolerance & 1e-4 \\ + CCP Alpha & 0.0 \\ + \bottomrule +\end{tabular} +\label{tab:gbr_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Extra Trees Regression} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Number of Estimators & 100 \\ + Max Depth & 10 \\ + Min Samples Split & 2 \\ + Min Samples Leaf & 1 \\ + Random State & 42 \\ + \bottomrule +\end{tabular} +\label{tab:extratrees_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for XGBoost} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Max Depth & 4 \\ + Min Child Weight & 5 \\ + Gamma & 0.1 \\ + Subsample & 0.7 \\ + Colsample Bytree & 0.5 \\ + Colsample Bylevel & 0.5 \\ + Colsample Bynode & 0.5 \\ + Lambda & 1 \\ + Alpha & 0.5 \\ + Learning Rate & 0.05 \\ + Number of Estimators & 100 \\ + Objective & reg:squarederror \\ + Eval Metric & rmse \\ + \bottomrule +\end{tabular} +\label{tab:xgboost_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for NGBoost} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + n/a & n/a \\ + \bottomrule +\end{tabular} +\label{tab:ngboost_hyperparameters} +\end{table} +\FloatBarrier + +\begin{table}[H] +\centering +\caption{Hyperparameters for Artificial Neural Networks} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Layer 1: Dense & (1024 units) \\ + Layer 2: Dropout & (rate: 0.5) \\ + Layer 3: Dense & (512 units) \\ + Layer 4: Dropout & (rate: 0.5) \\ + Layer 5: Dense & (256 units) \\ + Layer 6: Dense & (128 units) \\ + Layer 7: Dense & (1 unit) \\ + \bottomrule +\end{tabular} +\label{tab:ann_hyperparameters} +\end{table} +\FloatBarrier + +\clearpage +\begin{table}[H] +\centering +\caption{Hyperparameters for Convolutional Neural Networks} +\begin{tabular}{ll} + \toprule + \textbf{Hyperparameter} & \textbf{Value} \\ + \midrule + Layer 1: InputLayer & (6144, 1) \\ + Layer 2: BatchNormalization & (axis: -1) \\ + Layer 3: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ + Layer 4: MaxPooling1D & (pool size: 2) \\ + Layer 5: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ + Layer 6: MaxPooling1D & (pool size: 2) \\ + Layer 7: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ + Layer 8: Add & (merge with layer 6 output) \\ + Layer 9: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ + Layer 10: MaxPooling1D & (pool size: 2) \\ + Layer 11: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ + Layer 12: MaxPooling1D & (pool size: 2) \\ + Layer 13: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ + Layer 14: Add & (merge with layer 12 output) \\ + Layer 15: Flatten & (output) \\ + Layer 16: Dropout & (rate: 0.5) \\ + Layer 17: Dense & (512 units, activation: relu) \\ + Layer 18: Dense & (1 unit, activation: linear) \\ + Total Parameters & 25,274,245 \\ + Trainable Parameters & 25,274,243 \\ + Non-trainable Parameters & 2 \\ + \bottomrule +\end{tabular} +\label{tab:cnn_hyperparameters} +\end{table} +\FloatBarrier \ No newline at end of file diff --git a/report_thesis/src/sections/experiment_design/experimental_setup.tex b/report_thesis/src/sections/experiment_design/experimental_setup.tex index 12c30f60..49166338 100644 --- a/report_thesis/src/sections/experiment_design/experimental_setup.tex +++ b/report_thesis/src/sections/experiment_design/experimental_setup.tex @@ -3,4 +3,5 @@ \subsection{Experimental Setup} The CPU has a base clock speed of 2.80 GHz and a maximum turbo frequency of 3.90 GHz. The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. Models were implemented using Python 3.10.11. -The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. \ No newline at end of file +The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. +Additionally, all experiments were run using the hyperparameter tuning tool described in Section~\ref{subsec:hyperparameter_tuning_tool}. % TODO: Add correct ref once other PR is in \ No newline at end of file diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 82576884..6d1baad4 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -10,7 +10,9 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. However, due to the nature of the neural network models architecture, some extra time was spent on tuning the models to ensure a fair comparison. -Finally, all the models ran only one trial. +Finally, all the models ran only one trial per oxide. Although running multiple trials would allow for an average performance evaluation, we argue that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. Additionally, as stated, the goal of this experiment is merely to get an initial indication of the performance of the models. -Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. \ No newline at end of file +Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. + +The hyperparameters used for the models in the initial experiment can be found in the Appendix~\ref{subsec:initial_experiment_hyperparameters}. \ No newline at end of file From 0d909c322d4bdcc41b743946e7257687c745894b Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Sat, 8 Jun 2024 20:34:29 +0200 Subject: [PATCH 03/18] changed word --- .../src/sections/experiment_design/experimental_setup.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/experiment_design/experimental_setup.tex b/report_thesis/src/sections/experiment_design/experimental_setup.tex index 49166338..764343c0 100644 --- a/report_thesis/src/sections/experiment_design/experimental_setup.tex +++ b/report_thesis/src/sections/experiment_design/experimental_setup.tex @@ -4,4 +4,4 @@ \subsection{Experimental Setup} The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. Models were implemented using Python 3.10.11. The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. -Additionally, all experiments were run using the hyperparameter tuning tool described in Section~\ref{subsec:hyperparameter_tuning_tool}. % TODO: Add correct ref once other PR is in \ No newline at end of file +Additionally, all experiments were run using the hyperparameter optimization tool described in Section~\ref{subsec:hyperparameter_tuning_tool}. % TODO: Add correct ref once other PR is in \ No newline at end of file From 148219465a7c27ae1c3ea4156cdedae70d8725d5 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 8 Jun 2024 22:31:27 +0200 Subject: [PATCH 04/18] include the experiment design sections in the report --- report_thesis/src/index.tex | 1 + report_thesis/src/sections/experiment_design/index.tex | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/report_thesis/src/index.tex b/report_thesis/src/index.tex index 54f96d5f..9770e1fb 100644 --- a/report_thesis/src/index.tex +++ b/report_thesis/src/index.tex @@ -17,6 +17,7 @@ \subsubsection*{Acknowledgements:} \input{sections/proposed_approach/proposed_approach.tex} \input{sections/baseline_replica.tex} \input{sections/methodology.tex} +\input{sections/experiment_design/index.tex} \input{sections/results/results.tex} % \subsection{Data Analysis} diff --git a/report_thesis/src/sections/experiment_design/index.tex b/report_thesis/src/sections/experiment_design/index.tex index 3520f166..118dccbe 100644 --- a/report_thesis/src/sections/experiment_design/index.tex +++ b/report_thesis/src/sections/experiment_design/index.tex @@ -3,5 +3,6 @@ \section{Experimental Design}\label{sec:methodology} Then, using this knowledge, develop a pipeline that utilizes the strengths of these models and preprocessing techniques to improve prediction accuracy and robustness of the predictions. We first describe the datasets used, including their preparation and the method of splitting for model training. Next, we outline the preprocessing steps and the model selection process, followed by a detailed explanation of the experimental setup and evaluation metrics. Finally, we discuss our validation testing procedures and the approach taken to ensure unbiased final model evaluations. -\include{sections/experiment_design/data_preparation} -\include{sections/experiment_design/experimental_setup} \ No newline at end of file +\input{sections/experiment_design/data_preparation} +\input{sections/experiment_design/experimental_setup} +\input{sections/experiment_design/initial_experiment} \ No newline at end of file From 21fe26dc1e2d8b01a2e34c2c4b8e208c1d521c0f Mon Sep 17 00:00:00 2001 From: Ivikhostrup <56341364+Ivikhostrup@users.noreply.github.com> Date: Mon, 10 Jun 2024 19:05:31 +0200 Subject: [PATCH 05/18] Update report_thesis/src/sections/experiment_design/initial_experiment.tex Co-authored-by: Christian Bager Bach Houmann --- .../src/sections/experiment_design/initial_experiment.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 6d1baad4..3407f2e3 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -2,7 +2,7 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} The goal of the initial experiment is to identify the most promising models for further evaluation. This experiment was designed to give an initial indication of the performance of the models on the \gls{ccs} dataset. -To conduct the initial experiment and evaluate the twelve selected models shown in Table~\ref{tab:preprocessing-models} we wanted to ensure that the models were evaluated on a consistent basis. +To conduct the initial experiment and evaluate the twelve selected models shown in Table~\ref{tab:preprocessing-models}, we ensured that the models were evaluated on a consistent basis. We achieve this by setting the following conditions for the experiment. The only preprocessor used for all models was \texttt{Norm 3} normalization. This normalization technique was developed specifically by \citet{cleggRecalibrationMarsScience2017} for accomodating the characteristics of the \gls{libs} data, whereby the data represents the readings of three different spectrometers, as outlined in Section~\ref{sec:norm3}. From e786f928340f6a5a7f0016ff3412d68390376e68 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Mon, 10 Jun 2024 19:40:18 +0200 Subject: [PATCH 06/18] Update --- .../sections/experiment_design/initial_experiment.tex | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 3407f2e3..5ca2ceaf 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -2,11 +2,11 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} The goal of the initial experiment is to identify the most promising models for further evaluation. This experiment was designed to give an initial indication of the performance of the models on the \gls{ccs} dataset. -To conduct the initial experiment and evaluate the twelve selected models shown in Table~\ref{tab:preprocessing-models}, we ensured that the models were evaluated on a consistent basis. -We achieve this by setting the following conditions for the experiment. -The only preprocessor used for all models was \texttt{Norm 3} normalization. -This normalization technique was developed specifically by \citet{cleggRecalibrationMarsScience2017} for accomodating the characteristics of the \gls{libs} data, whereby the data represents the readings of three different spectrometers, as outlined in Section~\ref{sec:norm3}. -Therefore, \texttt{Norm 3} represents the most \gls{libs}-specific normalization technique available in from the pool of selected preprocessors and is the most appropriate choice for this experiment. +As described in Section~\ref{sec:proposed_approach}, we conducted a series of initial experiments to evaluate the performance of various machine learning models on the prediction of major oxide compositions from our \gls{libs} dataset. +These experiments aimed to provide a preliminary assessment of the models' performance, allowing us to identify the most promising models for further evaluation and inclusion in our stacking ensemble. +All models were trained on the same preprocessed data using the Norm 3 preprocessing method described in Section~\ref{sec:norm3}. +This ensured that the models' performance could be evaluated under consistent and comparable conditions + All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. However, due to the nature of the neural network models architecture, some extra time was spent on tuning the models to ensure a fair comparison. From a88113ffff378a26fbb434b0fa5b9883eefa4422 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Mon, 10 Jun 2024 19:41:35 +0200 Subject: [PATCH 07/18] fix --- .../src/sections/experiment_design/initial_experiment.tex | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 5ca2ceaf..6e3e1d6c 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -1,11 +1,8 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} -The goal of the initial experiment is to identify the most promising models for further evaluation. -This experiment was designed to give an initial indication of the performance of the models on the \gls{ccs} dataset. - As described in Section~\ref{sec:proposed_approach}, we conducted a series of initial experiments to evaluate the performance of various machine learning models on the prediction of major oxide compositions from our \gls{libs} dataset. These experiments aimed to provide a preliminary assessment of the models' performance, allowing us to identify the most promising models for further evaluation and inclusion in our stacking ensemble. All models were trained on the same preprocessed data using the Norm 3 preprocessing method described in Section~\ref{sec:norm3}. -This ensured that the models' performance could be evaluated under consistent and comparable conditions +This ensured that the models' performance could be evaluated under consistent and comparable conditions. All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. From 794958cb5088f24cc7eac1474bc69697576da839 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Mon, 10 Jun 2024 19:43:04 +0200 Subject: [PATCH 08/18] comment --- .../src/sections/experiment_design/initial_experiment.tex | 2 ++ 1 file changed, 2 insertions(+) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 6e3e1d6c..3c2ac0d4 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -12,4 +12,6 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} Additionally, as stated, the goal of this experiment is merely to get an initial indication of the performance of the models. Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. + +% Add the fact that we used batchnorm for CNN The hyperparameters used for the models in the initial experiment can be found in the Appendix~\ref{subsec:initial_experiment_hyperparameters}. \ No newline at end of file From a652fd5359db24a990929d20ee09072e7f878ce9 Mon Sep 17 00:00:00 2001 From: Pattrigue <57709490+Pattrigue@users.noreply.github.com> Date: Mon, 10 Jun 2024 20:19:41 +0200 Subject: [PATCH 09/18] Update report_thesis/src/sections/experiment_design/initial_experiment.tex Co-authored-by: Christian Bager Bach Houmann --- .../src/sections/experiment_design/initial_experiment.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 3c2ac0d4..6e8b5d2c 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -8,8 +8,8 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. However, due to the nature of the neural network models architecture, some extra time was spent on tuning the models to ensure a fair comparison. Finally, all the models ran only one trial per oxide. -Although running multiple trials would allow for an average performance evaluation, we argue that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. -Additionally, as stated, the goal of this experiment is merely to get an initial indication of the performance of the models. +Although running multiple trials would have allowed for an average performance evaluation, we argued that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. +Additionally, as stated, the goal of this experiment was merely to get an initial indication of the performance of the models. Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. From 2cbd3b2fdbca7f414480ab8f45b24c63ca691f38 Mon Sep 17 00:00:00 2001 From: Pattrigue <57709490+Pattrigue@users.noreply.github.com> Date: Mon, 10 Jun 2024 20:19:56 +0200 Subject: [PATCH 10/18] Update report_thesis/src/sections/experiment_design/initial_experiment.tex Co-authored-by: Christian Bager Bach Houmann --- .../src/sections/experiment_design/initial_experiment.tex | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 6e8b5d2c..7770e888 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -5,8 +5,8 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} This ensured that the models' performance could be evaluated under consistent and comparable conditions. All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. -To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible and those hyperparameters that did not have default options were selected based on values found in the literature. -However, due to the nature of the neural network models architecture, some extra time was spent on tuning the models to ensure a fair comparison. +To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible, and those hyperparameters that did not have default options were selected based on values found in the literature. +However, due to the nature of the neural network models' architecture, some extra time was spent on tuning the models to ensure a fair comparison. Finally, all the models ran only one trial per oxide. Although running multiple trials would have allowed for an average performance evaluation, we argued that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. Additionally, as stated, the goal of this experiment was merely to get an initial indication of the performance of the models. From ff06fe5e96fcc37defea96ae5dc61ead2105a179 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Mon, 10 Jun 2024 20:23:42 +0200 Subject: [PATCH 11/18] rm irrelevant params --- report_thesis/src/sections/appendix.tex | 3 --- 1 file changed, 3 deletions(-) diff --git a/report_thesis/src/sections/appendix.tex b/report_thesis/src/sections/appendix.tex index 6f9c3bc3..ea3c94c4 100644 --- a/report_thesis/src/sections/appendix.tex +++ b/report_thesis/src/sections/appendix.tex @@ -283,9 +283,6 @@ \subsection{Initial Experiment: Model Hyperparameters}\label{subsec:initial_expe Layer 16: Dropout & (rate: 0.5) \\ Layer 17: Dense & (512 units, activation: relu) \\ Layer 18: Dense & (1 unit, activation: linear) \\ - Total Parameters & 25,274,245 \\ - Trainable Parameters & 25,274,243 \\ - Non-trainable Parameters & 2 \\ \bottomrule \end{tabular} \label{tab:cnn_hyperparameters} From 5a1110d30454e6c581985d8293e1c8d88547dacc Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Mon, 10 Jun 2024 21:14:59 +0200 Subject: [PATCH 12/18] wip --- report_thesis/src/sections/appendix.tex | 330 +++++------------- .../experiment_design/experimental_setup.tex | 2 +- 2 files changed, 95 insertions(+), 237 deletions(-) diff --git a/report_thesis/src/sections/appendix.tex b/report_thesis/src/sections/appendix.tex index ea3c94c4..c7b6629f 100644 --- a/report_thesis/src/sections/appendix.tex +++ b/report_thesis/src/sections/appendix.tex @@ -46,245 +46,103 @@ \subsection{Cross-Validation Plots for Major Oxides}\label{subsec:cv_plots} } \subsection{Initial Experiment: Model Hyperparameters}\label{subsec:initial_experiment_hyperparameters} -This section provides a detailed overview of the hyperparameters used for the models in the initial experiment. - -\begin{table}[H] -\centering -\caption{Hyperparameters for Partial Least Squares} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Number of Components & 34 \\ - Scale & True \\ - Max Iter & 500 \\ - \bottomrule -\end{tabular} -\label{tab:pls_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Support Vector Regression} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Kernel & poly \\ - C & 100 \\ - Epsilon & 0.1 \\ - Gamma & scale \\ - Degree & 2 \\ - Coef0 & 1.0 \\ - \bottomrule -\end{tabular} -\label{tab:svr_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Elastic Nets} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ - L1 Ratio & [0.1, 0.5, 0.7, 0.9, 1.0] \\ - Max Iter & 1000 \\ - Tolerance & 1e-4 \\ - \bottomrule -\end{tabular} -\label{tab:elasticnet_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Lasso} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ - Max Iter & 1000 \\ - Tolerance & 1e-4 \\ - \bottomrule -\end{tabular} -\label{tab:lasso_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Ridge Regression} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Alphas & [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000] \\ - Max Iter & 1000 \\ - Tolerance & 1e-4 \\ - \bottomrule -\end{tabular} -\label{tab:ridge_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Random Forest} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Number of Estimators & 100 \\ - Max Depth & 10 \\ - Min Samples Split & 2 \\ - Min Samples Leaf & 1 \\ - Max Features & sqrt \\ - Random State & 42 \\ - \bottomrule -\end{tabular} -\label{tab:randomforest_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] +\begin{table*}[h] \centering -\caption{Hyperparameters for Gradient Boosting Regression} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Number of Estimators & 100 \\ - Max Depth & 3 \\ - Min Samples Split & 2 \\ - Min Samples Leaf & 1 \\ - Max Features & None \\ - Loss & squared\_error \\ - Learning Rate & 0.1 \\ - Subsample & 1.0 \\ - Criterion & friedman\_mse \\ - Random State & 42 \\ - Verbose & 0 \\ - Validation Fraction & 0.1 \\ - Number of Iterations No Change & None \\ - Tolerance & 1e-4 \\ - CCP Alpha & 0.0 \\ - \bottomrule +\begin{tabular}{@{}llp{0.5\textwidth}@{}} +\toprule +\textbf{Model} & \textbf{Hyperparameter} & \textbf{Value} \\ +\midrule +\multirow{3}{*}{\gls{pls}} +& \texttt{n\_components} & 34 \\ +& \texttt{scale} & True \\ +& \texttt{max\_iter} & 500 \\ +\midrule +\multirow{6}{*}{\gls{svr}} +& \texttt{kernel} & poly \\ +& \texttt{C} & 100 \\ +& \texttt{epsilon} & 0.1 \\ +& \texttt{gamma} & scale \\ +& \texttt{degree} & 2 \\ +& \texttt{coef0} & 1.0 \\ +\midrule +\multirow{3}{*}{Ridge Regression} +& \texttt{alphas} & \{$10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$, 1, 10, $10^2$, $10^3$\} \\ +& \texttt{max\_iter} & 1000 \\ +& \texttt{tol} & $10^{-4}$ \\ +\midrule +\multirow{3}{*}{\gls{lasso}} +& \texttt{alphas} & \{$10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$, 1, 10, $10^2$, $10^3$\} \\ +& \texttt{max\_iter} & 1000 \\ +& \texttt{tol} & $10^{-4}$ \\ +\midrule +\multirow{4}{*}{\gls{enet}} +& \texttt{alphas} & \{$10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$, 1, 10, $10^2$, $10^3$\} \\ +& \texttt{l1\_ratio} & \{0.1, 0.5, 0.7, 0.9, 1.0\} \\ +& \texttt{max\_iter} & 1000 \\ +& \texttt{tol} & $10^{-4}$ \\ +\midrule +\multirow{6}{*}{\gls{rf}} +& \texttt{n\_estimators} & 100 \\ +& \texttt{max\_depth} & 10 \\ +& \texttt{min\_samples\_split} & 2 \\ +& \texttt{min\_samples\_leaf} & 1 \\ +& \texttt{max\_features} & sqrt \\ +& \texttt{random\_state} & 42 \\ +\midrule +\multirow{5}{*}{\gls{etr}} +& \texttt{n\_estimators} & 100 \\ +& \texttt{max\_depth} & 10 \\ +& \texttt{min\_samples\_split} & 2 \\ +& \texttt{min\_samples\_leaf} & 1 \\ +& \texttt{random\_state} & 42 \\ +\midrule \end{tabular} -\label{tab:gbr_hyperparameters} -\end{table} -\FloatBarrier +\caption{Explictly set hyperparameters for the \gls{pls}, \gls{svr}, ridge, \gls{lasso}, \gls{enet}, \gls{rf}, and \gls{etr} models. When not explicitly set, the default hyperparameters provided by the libraries listed in Section~\ref{sec:subsec:experimental_setup} are used.} +\label{tab:combined_hyperparameters} +\end{table*} -\begin{table}[H] -\centering -\caption{Hyperparameters for Extra Trees Regression} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Number of Estimators & 100 \\ - Max Depth & 10 \\ - Min Samples Split & 2 \\ - Min Samples Leaf & 1 \\ - Random State & 42 \\ - \bottomrule -\end{tabular} -\label{tab:extratrees_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for XGBoost} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Max Depth & 4 \\ - Min Child Weight & 5 \\ - Gamma & 0.1 \\ - Subsample & 0.7 \\ - Colsample Bytree & 0.5 \\ - Colsample Bylevel & 0.5 \\ - Colsample Bynode & 0.5 \\ - Lambda & 1 \\ - Alpha & 0.5 \\ - Learning Rate & 0.05 \\ - Number of Estimators & 100 \\ - Objective & reg:squarederror \\ - Eval Metric & rmse \\ - \bottomrule -\end{tabular} -\label{tab:xgboost_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for NGBoost} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - n/a & n/a \\ - \bottomrule -\end{tabular} -\label{tab:ngboost_hyperparameters} -\end{table} -\FloatBarrier - -\begin{table}[H] -\centering -\caption{Hyperparameters for Artificial Neural Networks} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Layer 1: Dense & (1024 units) \\ - Layer 2: Dropout & (rate: 0.5) \\ - Layer 3: Dense & (512 units) \\ - Layer 4: Dropout & (rate: 0.5) \\ - Layer 5: Dense & (256 units) \\ - Layer 6: Dense & (128 units) \\ - Layer 7: Dense & (1 unit) \\ - \bottomrule -\end{tabular} -\label{tab:ann_hyperparameters} -\end{table} -\FloatBarrier -\clearpage -\begin{table}[H] +\begin{table*}[h] \centering -\caption{Hyperparameters for Convolutional Neural Networks} -\begin{tabular}{ll} - \toprule - \textbf{Hyperparameter} & \textbf{Value} \\ - \midrule - Layer 1: InputLayer & (6144, 1) \\ - Layer 2: BatchNormalization & (axis: -1) \\ - Layer 3: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ - Layer 4: MaxPooling1D & (pool size: 2) \\ - Layer 5: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ - Layer 6: MaxPooling1D & (pool size: 2) \\ - Layer 7: Conv1D & (filters: 64, kernel size: 3, activation: relu) \\ - Layer 8: Add & (merge with layer 6 output) \\ - Layer 9: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ - Layer 10: MaxPooling1D & (pool size: 2) \\ - Layer 11: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ - Layer 12: MaxPooling1D & (pool size: 2) \\ - Layer 13: Conv1D & (filters: 128, kernel size: 3, activation: relu) \\ - Layer 14: Add & (merge with layer 12 output) \\ - Layer 15: Flatten & (output) \\ - Layer 16: Dropout & (rate: 0.5) \\ - Layer 17: Dense & (512 units, activation: relu) \\ - Layer 18: Dense & (1 unit, activation: linear) \\ - \bottomrule +\begin{tabular}{@{}llp{0.5\textwidth}@{}} +\toprule +\textbf{Model} & \textbf{Hyperparameter} & \textbf{Value} \\ +\midrule +\multirow{14}{*}{\gls{gbr}} +& \texttt{n\_estimators} & 100 \\ +& \texttt{max\_depth} & 3 \\ +& \texttt{min\_samples\_split} & 2 \\ +& \texttt{min\_samples\_leaf} & 1 \\ +& \texttt{max\_features} & None \\ +& \texttt{loss} & squared\_error \\ +& \texttt{learning\_rate} & 0.1 \\ +& \texttt{subsample} & 1.0 \\ +& \texttt{criterion} & friedman\_mse \\ +& \texttt{random\_state} & 42 \\ +& \texttt{verbose} & 0 \\ +& \texttt{validation\_fraction} & 0.1 \\ +& \texttt{n\_iter\_no\_change} & None \\ +& \texttt{tol} & $10^{-4}$ \\ +& \texttt{ccp\_alpha} & 0.0 \\ +\midrule +\gls{ngboost} & - & - \\ +\midrule +\multirow{14}{*}{\gls{xgboost}} +& \texttt{max\_depth} & 4 \\ +& \texttt{min\_child\_weight} & 5 \\ +& \texttt{gamma} & 0.1 \\ +& \texttt{subsample} & 0.7 \\ +& \texttt{colsample\_bytree} & 0.5 \\ +& \texttt{colsample\_bylevel} & 0.5 \\ +& \texttt{colsample\_bynode} & 0.5 \\ +& \texttt{lambda} & 1 \\ +& \texttt{alpha} & 0.5 \\ +& \texttt{learning\_rate} & 0.05 \\ +& \texttt{n\_estimators} & 100 \\ +& \texttt{objective} & reg:squarederror \\ +& \texttt{eval\_metric} & rmse \\ +\bottomrule \end{tabular} -\label{tab:cnn_hyperparameters} -\end{table} -\FloatBarrier \ No newline at end of file +\caption{Explictly set hyperparameters for the \gls{gbr} and \gls{xgboost} models. When not explicitly set, the default hyperparameters provided by the libraries listed in Section~\ref{sec:subsec:experimental_setup} are used. The \gls{ngboost} model does not have any explicitly set hyperparameters.} +\label{tab:combined_hyperparameters} +\end{table*} diff --git a/report_thesis/src/sections/experiment_design/experimental_setup.tex b/report_thesis/src/sections/experiment_design/experimental_setup.tex index 764343c0..7b5754d8 100644 --- a/report_thesis/src/sections/experiment_design/experimental_setup.tex +++ b/report_thesis/src/sections/experiment_design/experimental_setup.tex @@ -1,4 +1,4 @@ -\subsection{Experimental Setup} +\subsection{Experimental Setup}\label{sec:experimental_setup} Experiments were conducted on a machine equipped with an Intel Xeon Gold 6242 CPU, featuring 16 cores and 32 threads. The CPU has a base clock speed of 2.80 GHz and a maximum turbo frequency of 3.90 GHz. The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. From 911bcc8b0a692bb3d2ca8b4ca2e6e2688d101b20 Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Mon, 10 Jun 2024 21:53:15 +0200 Subject: [PATCH 13/18] add nn architectur --- report_thesis/src/sections/appendix.tex | 50 +++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/report_thesis/src/sections/appendix.tex b/report_thesis/src/sections/appendix.tex index c7b6629f..53aeb2c6 100644 --- a/report_thesis/src/sections/appendix.tex +++ b/report_thesis/src/sections/appendix.tex @@ -146,3 +146,53 @@ \subsection{Initial Experiment: Model Hyperparameters}\label{subsec:initial_expe \caption{Explictly set hyperparameters for the \gls{gbr} and \gls{xgboost} models. When not explicitly set, the default hyperparameters provided by the libraries listed in Section~\ref{sec:subsec:experimental_setup} are used. The \gls{ngboost} model does not have any explicitly set hyperparameters.} \label{tab:combined_hyperparameters} \end{table*} + +\begin{table*}[h!] +\centering +\begin{minipage}{.45\textwidth} + \centering + \begin{tabular}{ll} + \toprule + \textbf{Layer} & \textbf{Output Shape} \\ \midrule + Input & (\textit{input\_dim},) \\ + Dense & (1024,) \\ + Dropout & (1024,) \\ + Dense & (512,) \\ + Dropout & (512,) \\ + Dense & (256,) \\ + Dense & (128,) \\ + Output & (\textit{output\_dim},) \\ \bottomrule + \end{tabular} + \caption{Summary of the Neural Network Architecture} + \label{tab:nn_architecture} +\end{minipage}% +\hspace{0.04\textwidth} +\begin{minipage}{.45\textwidth} + \centering + \begin{tabular}{ll} + \toprule + \textbf{Layer} & \textbf{Output Shape} \\ \midrule + Input & (\textit{input\_dim},) \\ + Reshape & (48, 128, 1) \\ + Conv2D & (48, 128, 32) \\ + BatchNormalization & (48, 128, 32) \\ + MaxPooling2D & (24, 64, 32) \\ + Conv2D & (24, 64, 32) \\ + BatchNormalization & (24, 64, 32) \\ + MaxPooling2D & (12, 32, 32) \\ + Conv2D & (12, 32, 64) \\ + BatchNormalization & (12, 32, 64) \\ + MaxPooling2D & (6, 16, 64) \\ + Conv2D & (6, 16, 128) \\ + BatchNormalization & (6, 16, 128) \\ + MaxPooling2D & (3, 8, 128) \\ + Flatten & (3072,) \\ + Dense & (256,) \\ + Dropout & (256,) \\ + Dense & (\textit{output\_dim},) \\ + Dense & (\textit{output\_dim},) \\ \bottomrule + \end{tabular} + \caption{Summary of the Convolutional Neural Network Architecture} + \label{tab:cnn_architecture} +\end{minipage} +\end{table*} From 7aae1a4c274ca3c0ce9121419c526e2588b0b7cb Mon Sep 17 00:00:00 2001 From: Pattrigue Date: Mon, 10 Jun 2024 22:12:42 +0200 Subject: [PATCH 14/18] nn tables 2 --- report_thesis/src/sections/appendix.tex | 82 ++++++++++++++----------- 1 file changed, 45 insertions(+), 37 deletions(-) diff --git a/report_thesis/src/sections/appendix.tex b/report_thesis/src/sections/appendix.tex index 53aeb2c6..6aa99b45 100644 --- a/report_thesis/src/sections/appendix.tex +++ b/report_thesis/src/sections/appendix.tex @@ -148,51 +148,59 @@ \subsection{Initial Experiment: Model Hyperparameters}\label{subsec:initial_expe \end{table*} \begin{table*}[h!] -\centering -\begin{minipage}{.45\textwidth} \centering - \begin{tabular}{ll} + \begin{tabular}{lll} \toprule - \textbf{Layer} & \textbf{Output Shape} \\ \midrule - Input & (\textit{input\_dim},) \\ - Dense & (1024,) \\ - Dropout & (1024,) \\ - Dense & (512,) \\ - Dropout & (512,) \\ - Dense & (256,) \\ - Dense & (128,) \\ - Output & (\textit{output\_dim},) \\ \bottomrule + \textbf{Layer} & \textbf{Output Shape} & \textbf{Hyperparameter} \\ \midrule + Input & (\textit{input\_dim},) & - \\ + Dense & (1024,) & activation = ReLU \\ + Dropout & (1024,) & rate = 0.3 \\ + Dense & (512,) & activation = ReLU \\ + Dropout & (512,) & rate = 0.3 \\ + Dense & (256,) & activation = ReLU \\ + Dense & (128,) & activation = ReLU \\ + Output & (\textit{output\_dim},) & - \\ + \midrule + \multicolumn{3}{l}{\textbf{Optimizer:} Adam} \\ + \multicolumn{3}{l}{\textbf{Learning Rate:} 0.001} \\ + \bottomrule \end{tabular} \caption{Summary of the Neural Network Architecture} \label{tab:nn_architecture} -\end{minipage}% -\hspace{0.04\textwidth} -\begin{minipage}{.45\textwidth} +\end{table*} + +\begin{table*}[h!] \centering - \begin{tabular}{ll} + \begin{tabular}{lll} \toprule - \textbf{Layer} & \textbf{Output Shape} \\ \midrule - Input & (\textit{input\_dim},) \\ - Reshape & (48, 128, 1) \\ - Conv2D & (48, 128, 32) \\ - BatchNormalization & (48, 128, 32) \\ - MaxPooling2D & (24, 64, 32) \\ - Conv2D & (24, 64, 32) \\ - BatchNormalization & (24, 64, 32) \\ - MaxPooling2D & (12, 32, 32) \\ - Conv2D & (12, 32, 64) \\ - BatchNormalization & (12, 32, 64) \\ - MaxPooling2D & (6, 16, 64) \\ - Conv2D & (6, 16, 128) \\ - BatchNormalization & (6, 16, 128) \\ - MaxPooling2D & (3, 8, 128) \\ - Flatten & (3072,) \\ - Dense & (256,) \\ - Dropout & (256,) \\ - Dense & (\textit{output\_dim},) \\ - Dense & (\textit{output\_dim},) \\ \bottomrule + \textbf{Layer} & \textbf{Output Shape} & \textbf{Hyperparameter} \\ \midrule + Input & (\textit{input\_dim},) & - \\ + Reshape & (48, 128, 1) & - \\ + Conv2D & (48, 128, 32) & filters = 32, kernel\_size = (3, 3), activation = ReLU, padding = 'same' \\ + BatchNormalization & (48, 128, 32) & - \\ + MaxPooling2D & (24, 64, 32) & pool\_size = (2, 2) \\ \midrule + + Conv2D & (24, 64, 32) & filters = 32, kernel\_size = (3, 3), activation = ReLU, padding = 'same' \\ + BatchNormalization & (24, 64, 32) & - \\ + MaxPooling2D & (12, 32, 32) & pool\_size = (2, 2) \\ \midrule + + Conv2D & (12, 32, 64) & filters = 64, kernel\_size = (3, 3), activation = ReLU, padding = 'same' \\ + BatchNormalization & (12, 32, 64) & - \\ + MaxPooling2D & (6, 16, 64) & pool\_size = (2, 2) \\ \midrule + + Conv2D & (6, 16, 128) & filters = 128, kernel\_size = (3, 3), activation = ReLU, padding = 'same' \\ + BatchNormalization & (6, 16, 128) & - \\ + MaxPooling2D & (3, 8, 128) & pool\_size = (2, 2) \\ \midrule + + Flatten & (3072,) & - \\ + Dense & (256,) & activation = ReLU \\ + Dropout & (256,) & rate = 0.5 \\ + Dense & (\textit{output\_dim},) & - \\ + Dense & (\textit{output\_dim},) & kernel\_regularizer = $L_2(0.01)$ \\ \midrule + \multicolumn{3}{l}{\textbf{Optimizer:} Adam} \\ + \multicolumn{3}{l}{\textbf{Learning Rate:} 0.001} \\ + \bottomrule \end{tabular} \caption{Summary of the Convolutional Neural Network Architecture} \label{tab:cnn_architecture} -\end{minipage} \end{table*} From 25a982ca2a93d4769ba00053540e87cdc0db2dd7 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Mon, 10 Jun 2024 22:26:22 +0200 Subject: [PATCH 15/18] fixed intro --- report_thesis/src/sections/experiment_design/index.tex | 9 ++++++--- .../src/sections/proposed_approach/model_selection.tex | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/index.tex b/report_thesis/src/sections/experiment_design/index.tex index 118dccbe..f504a445 100644 --- a/report_thesis/src/sections/experiment_design/index.tex +++ b/report_thesis/src/sections/experiment_design/index.tex @@ -1,7 +1,10 @@ \section{Experimental Design}\label{sec:methodology} -In this section, we outline the methodology used in this study to address the challenges identified in Section~\ref{sec:problem_definition}. Our objective was to identify the most promising machine learning models and preprocessing techniques proposed from the literature, as outlined in Section~\ref{sec:related-work}, for predicting major oxide compositions from \gls{libs} data. -Then, using this knowledge, develop a pipeline that utilizes the strengths of these models and preprocessing techniques to improve prediction accuracy and robustness of the predictions. -We first describe the datasets used, including their preparation and the method of splitting for model training. Next, we outline the preprocessing steps and the model selection process, followed by a detailed explanation of the experimental setup and evaluation metrics. Finally, we discuss our validation testing procedures and the approach taken to ensure unbiased final model evaluations. +This section outlines the experimental design used to identify the top-$n$ models to be used in our stacking ensemble. +We first describe the prequisite data preparation for all our experiments, followed by a description of the hardware and software used. +Next, we outline the design of our initial experiment, aimed at providing a preliminary assesment of the models selected in Section~\ref{subsec:model-selection}. +Following this, we present and discuss the results of this experiment. +We then describe the design of our main experiment, where we use our hyperparameter tuning framework to identify the top-$n$ models. +Finally, we present and discuss the results of this experiment. \input{sections/experiment_design/data_preparation} \input{sections/experiment_design/experimental_setup} diff --git a/report_thesis/src/sections/proposed_approach/model_selection.tex b/report_thesis/src/sections/proposed_approach/model_selection.tex index e1515b50..e1d44dc7 100644 --- a/report_thesis/src/sections/proposed_approach/model_selection.tex +++ b/report_thesis/src/sections/proposed_approach/model_selection.tex @@ -1,4 +1,4 @@ -\subsection{Model and Preprocessing Selection} +\subsection{Model and Preprocessing Selection}\label{subsec:model-selection} Choosing the right models and preprocessing techniques for \gls{libs} data analysis is a challenging task. We had several considerations to guide our selection of preprocessing techniques. From 290326faf45ec23a206510047575556e22528ab5 Mon Sep 17 00:00:00 2001 From: Ivikhostrup Date: Mon, 10 Jun 2024 22:31:38 +0200 Subject: [PATCH 16/18] Fixed comments --- .../sections/experiment_design/initial_experiment.tex | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 7770e888..7b5a46ea 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -7,11 +7,8 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible, and those hyperparameters that did not have default options were selected based on values found in the literature. However, due to the nature of the neural network models' architecture, some extra time was spent on tuning the models to ensure a fair comparison. -Finally, all the models ran only one trial per oxide. -Although running multiple trials would have allowed for an average performance evaluation, we argued that the variability of the results in each trial would be small enough that an average performance would not change the ranking of the models. -Additionally, as stated, the goal of this experiment was merely to get an initial indication of the performance of the models. -Therefore, along with the choice of hyperparameters, it was not meaningful to run multiple trials for each model. +This included using batch normalization for the \gls{cnn} model, as early assesments showed that this was necessary to produce reasonable results. +Finally, we evaluated each model once per oxide given the selected configuration of hyperparameters. +As stated, the goal of this experiment was merely to get an initial indication of the performance of the models. - -% Add the fact that we used batchnorm for CNN The hyperparameters used for the models in the initial experiment can be found in the Appendix~\ref{subsec:initial_experiment_hyperparameters}. \ No newline at end of file From 11de0c9fb4a3f9edb951d12c54cb25b266aa5bf1 Mon Sep 17 00:00:00 2001 From: Ivikhostrup <56341364+Ivikhostrup@users.noreply.github.com> Date: Tue, 11 Jun 2024 11:26:14 +0200 Subject: [PATCH 17/18] Update report_thesis/src/sections/experiment_design/experimental_setup.tex --- .../src/sections/experiment_design/experimental_setup.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/experiment_design/experimental_setup.tex b/report_thesis/src/sections/experiment_design/experimental_setup.tex index 7b5754d8..ec1a8455 100644 --- a/report_thesis/src/sections/experiment_design/experimental_setup.tex +++ b/report_thesis/src/sections/experiment_design/experimental_setup.tex @@ -4,4 +4,4 @@ \subsection{Experimental Setup}\label{sec:experimental_setup} The system has 64 GB of RAM and runs on Ubuntu 22.04.2 LTS. Models were implemented using Python 3.10.11. The primary libraries used were Scikit-learn 1.4.2, XGBoost 2.0.3, Torch 2.2.2, NumPy 1.26.4, Pandas 2.2.1, Keras 3.2.1 and Optuna 3.6.1. -Additionally, all experiments were run using the hyperparameter optimization tool described in Section~\ref{subsec:hyperparameter_tuning_tool}. % TODO: Add correct ref once other PR is in \ No newline at end of file +Additionally, all experiments were run using the hyperparameter optimization tool described in Section~\ref{sec:optimization_framework}. \ No newline at end of file From bd51a5535f03afed9a7009937cdcce6a921cce19 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Tue, 11 Jun 2024 11:31:41 +0200 Subject: [PATCH 18/18] Update report_thesis/src/sections/experiment_design/initial_experiment.tex --- .../src/sections/experiment_design/initial_experiment.tex | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/report_thesis/src/sections/experiment_design/initial_experiment.tex b/report_thesis/src/sections/experiment_design/initial_experiment.tex index 7b5a46ea..ca1ce33c 100644 --- a/report_thesis/src/sections/experiment_design/initial_experiment.tex +++ b/report_thesis/src/sections/experiment_design/initial_experiment.tex @@ -4,7 +4,7 @@ \subsection{Design for Initial Experiment}\label{sec:initial-experiment} All models were trained on the same preprocessed data using the Norm 3 preprocessing method described in Section~\ref{sec:norm3}. This ensured that the models' performance could be evaluated under consistent and comparable conditions. -All models were trained using our data partitioning and cross-validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. +Furthermore, all experiments used our data partitioning and were evaluated using our testing and validation strategy, as described in Section~\ref{subsec:validation_testing_procedures}. To ensure as fair of a comparison between models as possible, all models were trained using as many default hyperparameters as possible, and those hyperparameters that did not have default options were selected based on values found in the literature. However, due to the nature of the neural network models' architecture, some extra time was spent on tuning the models to ensure a fair comparison. This included using batch normalization for the \gls{cnn} model, as early assesments showed that this was necessary to produce reasonable results.