Skip to content

Latest commit

 

History

History
174 lines (119 loc) · 10.2 KB

README.md

File metadata and controls

174 lines (119 loc) · 10.2 KB
Raw

Open Performance Benchmark on Tabular Data

Basis for various experiments on deep learning models for tabular data. See the Deep Neural Networks and Tabular Data: A Survey paper.

Results

Open performance benchmark results based on (stratified) 5-fold cross-validation. We use the same fold splitting strategy for every data set. The top results for each data set are in bold. The mean and standard deviation values are reported for each baseline model. Missing results indicate that the corresponding model could not be applied to the task type (regression or multi-class classification)

Method HELOC Adult HIGGS Covertype Cal. Housing
Acc↑ AUC↑ Acc↑ AUC↑ Acc↑ AUC↑ Acc↑ AUC↑ MSE↓
Linear Model 73.0±0.0 80.1±0.1 82.5±0.2 85.4±0.2 64.1±0.0 68.4±0.0 72.4±0.0 92.8±0.0 0.528±0.008
KNN 72.2±0.0 79.0±0.1 83.2±0.2 87.5±0.2 62.3±0.1 67.1±0.0 70.2±0.1 90.1±0.2 0.421±0.009
Decision Tree 80.3±0.0 89.3±0.1 85.3±0.2 89.8±0.1 71.3±0.0 78.7±0.0 79.1±0.0 95.0±0.0 0.404±0.007
Random Forest 82.1±0.3 90.0±0.2 86.1±0.2 91.7±0.2 71.9±0.0 79.7±0.0 78.1±0.1 96.1±0.0 0.272±0.006
XGBoost 83.5±0.2 92.2±0.0 87.3±0.2 92.8±0.1 77.6±0.0 85.9±0.0 97.3±0.0 99.9±0.0 0.206±0.005
LightGBM 83.5±0.1 92.3±0.0 87.4±0.2 92.9±0.1 77.1±0.0 85.5±0.0 93.5±0.0 99.7±0.0 0.195±0.005
CatBoost 83.6±0.3 92.4±0.1 87.2±0.2 92.8±0.1 77.5±0.0 85.8±0.0 96.4±0.0 99.8±0.0 0.196±0.004
Model Trees 82.6±0.2 91.5±0.0 85.0±0.2 90.4±0.1 69.8±0.0 76.7±0.0 - - 0.385±0.019
MLP 73.2±0.3 80.3±0.1 84.8±0.1 90.3±0.2 77.1±0.0 85.6±0.0 91.0±0.4 76.1±3.0 0.263±0.008
VIME 72.7±0.0 79.2±0.0 84.8±0.2 90.5±0.2 76.9±0.2 85.5±0.1 90.9±0.1 82.9±0.7 0.275±0.007
DeepFM 73.6±0.2 80.4±0.1 86.1±0.2 91.7±0.1 76.9±0.0 83.4±0.0 - - 0.260±0.006
DeepGBM 78.0±0.4 84.1±0.1 84.6±0.3 90.8±0.1 74.5±0.0 83.0±0.0 - - 0.856±0.065
NODE 79.8±0.2 87.5±0.2 85.6±0.3 91.1±0.2 76.9±0.1 85.4±0.1 89.9±0.1 98.7±0.0 0.276±0.005
NAM 73.3±0.1 80.7±0.3 83.4±0.1 86.6±0.1 53.9±0.6 55.0±1.2 - - 0.725±0.022
Net-DNF 82.6±0.4 91.5±0.2 85.7±0.2 91.3±0.1 76.6±0.1 85.1±0.1 94.2±0.1 99.1±0.0 -
TabNet 81.0±0.1 90.0±0.1 85.4±0.2 91.1±0.1 76.5±1.3 84.9±1.4 93.1±0.2 99.4±0.0 0.346±0.007
TabTransformer 73.3±0.1 80.1±0.2 85.2±0.2 90.6±0.2 73.8±0.0 81.9±0.0 76.5±0.3 72.9±2.3 0.451±0.014
SAINT 82.1±0.3 90.7±0.2 86.1±0.3 91.6±0.2 79.8±0.0 88.3±0.0 96.3±0.1 99.8±0.0 0.226±0.004
RLN 73.2±0.4 80.1±0.4 81.0±1.6 75.9±8.2 71.8±0.2 79.4±0.2 77.2±1.5 92.0±0.9 0.348±0.013
STG 73.1±0.1 80.0±0.1 85.4±0.1 90.9±0.1 73.9±0.1 81.9±0.1 81.8±0.3 96.2±0.0 0.285±0.006

How to use

Using the docker container

The code is designed to run inside a docker container. See the Dockerfile. In the docker file, different conda environments are specified for the various requirements of the models. Therefore, building the container for the first time takes a while.

Just build it as usual via docker build -t <image name> <path to Dockerfile>.

To start the docker container then run:

docker run -v ~/output:/opt/notebooks/output -p 3123:3123 --rm -it --gpus all <image name>

  • The -v ~/output:/opt/notebooks/output option is recommended to have access to the outputs of the experiments on your local machine.

  • The docker run command starts a jupyter notebook (to have a nice editor for small changes or experiments). To have access to the notebook from outside the docker container, -p 3123:3123 connects the notebook to your local machine. You can change the port number in the Dockerfile.

  • If you have GPUs available, add also the --gpus all option to have access to them from inside the docker container.

To enter the running docker container via the command do the following:

  • Call docker ps to find the ID of the running container.
  • Call docker exec -it <container id> bash to enter the container. Now you can navigate to the right directory with cd opt/notebooks/.

Run a single model on a single dataset

To run a single model on a single dataset call:

python train.py --config/<config-file of the dataset>.yml --model_name <Name of the Model>

All parameters set in the config file, can be overwritten by command line arguments, for example:

  • --optimize_hyperparameters Uses Optuna to run a hyperparameter optimization. If not set, the parameters listed in the best_params.yml file are used.

  • --n_trails <number trials> Number of trials to run for the hyperparameter search

  • --epochs <number epochs> Max number of epochs

  • --use_gpu If set, available GPUs are used (specified by gpu_ids)

  • ... and so on. All possible parameters can be found in the config files or calling: python train.y -h

If you are using the docker container, first enter the right conda environment using conda activate <env name> to have all required packages. The train.py file is in the opt/notebooks/ directory.

Run multiple models on multiple datasets

To run multiple models on multiple datasets, there is the bash script testall.sh provided. In the bash script the models and datasets can be specified. Every model needs to know in which conda environment in has to be executed.

If you run inside our docker container, just comment out all models and datasets you don't want to run and then call:

./testall.sh

Computing model attributions (currently supported for SAINT, TabTransformer, TabNet)

The framework provides implementations to compute feature attribution explanations for several models. Additionally, the feature attributions can be automatically compared to SHAP values and a global ablation test which successively perturbs the most important features, can be run. The same parameters as before can be passed, but with some additions:

attribute.py --model_name <Name of the Model> [--globalbenchmark] [--compareshap] [--numruns <int>] [--strategy diag]

  • --globalbenchmark Additionally run the global perturbation benchmark

  • --compareshap Compare attributions to shapley values

  • --numruns <number run> Number of repetitions for the global benchmark

  • --strategy diag SAINT and TabTransformer support another attribution strategy, where the diagonal of the attention map is used. Pass this argument to use it.

Add new models

Every new model should inherit from the base class BaseModel. Implement the following methods:

  • def __init__(self, params, args): Define your model here.
  • def fit(self, X, y, X_val=None, y_val=None): Implement the training process. (Return the loss and validation history)
  • def predict(self, X): Save and return the predictions on the test data - the regression values or the concrete classes for classification tasks
  • def predict_proba(self, X): Only for classification tasks. Save and return the probability distribution over the classes.
  • def define_trial_parameters(cls, trial, args): Define the hyperparameters that should be optimized.
  • (optional) def save_model: If you want to save your model in a specific manner, override this function to.

Add your <model>.py file to the models directory and do not forget to update the models/__init__.py file.

Add new datasets

Every dataset needs a config file specifying its features. Add the config file to the config directory.

Necessary information are:

  • dataset: Name of the dataset
  • objective: Binary, classification or regression task
  • direction: Direction of optimization. In the current implementation the binary scorer returns the AUC-score, hence, should be maximized. The classification scorer uses the log loss and the regression scorer mse, therefore both should be minimized.
  • num_features: Total number of features in the dataset
  • num_classes: Number of classes in classification task. Set to 1 for binary or regression task.
  • cat_idx: List the indices of the categorical features in your dataset (if there are any).

It is recommended to specify the remaining hyperparameters here as well.

Citation

If you use this codebase, please cite our work:

@article{borisov2022deep,
 author={Borisov, Vadim and Leemann, Tobias and Seßler, Kathrin and Haug, Johannes and Pawelczyk, Martin and Kasneci, Gjergji},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={Deep Neural Networks and Tabular Data: A Survey}, 
  year={2022},
  volume={},
  number={},
  pages={1-21},
  doi={10.1109/TNNLS.2022.3229161}
}