Learning to Defer with Limited Expert Predictions

This repository includes the code to reproduce the experiments for our paper (paper-source).

Contents

This repository is structured as follows:

• Embedding-Semi-Supervised: Code for the proposed Emb-SSL implementations Embedding-FixMatch and Embedding-CoMatch
• Embedding-Supervised: Code for the Emb-SL baselines Embedding-SVM and Embedding-NN
• Semi-Supervised: Code for the SSL baselines FixMatch and CoMatch
• Learning-to-Defer-Algs: Implementations of the learning to defer algorithms of Mozannar & Sontag, Okati et al., and Raghu et al..

Requirements

This code depends on the following packages:

torch
torchvision
torchtext
timm
scipy
seaborn
numpy
matplotlib
scikit-learn
tensorboard_logger

Execution

Follow these steps to reproduce our experiments:

1. (NIH experiments only) Download and extract the NIH dataset to nih_images/
2. (CIFAR experiments only) Generate the synthetic expert labels
3. Execute the training of one of the proposed Embedding-Semi-Supervised approaches (or any baseline)
4. Generate the artificial expert labels
5. Execute the training of one of the learning to defer algorithms

Detailed instructions can be found in the respective subfolders.