Training:
./train.py --config <config.yml>
Out-of-fold prediction:
./train.py --predict_oof --weights <model.pth> or ./predict_all.sh to use all pth files in the current directory.
Searching for blend coefficients:
./ensemble_search_scipy_optimize.py <ensemble_name_here> <prediction1.npy> <prediction2.npy> ... (one fold per model, other folds will be found automatically in the same directory).
This will generate ensemble_name_here.yml like this: https://github.com/artyompal/imet/blob/master/best_ensemble_val_0.6397_lb_651.yml
Predicting on the test set and generating submission file:
./ensemble_inference.py <ensemble.yml>
Generating a Kaggle kernels with submission (you should add all .pth and .yml files for every model into datasets):
./deploy_kernel.py ensemble_inference.py <ensemble.yml>
I decided that a big number of TTAs means essentially a blend of the same model, so I chose to use many different models. I even ran out of 20 Gb space of private datasets, so I added encrypted models in public datasets (they are not being used in the best ensemble, though).
My best ensemble is 7 models with TTA x2. It's supposed to finish Stage 2 prediction in 8 hrs 40 mins:
- SE-ResNext50 at 288x288;
- SE-ResNext101 at 288x288;
- CBAM-ResNet50 at 288x288;
- PNASNet5 Large at 288x288;
- another SE-ResNext101 at 288x288 with fewer augmentations and dropout 0.3;
- two more SE-ResNext101 at 352x352 with different augmentations.
The batch size was 16. This worked better than 32 or 64, and batch accumulation 4x or 8x didn't improve the score.
I used cross-entropy loss. Focal loss and F2 loss weren't better. After I realized there's a lot of missing labels, I came up with this loss:
class ForgivingLoss(nn.Module):
def __init__(self, weight: float) -> None:
super().__init__()
self.bce_loss = binary_cross_entropy()
self.weight = weight
def forward(self, logits: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
return self.bce_loss(logits, labels) + self.bce_loss(logits * labels, labels) * self.weight
Interestingly, it worked a little better with Inception-like models, namely Xception and InceptionResNetV2, but not with ResNext-like models.
I trained 5 folds of each model. Then I used scipy.optimize.minimize to find the best blend coefficients.
I used Kostia's method of deployment, which packs everything into a single py-file. Also, I wrote a code which automatically searches for all available models in ../input/ and decrypts them.
What didn't work: I tried pseudo-labeling, it improved a single fold score from 611 to 622, but worked worse on LB. Probably, I generated too many labels. I would be possible to achieve gold otherwise.