Image size error when training with Swin Backbone and CIFAR datasets

[ImaGonEs]

Problem
When I try to run CIFAR datasets with Swin-Tiny Backbone, it return the error:
"AssertionError: Input image height (32) doesn't match model (224)."

Reproducibility
I'm using the following configuration file:

defaults:
  - _self_
  - augmentations: asymmetric.yaml
  - wandb: private.yaml
  - override hydra/hydra_logging: disabled
  - override hydra/job_logging: disabled



hydra:
  output_subdir: null
  run:
    dir: .

name: "nnclr-cifar100" # change here for cifar100
method: "nnclr"
backbone:
  name: "swin_tiny"
method_kwargs:
  temperature: 0.2
  proj_hidden_dim: 2048
  pred_hidden_dim: 4096
  proj_output_dim: 256
  queue_size: 98304
data:
  dataset: cifar100 # change here for cifar100
  train_path: "./datasets"
  val_path: "./datasets"
  format: "image_folder"
  num_workers: 4
optimizer:
  name: "lars"
  batch_size: 256
  lr: 0.4
  classifier_lr: 0.1
  weight_decay: 1e-5
  kwargs:
    clip_lr: True
    eta: 0.02
    exclude_bias_n_norm: True
scheduler:
  name: "warmup_cosine"
checkpoint:
  enabled: True
  dir: "trained_models"
  frequency: 1
auto_resume:
  enabled: False

# overwrite PL stuff
max_epochs: 1000
devices: [0]
sync_batchnorm: True
accelerator: "gpu"
strategy: "ddp"
precision: 16

Version used
I'm running the 92ec35f54a507fb5d4edc1dc5723abc022b39eb3 version of the library and this is my environment:

aiohttp==3.8.3
aiosignal==1.3.1
antlr4-python3-runtime==4.9.3
appdirs==1.4.4
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.2.0
certifi @ file:///croot/certifi_1671487769961/work/certifi
charset-normalizer==2.1.1
click==8.1.3
contourpy==1.0.7
cycler==0.11.0
docker-pycreds==0.4.0
einops==0.6.0
filelock==3.9.0
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2023.1.0
gast==0.4.0
gitdb==4.0.10
GitPython==3.1.30
h5py==3.8.0
huggingface-hub==0.12.0
hydra-core==1.3.1
idna==3.4
joblib==1.2.0
kiwisolver==1.4.4
lightning-bolts==0.6.0.post1
lightning-utilities==0.6.0.post0
llvmlite==0.39.1
matplotlib==3.6.3
multidict==6.0.4
numba==0.56.4
numpy==1.23.5
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-dali-cuda110==1.22.0
omegaconf==2.3.0
packaging==23.0
pandas==1.5.3
pathtools==0.1.2
Pillow==9.4.0
positional-encodings==6.0.1
protobuf==3.20.1
psutil==5.9.4
pynndescent==0.5.8
pyparsing==3.0.9
python-dateutil==2.8.2
pytorch-lightning==1.8.6
pytz==2022.7.1
PyYAML==6.0
requests==2.28.2
scikit-learn==1.2.1
scipy==1.10.0
seaborn==0.12.2
sentry-sdk==1.14.0
setproctitle==1.3.2
six==1.16.0
smmap==5.0.0
-e git+ssh://[email protected]/ImaGonEs/MyNewSolo.git@68e680143284db1992518146f075206e64e451ae#egg=solo_learn
tensorboardX==2.5.1
threadpoolctl==3.1.0
timm==0.6.12
torch==1.13.1
torchmetrics==0.11.0
torchvision==0.14.1
tqdm==4.64.1
typing_extensions==4.4.0
umap-learn==0.5.3
urllib3==1.26.14
wandb==0.13.9
yarl==1.8.2`

Additional comments
I think that on older versions of the library (before the omegaconf update) I could run the Swin Backbone without problems. Also, I believe that, at first, there was a line that changed the window size from 7 to 4 in the case of CIFAR dataset but I can't find it anymore.

[vturrisi]

Hey. Thanks for letting us know. I think it's just an issue with the config file. When we moved to omegaconf, we decided to allow the user to have more control of the settings. You should set all backbone kwargs like below (it should also fix your issue):

backbone:
  name: "swin_tiny"
  kwargs:
    window_size: 4

For reference, this is how we access the kwargs for the backbone:

solo-learn/solo/methods/base.py

Line 183 in 1aeb6e9

self.backbone_args: Dict[str, Any] = cfg.backbone.kwargs

And this is the swin backbone constructor:

solo-learn/solo/backbones/swin/swin.py

Lines 27 to 37 in 1aeb6e9

	def swin_tiny(window_size=7, **kwargs):
	model_kwargs = dict(
	patch_size=4,
	window_size=window_size,
	embed_dim=96,
	depths=(2, 2, 6, 2),
	num_heads=(3, 6, 12, 24),
	num_classes=0,
	**kwargs,
	)
	return _create_swin_transformer("swin_tiny_patch4_window7_224", **model_kwargs)

Let me know if this fixes your issue and I'll also update the config files :)

[ImaGonEs]

Hi!

Thanks for your fast response. I've just tried what you mention but, even if that solves the previous error, it raises a new one:

RuntimeError: shape '[1, 3, 4, 3, 4, 1]' is invalid for input of size 196

[vturrisi]

Hmm, can you link the full error? I'll try to check it later.

[ImaGonEs]

Sure!

The configuration file is:

defaults:
  - _self_
  - augmentations: asymmetric.yaml
  - wandb: private.yaml
  - override hydra/hydra_logging: disabled
  - override hydra/job_logging: disabled

# disable hydra outputs
hydra:
  output_subdir: null
  run:
    dir: .

name: "nnclr-cifar100" # change here for cifar100
method: "nnclr"
backbone:
  name: "swin_tiny"
  kwargs:
    window_size: 4  
method_kwargs:
  temperature: 0.2
  proj_hidden_dim: 2048
  pred_hidden_dim: 4096
  proj_output_dim: 256
  queue_size: 98304
data:
  dataset: cifar100 # change here for cifar100
  train_path: "./datasets"
  val_path: "./datasets"
  format: "image_folder"
  num_workers: 4
optimizer:
  name: "lars"
  batch_size: 256
  lr: 0.4
  classifier_lr: 0.1
  weight_decay: 1e-5
  kwargs:
    clip_lr: True
    eta: 0.02
    exclude_bias_n_norm: True
scheduler:
  name: "warmup_cosine"
checkpoint:
  enabled: True
  dir: "trained_models"
  frequency: 1
auto_resume:
  enabled: False

# overwrite PL stuff
max_epochs: 1000
devices: [0]
sync_batchnorm: True
accelerator: "gpu"
strategy: "ddp"
precision: 16

And the error is:

  File "/media/HDD_4TB/ima/work_dir/py/main_pretrain.py", line 75, in main
    model = METHODS[cfg.method](cfg)
  File "/media/HDD_4TB/ima/envs/neo_ssl/solo-learn/solo/methods/nnclr.py", line 43, in __init__
    super().__init__(cfg)
  File "/media/HDD_4TB/ima/envs/neo_ssl/solo-learn/solo/methods/base.py", line 192, in __init__
    self.backbone: nn.Module = self.base_model(method, **kwargs)
  File "/media/HDD_4TB/ima/envs/neo_ssl/solo-learn/solo/backbones/swin/__init__.py", line 27, in swin_tiny
    return default_swin_tiny(*args, **kwargs)
  File "/media/HDD_4TB/ima/envs/neo_ssl/solo-learn/solo/backbones/swin/swin.py", line 40, in swin_tiny
    return _create_swin_transformer("swin_tiny_patch4_window7_224", **model_kwargs)
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 574, in _create_swin_transformer
    model = build_model_with_cfg(
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/helpers.py", line 537, in build_model_with_cfg
    model = model_cls(**kwargs) if model_cfg is None else model_cls(cfg=model_cfg, **kwargs)
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 486, in __init__
    layers += [BasicLayer(
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 402, in __init__
    self.blocks = nn.Sequential(*[
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 403, in <listcomp>
    SwinTransformerBlock(
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 281, in __init__
    mask_windows = window_partition(img_mask, self.window_size)  # num_win, window_size, window_size, 1
  File "/media/HDD_4TB/ima/envs/neo_ssl/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 111, in window_partition
    x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
RuntimeError: shape '[1, 3, 4, 3, 4, 1]' is invalid for input of size 196

Thanks in advance for the help!

[vturrisi]

Hey. Sorry but I didn't have time to check it yet. I'll try to do it next week.