Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Enhancement on the output presentation #118

Open
Autochthonal opened this issue Oct 31, 2024 · 2 comments
Open

✨ Enhancement on the output presentation #118

Autochthonal opened this issue Oct 31, 2024 · 2 comments
Assignees
Labels
question Further information is requested

Comments

@Autochthonal
Copy link

Hi there!

The model will print the progress of loading the Validation dataloader redundantly besides each training epoch:

Epoch 147: 100%|██████████| 782/782 [00:18<00:00, 41.75it/s, v_num=2, train_loss=3.360, Acc%=37.70]
Validation: |          | 0/? [00:00<?, ?it/s]
Validation:   0%|          | 0/157 [00:00<?, ?it/s]
Validation DataLoader 0:   0%|          | 0/157 [00:00<?, ?it/s]
Validation DataLoader 0:   1%|          | 1/157 [00:00<00:00, 217.64it/s]
Validation DataLoader 0:   1%|▏         | 2/157 [00:00<00:01, 124.62it/s]
Validation DataLoader 0:   2%|▏         | 3/157 [00:00<00:01, 100.94it/s]
Validation DataLoader 0:   3%|▎         | 4/157 [00:00<00:01, 95.37it/s] 
Validation DataLoader 0:   3%|▎         | 5/157 [00:00<00:01, 92.54it/s]
Validation DataLoader 0:   4%|▍         | 6/157 [00:00<00:01, 91.46it/s]
......
Validation DataLoader 0: 100%|██████████| 157/157 [00:01<00:00, 92.35it/s]
Epoch 148: 100%|██████████| 782/782 [00:17<00:00, 45.04it/s, v_num=2, train_loss=4.180, Acc%=38.80]

I wonder whether there exists a parameter to avoid it while presents the validation acc in each training epochs just like:

Epoch 147: 100%|██████████| 782/782 [00:18<00:00, 41.75it/s, v_num=2, train_loss=3.360, Acc%=37.70]
Epoch 148: 100%|██████████| 782/782 [00:17<00:00, 45.04it/s, v_num=2, train_loss=4.180, Acc%=38.80]
Epoch 149: 100%|██████████| 782/782 [00:21<00:00, 37.12it/s, v_num=2, train_loss=3.970, Acc%=36.50]

Thanks alot!

@o-laurent o-laurent self-assigned this Nov 1, 2024
@o-laurent o-laurent added the question Further information is requested label Nov 1, 2024
@o-laurent
Copy link
Contributor

Hi @Autochthonal!

I've just tried a run and the validation progress bar has the same behavior as the training progress bar. It also disappears before switching to the next epoch, as shown here:

Sanity Checking: |                                                                                                                                          | 0/? [00:00<?, ?it/s]
Epoch 2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 391/391 [00:16<00:00, 23.56it/s, v_num=104, train_loss=0.923, Acc%=66.20]`Trainer.fit` stopped: `max_epochs=3` reached.                                                                                                                                    
Epoch 2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 391/391 [00:17<00:00, 22.71it/s, v_num=104, train_loss=0.923, Acc%=66.20]
Files already downloaded and verified
Files already downloaded and verified
Restoring states from the checkpoint path at logs/resnet18/standard/version_104/checkpoints/epoch=2-step=1173.ckpt
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Loaded model weights from the checkpoint at logs/resnet18/standard/version_104/checkpoints/epoch=2-step=1173.ckpt
Testing DataLoader 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 79/79 [00:01<00:00, 45.49it/s]

What is the resolution of your screen? Sometimes, when the resolution is too small or when you change the window size while training, the progress bars are not updated correctly. @alafage, do you have an idea for this?

Have a nice day!

@alafage
Copy link
Contributor

alafage commented Nov 4, 2024

I agree with @o-laurent. I only have this weird behavior when I change the terminal window size while training. We cannot easily change that since we rely on Lightning for the progress bar. Let us know if you think there might be a bug on our side we missed.

Best!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants