-
|
I have tried several times to execute the W CUDAFunctions.cpp:109] Warning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (function operator())
Epoch: 1 | Batch: 0 | Training loss: 2,3138 | Eval loss: 2,3010 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 200 | Training loss: 0,7689 | Eval loss: 0,7535 | Eval accuracy: 0,7741
Epoch: 1 | Batch: 400 | Training loss: 0,8344 | Eval loss: 0,9345 | Eval accuracy: 0,6736
Epoch: 1 | Batch: 600 | Training loss: 6,6774 | Eval loss: 9125501,0000 | Eval accuracy: 0,3796
Epoch: 1 | Batch: 800 | Training loss: 130458,1641 | Eval loss: 154628,1406 | Eval accuracy: 0,0974
Epoch: 1 | Batch: 1000 | Training loss: 363175656554496,0000 | Eval loss: 5094437340315648,0000 | Eval accuracy: 0,1421
Epoch: 1 | Batch: 1200 | Training loss: 178396147482624,0000 | Eval loss: 15321602761520103000000000,0000 | Eval accuracy: 0,1257
Epoch: 1 | Batch: 1400 | Training loss: 52412235874540960000000000,0000 | Eval loss: 188815448995733300000000000,0000 | Eval accuracy: 0,0654
Epoch: 1 | Batch: 1600 | Training loss: 289345426490735270000000000,0000 | Eval loss: 198066029980096800000000000,0000 | Eval accuracy: 0,0990
Epoch: 1 | Batch: 1800 | Training loss: 9335885960084007000000000000,0000 | Eval loss: 83514956802220020000000000000,0000 | Eval accuracy: 0,0905
Epoch: 2 | Batch: 0 | Training loss: 748397145925992400000000000000000000,0000 | Eval loss: Infinity | Eval accuracy: 0,1019
Epoch: 2 | Batch: 200 | Training loss: 445750120740847160000000000000000000,0000 | Eval loss: Infinity | Eval accuracy: 0,1017
Epoch: 2 | Batch: 400 | Training loss: 935137766515788400000000000000000000,0000 | Eval loss: Infinity | Eval accuracy: 0,1006
Epoch: 2 | Batch: 600 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 800 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 1000 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 1200 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 1400 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 1600 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 2 | Batch: 1800 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 3 | Batch: 0 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 3 | Batch: 200 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 3 | Batch: 400 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980
Epoch: 3 | Batch: 600 | Training loss: NaN | Eval loss: NaN | Eval accuracy: 0,0980Any suggestion on changes I try to get this to work? TIA. |
Beta Was this translation helpful? Give feedback.
Answered by
sbrunk
Jul 21, 2023
Replies: 3 comments 5 replies
-
|
Quick note: I updated my Linux OS CUDA packages to version 12.2 so no warning message appears now. However, it still does not converge. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Just for comparison because this is a very small example. Can you try to run on the CPU? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply. The initial experiments were on CPU and failed. I redid the following tests: Libraries: ivy"org.bytedeco:pytorch:$pytorchVersion-${javaCppVersion};classifier=${javaCPPPlatform()}",
ivy"org.bytedeco:pytorch-platform:$pytorchVersion-${javaCppVersion}",and get: $ ./mill examples.runMain LeNetApp
[114/114] examples.runMain
Epoch: 1 | Batch: 0 | Training loss: 2,2991 | Eval loss: 2,3013 | Eval accuracy: 0,1135
Epoch: 1 | Batch: 200 | Training loss: 0,8807 | Eval loss: 0,7830 | Eval accuracy: 0,7643
Epoch: 1 | Batch: 400 | Training loss: 0,6034 | Eval loss: 0,7869 | Eval accuracy: 0,7779
Epoch: 1 | Batch: 600 | Training loss: 0,7957 | Eval loss: 0,7627 | Eval accuracy: 0,7339
Epoch: 1 | Batch: 800 | Training loss: 0,6067 | Eval loss: 76667992,0000 | Eval accuracy: 0,8280
Epoch: 1 | Batch: 1000 | Training loss: 11240,5742 | Eval loss: 10109,1914 | Eval accuracy: 0,0958
Epoch: 1 | Batch: 1200 | Training loss: 607367028801536,0000 | Eval loss: 948949468839936,0000 | Eval accuracy: 0,0924With this setup: ivy"org.bytedeco:pytorch:$pytorchVersion-${javaCppVersion};classifier=${javaCPPPlatform()}",
ivy"org.bytedeco:pytorch-platform:$pytorchVersion-${javaCppVersion}",
// Additional dependencies to use bundled full version of MKL
ivy"org.bytedeco:mkl-platform-redist:$mklVersion-${javaCppVersion}",I get a core dump: [114/114] examples.runMain
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f771d1e2d3f, pid=18127, tid=18128
#
# JRE version: OpenJDK Runtime Environment (17.0.7+7) (build 17.0.7+7-Ubuntu-0ubuntu122.04.2)
# Java VM: OpenJDK 64-Bit Server VM (17.0.7+7-Ubuntu-0ubuntu122.04.2, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C [libjnitorch.so+0xfe2d3f] Java_org_bytedeco_pytorch_TensorVector_put__JLorg_bytedeco_pytorch_Tensor_2+0xcf
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /mnt/ssd2/hmf/VSCodeProjects/storch/core.18127)
#
# An error report file with more information is saved as:
# /mnt/ssd2/hmf/VSCodeProjects/storch/hs_err_pid18127.log
#
# If you would like to submit a bug report, please visit:
# Unknown
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#Libs seem to be ok, unless I misunderstood and need to install at the OS level: Finally GPU: ivy"org.bytedeco:pytorch:$pytorchVersion-${javaCppVersion};classifier=${javaCPPPlatform()}",
ivy"org.bytedeco:pytorch-platform:$pytorchVersion-${javaCppVersion}",
// Additional dependencies required to use CUDA, cuDNN, and NCCL
ivy"org.bytedeco:pytorch-platform-gpu:$pytorchVersion-${javaCppVersion}",
// Additional dependencies to use bundled CUDA, cuDNN, and NCCL
ivy"org.bytedeco:cuda-platform-redist:$cudaVersion-${javaCppVersion}",And I get: $ ./mill examples.runMain LeNetApp
[114/114] examples.runMain
Epoch: 1 | Batch: 0 | Training loss: 2,3202 | Eval loss: 2,3042 | Eval accuracy: 0,1043
Epoch: 1 | Batch: 200 | Training loss: 0,9289 | Eval loss: 0,7619 | Eval accuracy: 0,7793
Epoch: 1 | Batch: 400 | Training loss: 0,4991 | Eval loss: 0,7101 | Eval accuracy: 0,7751
Epoch: 1 | Batch: 600 | Training loss: 1198,2510 | Eval loss: 1513,2375 | Eval accuracy: 0,1669
Epoch: 1 | Batch: 800 | Training loss: 2856,6753 | Eval loss: 5754607104,0000 | Eval accuracy: 0,1888
Epoch: 1 | Batch: 1000 | Training loss: 31302599063526965000,0000 | Eval loss: 90259833113899170000,0000 | Eval accuracy: 0,0857
Epoch: 1 | Batch: 1200 | Training loss: 306300225231605700000,0000 | Eval loss: 94433007506890420000,0000 | Eval accuracy: 0,1359The tests I initially reported used all libraries, so I assume it was for |
Beta Was this translation helpful? Give feedback.
-
|
I was able to reproduce the diverging loss on Linux. Interestingly, it does not diverge on a MacOS machine. Could you try to reduce the learning rate like so? - val optimizer = Adam(model.parameters, lr = 0.001)
+ val optimizer = Adam(model.parameters, lr = 1e-5))In my case with that learning rate it did converge reliably then on Linux as well, but I'm still wondering why it behaves different here. Note that even with CUDA enabled, the LeNet example currently runs on the CPU. I've fixed that now locally, PR coming soon. |
Beta Was this translation helpful? Give feedback.
Answer selected by
hmf
-
|
Or perhaps try AMSGrad instead, that seems to fix the convergence issue as well, but without having to reduce the learning rate: val optimizer = Adam(model.parameters, lr = 1e-3, amsgrad = true) |
Beta Was this translation helpful? Give feedback.
-
Got the following result. Seem to be working. Is the evaluation accuracy on par with your results? [114/114] examples.runMain
Epoch: 1 | Batch: 0 | Training loss: 2,3218 | Eval loss: 2,3066 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 200 | Training loss: 2,3020 | Eval loss: 2,3032 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 400 | Training loss: 2,3061 | Eval loss: 2,2997 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 600 | Training loss: 2,2962 | Eval loss: 2,2959 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 800 | Training loss: 2,3058 | Eval loss: 2,2920 | Eval accuracy: 0,1009
Epoch: 1 | Batch: 1000 | Training loss: 2,3084 | Eval loss: 2,2875 | Eval accuracy: 0,1257
Epoch: 1 | Batch: 1200 | Training loss: 2,2995 | Eval loss: 2,2824 | Eval accuracy: 0,1662
Epoch: 1 | Batch: 1400 | Training loss: 2,2786 | Eval loss: 2,2766 | Eval accuracy: 0,1816
Epoch: 1 | Batch: 1600 | Training loss: 2,2550 | Eval loss: 2,2703 | Eval accuracy: 0,1891
Epoch: 1 | Batch: 1800 | Training loss: 2,2607 | Eval loss: 2,2630 | Eval accuracy: 0,1915
Epoch: 2 | Batch: 0 | Training loss: 2,2662 | Eval loss: 2,2601 | Eval accuracy: 0,1922
Epoch: 2 | Batch: 200 | Training loss: 2,2415 | Eval loss: 2,2512 | Eval accuracy: 0,2195
Epoch: 2 | Batch: 400 | Training loss: 2,2546 | Eval loss: 2,2411 | Eval accuracy: 0,2698
Epoch: 2 | Batch: 600 | Training loss: 2,2348 | Eval loss: 2,2296 | Eval accuracy: 0,3241
Epoch: 2 | Batch: 800 | Training loss: 2,2041 | Eval loss: 2,2164 | Eval accuracy: 0,3674
Epoch: 2 | Batch: 1000 | Training loss: 2,2042 | Eval loss: 2,2012 | Eval accuracy: 0,4055
Epoch: 2 | Batch: 1200 | Training loss: 2,1626 | Eval loss: 2,1842 | Eval accuracy: 0,4339
Epoch: 2 | Batch: 1400 | Training loss: 2,1643 | Eval loss: 2,1652 | Eval accuracy: 0,4446
Epoch: 2 | Batch: 1600 | Training loss: 2,1702 | Eval loss: 2,1437 | Eval accuracy: 0,4515
Epoch: 2 | Batch: 1800 | Training loss: 2,0946 | Eval loss: 2,1201 | Eval accuracy: 0,4648
Epoch: 3 | Batch: 0 | Training loss: 2,1093 | Eval loss: 2,1104 | Eval accuracy: 0,4737
Epoch: 3 | Batch: 200 | Training loss: 2,1075 | Eval loss: 2,0827 | Eval accuracy: 0,4939
Epoch: 3 | Batch: 400 | Training loss: 2,0267 | Eval loss: 2,0521 | Eval accuracy: 0,5141
Epoch: 3 | Batch: 600 | Training loss: 2,0420 | Eval loss: 2,0185 | Eval accuracy: 0,5325
Epoch: 3 | Batch: 800 | Training loss: 2,0227 | Eval loss: 1,9819 | Eval accuracy: 0,5486
Epoch: 3 | Batch: 1000 | Training loss: 1,8107 | Eval loss: 1,9419 | Eval accuracy: 0,5571
Epoch: 3 | Batch: 1200 | Training loss: 2,0021 | Eval loss: 1,8979 | Eval accuracy: 0,5690
Epoch: 3 | Batch: 1400 | Training loss: 1,8006 | Eval loss: 1,8506 | Eval accuracy: 0,5808
Epoch: 3 | Batch: 1600 | Training loss: 1,8354 | Eval loss: 1,7988 | Eval accuracy: 0,5964
Epoch: 3 | Batch: 1800 | Training loss: 1,7597 | Eval loss: 1,7438 | Eval accuracy: 0,6145
Epoch: 4 | Batch: 0 | Training loss: 1,6578 | Eval loss: 1,7229 | Eval accuracy: 0,6216
Epoch: 4 | Batch: 200 | Training loss: 1,6703 | Eval loss: 1,6633 | Eval accuracy: 0,6422
Epoch: 4 | Batch: 400 | Training loss: 1,6405 | Eval loss: 1,6010 | Eval accuracy: 0,6685
Epoch: 4 | Batch: 600 | Training loss: 1,4752 | Eval loss: 1,5360 | Eval accuracy: 0,6874
Epoch: 4 | Batch: 800 | Training loss: 1,5053 | Eval loss: 1,4708 | Eval accuracy: 0,6931
Epoch: 4 | Batch: 1000 | Training loss: 1,4369 | Eval loss: 1,4053 | Eval accuracy: 0,6963
Epoch: 4 | Batch: 1200 | Training loss: 1,4713 | Eval loss: 1,3386 | Eval accuracy: 0,7060
Epoch: 4 | Batch: 1400 | Training loss: 1,4172 | Eval loss: 1,2728 | Eval accuracy: 0,7191
Epoch: 4 | Batch: 1600 | Training loss: 1,0480 | Eval loss: 1,2089 | Eval accuracy: 0,7330
Epoch: 4 | Batch: 1800 | Training loss: 0,9794 | Eval loss: 1,1451 | Eval accuracy: 0,7445
Epoch: 5 | Batch: 0 | Training loss: 1,1458 | Eval loss: 1,1213 | Eval accuracy: 0,7502
Epoch: 5 | Batch: 200 | Training loss: 0,9995 | Eval loss: 1,0623 | Eval accuracy: 0,7555
Epoch: 5 | Batch: 400 | Training loss: 1,2872 | Eval loss: 1,0097 | Eval accuracy: 0,7600
Epoch: 5 | Batch: 600 | Training loss: 1,1233 | Eval loss: 0,9594 | Eval accuracy: 0,7621
Epoch: 5 | Batch: 800 | Training loss: 0,9590 | Eval loss: 0,9145 | Eval accuracy: 0,7651
Epoch: 5 | Batch: 1000 | Training loss: 0,8528 | Eval loss: 0,8743 | Eval accuracy: 0,7713
Epoch: 5 | Batch: 1200 | Training loss: 0,9152 | Eval loss: 0,8302 | Eval accuracy: 0,7811
Epoch: 5 | Batch: 1400 | Training loss: 0,8444 | Eval loss: 0,7945 | Eval accuracy: 0,7849
Epoch: 5 | Batch: 1600 | Training loss: 0,7744 | Eval loss: 0,7660 | Eval accuracy: 0,7859
Epoch: 5 | Batch: 1800 | Training loss: 0,8351 | Eval loss: 0,7372 | Eval accuracy: 0,7932
Strange indeed. Issue with the native libraries?
Nice. I can time this with the new version to see the difference. Thank you. |
Beta Was this translation helpful? Give feedback.
-
Much better results. Here is what I got. [114/114] examples.runMain
Epoch: 1 | Batch: 0 | Training loss: 2,2900 | Eval loss: 2,3009 | Eval accuracy: 0,1135
Epoch: 1 | Batch: 200 | Training loss: 0,6541 | Eval loss: 0,7396 | Eval accuracy: 0,7648
Epoch: 1 | Batch: 400 | Training loss: 0,3216 | Eval loss: 0,4579 | Eval accuracy: 0,8716
Epoch: 1 | Batch: 600 | Training loss: 0,4101 | Eval loss: 0,3639 | Eval accuracy: 0,8919
Epoch: 1 | Batch: 800 | Training loss: 0,5692 | Eval loss: 0,2988 | Eval accuracy: 0,9087
Epoch: 1 | Batch: 1000 | Training loss: 0,3532 | Eval loss: 0,2647 | Eval accuracy: 0,9182
Epoch: 1 | Batch: 1200 | Training loss: 0,3281 | Eval loss: 0,2149 | Eval accuracy: 0,9352
Epoch: 1 | Batch: 1400 | Training loss: 0,1929 | Eval loss: 0,1813 | Eval accuracy: 0,9446
Epoch: 1 | Batch: 1600 | Training loss: 0,2118 | Eval loss: 0,1639 | Eval accuracy: 0,9493
Epoch: 1 | Batch: 1800 | Training loss: 0,1123 | Eval loss: 0,1424 | Eval accuracy: 0,9552
Epoch: 2 | Batch: 0 | Training loss: 0,1060 | Eval loss: 0,1429 | Eval accuracy: 0,9543
Epoch: 2 | Batch: 200 | Training loss: 0,1787 | Eval loss: 0,1418 | Eval accuracy: 0,9551
Epoch: 2 | Batch: 400 | Training loss: 0,1474 | Eval loss: 0,1350 | Eval accuracy: 0,9582
Epoch: 2 | Batch: 600 | Training loss: 0,1649 | Eval loss: 0,1276 | Eval accuracy: 0,9609
Epoch: 2 | Batch: 800 | Training loss: 0,0265 | Eval loss: 0,1179 | Eval accuracy: 0,9620
Epoch: 2 | Batch: 1000 | Training loss: 0,1586 | Eval loss: 0,1059 | Eval accuracy: 0,9660
Epoch: 2 | Batch: 1200 | Training loss: 0,3089 | Eval loss: 0,0929 | Eval accuracy: 0,9705
Epoch: 2 | Batch: 1400 | Training loss: 0,0476 | Eval loss: 0,0866 | Eval accuracy: 0,9722
Epoch: 2 | Batch: 1600 | Training loss: 0,1712 | Eval loss: 0,0849 | Eval accuracy: 0,9725
Epoch: 2 | Batch: 1800 | Training loss: 0,0953 | Eval loss: 0,0797 | Eval accuracy: 0,9745
Epoch: 3 | Batch: 0 | Training loss: 0,1474 | Eval loss: 0,0752 | Eval accuracy: 0,9771
Epoch: 3 | Batch: 200 | Training loss: 0,2855 | Eval loss: 0,0782 | Eval accuracy: 0,9753
Epoch: 3 | Batch: 400 | Training loss: 0,0772 | Eval loss: 0,0849 | Eval accuracy: 0,9728
Epoch: 3 | Batch: 600 | Training loss: 0,0579 | Eval loss: 0,0683 | Eval accuracy: 0,9789
Epoch: 3 | Batch: 800 | Training loss: 0,1340 | Eval loss: 0,0673 | Eval accuracy: 0,9797
Epoch: 3 | Batch: 1000 | Training loss: 0,0084 | Eval loss: 0,0690 | Eval accuracy: 0,9774
Epoch: 3 | Batch: 1200 | Training loss: 0,0163 | Eval loss: 0,0678 | Eval accuracy: 0,9776
Epoch: 3 | Batch: 1400 | Training loss: 0,0359 | Eval loss: 0,0692 | Eval accuracy: 0,9779
Epoch: 3 | Batch: 1600 | Training loss: 0,0136 | Eval loss: 0,0596 | Eval accuracy: 0,9799
Epoch: 3 | Batch: 1800 | Training loss: 0,0185 | Eval loss: 0,0640 | Eval accuracy: 0,9799
Epoch: 4 | Batch: 0 | Training loss: 0,0308 | Eval loss: 0,0602 | Eval accuracy: 0,9808
Epoch: 4 | Batch: 200 | Training loss: 0,1141 | Eval loss: 0,0620 | Eval accuracy: 0,9800
Epoch: 4 | Batch: 400 | Training loss: 0,0544 | Eval loss: 0,0553 | Eval accuracy: 0,9823
Epoch: 4 | Batch: 600 | Training loss: 0,0032 | Eval loss: 0,0587 | Eval accuracy: 0,9809
Epoch: 4 | Batch: 800 | Training loss: 0,0361 | Eval loss: 0,0553 | Eval accuracy: 0,9820
Epoch: 4 | Batch: 1000 | Training loss: 0,0353 | Eval loss: 0,0542 | Eval accuracy: 0,9833
Epoch: 4 | Batch: 1200 | Training loss: 0,2258 | Eval loss: 0,0511 | Eval accuracy: 0,9832
Epoch: 4 | Batch: 1400 | Training loss: 0,0433 | Eval loss: 0,0482 | Eval accuracy: 0,9839
Epoch: 4 | Batch: 1600 | Training loss: 0,0448 | Eval loss: 0,0502 | Eval accuracy: 0,9829
Epoch: 4 | Batch: 1800 | Training loss: 0,0116 | Eval loss: 0,0504 | Eval accuracy: 0,9836
Epoch: 5 | Batch: 0 | Training loss: 0,0858 | Eval loss: 0,0516 | Eval accuracy: 0,9835
Epoch: 5 | Batch: 200 | Training loss: 0,0135 | Eval loss: 0,0483 | Eval accuracy: 0,9840
Epoch: 5 | Batch: 400 | Training loss: 0,0606 | Eval loss: 0,0502 | Eval accuracy: 0,9840
Epoch: 5 | Batch: 600 | Training loss: 0,0653 | Eval loss: 0,0472 | Eval accuracy: 0,9854
Epoch: 5 | Batch: 800 | Training loss: 0,0317 | Eval loss: 0,0480 | Eval accuracy: 0,9834
Epoch: 5 | Batch: 1000 | Training loss: 0,2204 | Eval loss: 0,0489 | Eval accuracy: 0,9839
Epoch: 5 | Batch: 1200 | Training loss: 0,2653 | Eval loss: 0,0508 | Eval accuracy: 0,9842
Epoch: 5 | Batch: 1400 | Training loss: 0,0172 | Eval loss: 0,0423 | Eval accuracy: 0,9856
Epoch: 5 | Batch: 1600 | Training loss: 0,0535 | Eval loss: 0,0433 | Eval accuracy: 0,9845
Epoch: 5 | Batch: 1800 | Training loss: 0,1550 | Eval loss: 0,0433 | Eval accuracy: 0,9863 |
Beta Was this translation helpful? Give feedback.
0 replies
-
It is, yes, and with amsgrad enabled it is also around 0.98 on my machine. Here's the PR that should enable the LeNet example to run on the GPU: #43 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was able to reproduce the diverging loss on Linux. Interestingly, it does not diverge on a MacOS machine.
Could you try to reduce the learning rate like so?
In my case with that learning rate it did converge reliably then on Linux as well, but I'm still wondering why it behaves different here.
Note that even with CUDA enabled, the LeNet example currently runs on the CPU. I've fixed that now locally, PR coming soon.