-
Notifications
You must be signed in to change notification settings - Fork 13
/
log_x3d_kinetics_multigrid.txt
805 lines (800 loc) · 56.7 KB
/
log_x3d_kinetics_multigrid.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
python train_x3d_kinetics_multigrid.py -gpu 4,5,6,7
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/train_frames/labeldata_80.npy exists
0.8.0a0+6e10e3f 10010
1.7.0a0+e5f6e5a 10.1
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/valid_frames/labeldata_80.npy exists
train 215901 val 17227
Total iterations: 206160 Total steps: 206160
datasets created
model loaded
LR:0.200000
Step 0 Epoch 0
----------
update:
train
1/1718 [..............................] - 18.1s ***** LR 1.6 Frames 20/10 BS (2048,1024) W/H (111,158) BN_splits 32 long_ind 0 *****
859/1718 [==============>...............] - 1357.3s Epoch:1 train steps: 859 Loc Loss: 0.0000 Cls Loss: 5.3228 Tot Loss: 5.3228 Acc: 0.0373
1718/1718 [==============================] - 2693.4s
Epoch:1 train steps: 1718 Loc Loss: 0.0000 Cls Loss: 4.4197 Tot Loss: 4.4197 Acc: 0.1185
1720/1718 [==============================] - 2699.3supdate:
train
858/1718 [=============>................] - 1360.1s Epoch:2 train steps: 2577 Loc Loss: 0.0000 Cls Loss: 4.0100 Tot Loss: 4.0100 Acc: 0.1704
1717/1718 [============================>.] - 2629.5s Epoch:2 train steps: 3436 Loc Loss: 0.0000 Cls Loss: 3.7626 Tot Loss: 3.7626 Acc: 0.2073
1718/1718 [==============================] - 2630.7s
1720/1718 [==============================] - 2633.1supdate:
train
857/1718 [=============>................] - 1211.3s Epoch:3 train steps: 4295 Loc Loss: 0.0000 Cls Loss: 3.5915 Tot Loss: 3.5915 Acc: 0.2320
1716/1718 [============================>.] - 2395.3s Epoch:3 train steps: 5154 Loc Loss: 0.0000 Cls Loss: 3.4967 Tot Loss: 3.4967 Acc: 0.2492
1718/1718 [==============================] - 2397.8s
1720/1718 [==============================] - 2400.6supdate:
train
856/1718 [=============>................] - 1477.7s Epoch:4 train steps: 6013 Loc Loss: 0.0000 Cls Loss: 3.4304 Tot Loss: 3.4304 Acc: 0.2583
1715/1718 [============================>.] - 3359.5s Epoch:4 train steps: 6872 Loc Loss: 0.0000 Cls Loss: 3.4175 Tot Loss: 3.4175 Acc: 0.2624
1718/1718 [==============================] - 3362.9s
1720/1718 [==============================] - 3366.1supdate:
val
135/136 [============================>.] - 343.8s Epoch:4 val Loc Loss: 0.0000 Cls Loss: 5.1318 Tot Loss: 5.1318 Acc: 0.0701
Step 6876 Epoch 4
----------
update:
train
855/1718 [=============>................] - 1415.5s Epoch:5 train steps: 7731 Loc Loss: 0.0000 Cls Loss: 3.3922 Tot Loss: 3.3922 Acc: 0.2647
1714/1718 [============================>.] - 2558.4s Epoch:5 train steps: 8590 Loc Loss: 0.0000 Cls Loss: 3.4043 Tot Loss: 3.4043 Acc: 0.2660
1718/1718 [==============================] - 2563.3s
1720/1718 [==============================] - 2568.7supdate:
train
854/1718 [=============>................] - 1142.5s Epoch:6 train steps: 9449 Loc Loss: 0.0000 Cls Loss: 3.3526 Tot Loss: 3.3526 Acc: 0.2707
1713/1718 [============================>.] - 2311.2s Epoch:6 train steps: 10308 Loc Loss: 0.0000 Cls Loss: 3.3461 Tot Loss: 3.3461 Acc: 0.2752
1718/1718 [==============================] - 2317.0s
1720/1718 [==============================] - 2323.1supdate:
train
853/1718 [=============>................] - 1179.1s Epoch:7 train steps: 11167 Loc Loss: 0.0000 Cls Loss: 3.3032 Tot Loss: 3.3032 Acc: 0.2781
1712/1718 [============================>.] - 2339.6s Epoch:7 train steps: 12026 Loc Loss: 0.0000 Cls Loss: 3.3098 Tot Loss: 3.3098 Acc: 0.2821
1718/1718 [==============================] - 2347.6s
1720/1718 [==============================] - 2350.0supdate:
train
852/1718 [=============>................] - 1173.7s Epoch:8 train steps: 12885 Loc Loss: 0.0000 Cls Loss: 3.2710 Tot Loss: 3.2710 Acc: 0.2833
1711/1718 [============================>.] - 2332.1s Epoch:8 train steps: 13744 Loc Loss: 0.0000 Cls Loss: 3.2902 Tot Loss: 3.2902 Acc: 0.2844
1718/1718 [==============================] - 2341.2s
1720/1718 [==============================] - 2343.6supdate:
val
135/136 [============================>.] - 233.1s Epoch:8 val Loc Loss: 0.0000 Cls Loss: 5.4239 Tot Loss: 5.4239 Acc: 0.0501
Step 13752 Epoch 8
----------
update:
train
851/1718 [=============>................] - 1158.8s Epoch:9 train steps: 14603 Loc Loss: 0.0000 Cls Loss: 3.2507 Tot Loss: 3.2507 Acc: 0.2864
1710/1718 [============================>.] - 2284.2s Epoch:9 train steps: 15462 Loc Loss: 0.0000 Cls Loss: 3.2704 Tot Loss: 3.2704 Acc: 0.2883
1718/1718 [==============================] - 2294.3s
1720/1718 [==============================] - 2296.7supdate:
train
850/1718 [=============>................] - 1143.7s Epoch:10 train steps: 16321 Loc Loss: 0.0000 Cls Loss: 3.2301 Tot Loss: 3.2301 Acc: 0.2889
1709/1718 [============================>.] - 2311.8s Epoch:10 train steps: 17180 Loc Loss: 0.0000 Cls Loss: 3.2570 Tot Loss: 3.2570 Acc: 0.2902
1718/1718 [==============================] - 2322.9s
1720/1718 [==============================] - 2325.3supdate:
train
849/1718 [=============>................] - 1162.8s Epoch:11 train steps: 18039 Loc Loss: 0.0000 Cls Loss: 3.2154 Tot Loss: 3.2154 Acc: 0.2913
1708/1718 [============================>.] - 2307.8s Epoch:11 train steps: 18898 Loc Loss: 0.0000 Cls Loss: 3.2459 Tot Loss: 3.2459 Acc: 0.2924
1718/1718 [==============================] - 2320.6s
1720/1718 [==============================] - 2323.1supdate:
train
848/1718 [=============>................] - 1151.4s Epoch:12 train steps: 19757 Loc Loss: 0.0000 Cls Loss: 3.2015 Tot Loss: 3.2015 Acc: 0.2929
1444/1718 [========================>.....] - 1941.5s ***** LR 0.8 Frames 40/10 BS (1024,512) W/H (111,158) BN_splits 16 long_ind 1 *****
1707/1718 [============================>.] - 2285.5s Epoch:12 train steps: 20616 Loc Loss: 0.0000 Cls Loss: 3.2031 Tot Loss: 3.2031 Acc: 0.2959
1718/1718 [==============================] - 2303.7s
1720/1718 [==============================] - 2306.2supdate:
val
135/136 [============================>.] - 223.9s Epoch:12 val Loc Loss: 0.0000 Cls Loss: 3.8219 Tot Loss: 3.8219 Acc: 0.2374
Step 20628 Epoch 12
----------
update:
train
847/1718 [=============>................] - 1130.8s Epoch:13 train steps: 21475 Loc Loss: 0.0000 Cls Loss: 3.0094 Tot Loss: 3.0094 Acc: 0.3213
1706/1718 [============================>.] - 2379.5s Epoch:13 train steps: 22334 Loc Loss: 0.0000 Cls Loss: 3.0311 Tot Loss: 3.0311 Acc: 0.3243
1718/1718 [==============================] - 2399.8s
1720/1718 [==============================] - 2402.2supdate:
train
846/1718 [=============>................] - 1176.7s Epoch:14 train steps: 23193 Loc Loss: 0.0000 Cls Loss: 2.9749 Tot Loss: 2.9749 Acc: 0.3266
1705/1718 [============================>.] - 2309.8s Epoch:14 train steps: 24052 Loc Loss: 0.0000 Cls Loss: 3.0092 Tot Loss: 3.0092 Acc: 0.3286
1718/1718 [==============================] - 2329.4s
1720/1718 [==============================] - 2332.6supdate:
train
845/1718 [=============>................] - 1242.8s Epoch:15 train steps: 24911 Loc Loss: 0.0000 Cls Loss: 2.9556 Tot Loss: 2.9556 Acc: 0.3296
1704/1718 [============================>.] - 2486.3s Epoch:15 train steps: 25770 Loc Loss: 0.0000 Cls Loss: 3.0015 Tot Loss: 3.0015 Acc: 0.3292
1718/1718 [==============================] - 2507.6s
1720/1718 [==============================] - 2509.9supdate:
train
844/1718 [=============>................] - 1131.0s Epoch:16 train steps: 26629 Loc Loss: 0.0000 Cls Loss: 2.9427 Tot Loss: 2.9427 Acc: 0.3306
1703/1718 [============================>.] - 2271.7s Epoch:16 train steps: 27488 Loc Loss: 0.0000 Cls Loss: 2.9880 Tot Loss: 2.9880 Acc: 0.3319
1718/1718 [==============================] - 2290.1s
1720/1718 [==============================] - 2292.5supdate:
val
135/136 [============================>.] - 229.6s Epoch:16 val Loc Loss: 0.0000 Cls Loss: 3.6513 Tot Loss: 3.6513 Acc: 0.2480
Step 27504 Epoch 16
----------
update:
train
843/1718 [=============>................] - 1132.6s Epoch:17 train steps: 28347 Loc Loss: 0.0000 Cls Loss: 2.9352 Tot Loss: 2.9352 Acc: 0.3316
1702/1718 [============================>.] - 2223.6s Epoch:17 train steps: 29206 Loc Loss: 0.0000 Cls Loss: 2.9800 Tot Loss: 2.9800 Acc: 0.3339
1718/1718 [==============================] - 2243.9s
1720/1718 [==============================] - 2246.3supdate:
train
842/1718 [=============>................] - 1096.5s Epoch:18 train steps: 30065 Loc Loss: 0.0000 Cls Loss: 2.9208 Tot Loss: 2.9208 Acc: 0.3334
1701/1718 [============================>.] - 2218.4s Epoch:18 train steps: 30924 Loc Loss: 0.0000 Cls Loss: 2.9799 Tot Loss: 2.9799 Acc: 0.3339
1718/1718 [==============================] - 2243.0s
1720/1718 [==============================] - 2245.3supdate:
train
841/1718 [=============>................] - 1121.2s Epoch:19 train steps: 31783 Loc Loss: 0.0000 Cls Loss: 2.9115 Tot Loss: 2.9115 Acc: 0.3345
1700/1718 [============================>.] - 2243.3s Epoch:19 train steps: 32642 Loc Loss: 0.0000 Cls Loss: 2.9682 Tot Loss: 2.9682 Acc: 0.3355
1718/1718 [==============================] - 2268.3s
1720/1718 [==============================] - 2270.7supdate:
train
840/1718 [=============>................] - 1118.4s Epoch:20 train steps: 33501 Loc Loss: 0.0000 Cls Loss: 2.9036 Tot Loss: 2.9036 Acc: 0.3352
1699/1718 [============================>.] - 2254.5s Epoch:20 train steps: 34360 Loc Loss: 0.0000 Cls Loss: 2.9687 Tot Loss: 2.9687 Acc: 0.3356
1718/1718 [==============================] - 2278.1s
1720/1718 [==============================] - 2280.5supdate:
val
135/136 [============================>.] - 236.5s Epoch:20 val Loc Loss: 0.0000 Cls Loss: 3.6407 Tot Loss: 3.6407 Acc: 0.2444
Step 34380 Epoch 20
----------
update:
train
839/1718 [=============>................] - 1115.7s Epoch:21 train steps: 35219 Loc Loss: 0.0000 Cls Loss: 2.8998 Tot Loss: 2.8998 Acc: 0.3363
1698/1718 [============================>.] - 2217.3s Epoch:21 train steps: 36078 Loc Loss: 0.0000 Cls Loss: 2.9658 Tot Loss: 2.9658 Acc: 0.3364
1718/1718 [==============================] - 2242.6s
1720/1718 [==============================] - 2245.0supdate:
train
838/1718 [=============>................] - 1084.9s Epoch:22 train steps: 36937 Loc Loss: 0.0000 Cls Loss: 2.8905 Tot Loss: 2.8905 Acc: 0.3370
1697/1718 [============================>.] - 2208.0s Epoch:22 train steps: 37796 Loc Loss: 0.0000 Cls Loss: 2.9594 Tot Loss: 2.9594 Acc: 0.3366
1718/1718 [==============================] - 2233.9s
1720/1718 [==============================] - 2237.0supdate:
train
837/1718 [=============>................] - 1104.7s Epoch:23 train steps: 38655 Loc Loss: 0.0000 Cls Loss: 2.8839 Tot Loss: 2.8839 Acc: 0.3375
1696/1718 [============================>.] - 2222.3s Epoch:23 train steps: 39514 Loc Loss: 0.0000 Cls Loss: 2.9582 Tot Loss: 2.9582 Acc: 0.3381
1718/1718 [==============================] - 2249.3s
1720/1718 [==============================] - 2251.7supdate:
train
836/1718 [=============>................] - 1109.7s Epoch:24 train steps: 40373 Loc Loss: 0.0000 Cls Loss: 2.8765 Tot Loss: 2.8765 Acc: 0.3382
1144/1718 [==================>...........] - 1509.2s ***** LR 0.4 Frames 40/10 BS (1024,512,256) W/H (112,158,224) BN_splits 8 long_ind 2 *****
1695/1718 [============================>.] - 2148.3s Epoch:24 train steps: 41232 Loc Loss: 0.0000 Cls Loss: 2.8532 Tot Loss: 2.8532 Acc: 0.3569
1718/1718 [==============================] - 2174.5s
1720/1718 [==============================] - 2176.4supdate:
val
135/136 [============================>.] - 231.9s Epoch:24 val Loc Loss: 0.0000 Cls Loss: 2.9533 Tot Loss: 2.9533 Acc: 0.3568
Step 41256 Epoch 24
----------
update:
train
835/1718 [=============>................] - 991.8s Epoch:25 train steps: 42091 Loc Loss: 0.0000 Cls Loss: 2.7018 Tot Loss: 2.7018 Acc: 0.3696
1694/1718 [============================>.] - 2016.8s Epoch:25 train steps: 42950 Loc Loss: 0.0000 Cls Loss: 2.7829 Tot Loss: 2.7829 Acc: 0.3701
1718/1718 [==============================] - 2045.7s
1720/1718 [==============================] - 2047.7supdate:
train
834/1718 [=============>................] - 984.1s Epoch:26 train steps: 43809 Loc Loss: 0.0000 Cls Loss: 2.7009 Tot Loss: 2.7009 Acc: 0.3700
1693/1718 [============================>.] - 1993.2s Epoch:26 train steps: 44668 Loc Loss: 0.0000 Cls Loss: 2.7806 Tot Loss: 2.7806 Acc: 0.3703
1718/1718 [==============================] - 2021.9s
1720/1718 [==============================] - 2024.5supdate:
train
833/1718 [=============>................] - 995.6s Epoch:27 train steps: 45527 Loc Loss: 0.0000 Cls Loss: 2.6946 Tot Loss: 2.6946 Acc: 0.3699
1692/1718 [============================>.] - 2012.1s Epoch:27 train steps: 46386 Loc Loss: 0.0000 Cls Loss: 2.7800 Tot Loss: 2.7800 Acc: 0.3698
1718/1718 [==============================] - 2042.0s
1720/1718 [==============================] - 2044.0supdate:
train
832/1718 [=============>................] - 996.5s Epoch:28 train steps: 47245 Loc Loss: 0.0000 Cls Loss: 2.6890 Tot Loss: 2.6890 Acc: 0.3695
1691/1718 [============================>.] - 2020.3s Epoch:28 train steps: 48104 Loc Loss: 0.0000 Cls Loss: 2.7793 Tot Loss: 2.7793 Acc: 0.3698
1718/1718 [==============================] - 2051.3s
1720/1718 [==============================] - 2053.4supdate:
val
135/136 [============================>.] - 239.0s Epoch:28 val Loc Loss: 0.0000 Cls Loss: 2.8724 Tot Loss: 2.8724 Acc: 0.3728
Step 48132 Epoch 28
----------
update:
train
831/1718 [=============>................] - 1007.6s Epoch:29 train steps: 48963 Loc Loss: 0.0000 Cls Loss: 2.6900 Tot Loss: 2.6900 Acc: 0.3696
1690/1718 [============================>.] - 2022.2s Epoch:29 train steps: 49822 Loc Loss: 0.0000 Cls Loss: 2.7695 Tot Loss: 2.7695 Acc: 0.3708
1718/1718 [==============================] - 2054.4s
1720/1718 [==============================] - 2057.2supdate:
train
830/1718 [=============>................] - 989.7s Epoch:30 train steps: 50681 Loc Loss: 0.0000 Cls Loss: 2.6752 Tot Loss: 2.6752 Acc: 0.3715
1689/1718 [============================>.] - 1980.1s Epoch:30 train steps: 51540 Loc Loss: 0.0000 Cls Loss: 2.7775 Tot Loss: 2.7775 Acc: 0.3699
1718/1718 [==============================] - 2014.1s
1720/1718 [==============================] - 2016.1supdate:
train
829/1718 [=============>................] - 971.0s Epoch:31 train steps: 52399 Loc Loss: 0.0000 Cls Loss: 2.6709 Tot Loss: 2.6709 Acc: 0.3715
1688/1718 [============================>.] - 1961.6s Epoch:31 train steps: 53258 Loc Loss: 0.0000 Cls Loss: 2.7690 Tot Loss: 2.7690 Acc: 0.3713
1718/1718 [==============================] - 1996.7s
1720/1718 [==============================] - 1998.7supdate:
train
828/1718 [=============>................] - 972.7s Epoch:32 train steps: 54117 Loc Loss: 0.0000 Cls Loss: 2.6654 Tot Loss: 2.6654 Acc: 0.3724
1687/1718 [============================>.] - 1963.3s Epoch:32 train steps: 54976 Loc Loss: 0.0000 Cls Loss: 2.7641 Tot Loss: 2.7641 Acc: 0.3720
1718/1718 [==============================] - 1998.6s
1720/1718 [==============================] - 2001.7supdate:
val
135/136 [============================>.] - 230.3s Epoch:32 val Loc Loss: 0.0000 Cls Loss: 2.8942 Tot Loss: 2.8942 Acc: 0.3683
Step 55008 Epoch 32
----------
update:
train
827/1718 [=============>................] - 974.0s Epoch:33 train steps: 55835 Loc Loss: 0.0000 Cls Loss: 2.6589 Tot Loss: 2.6589 Acc: 0.3727
1686/1718 [============================>.] - 1961.0s Epoch:33 train steps: 56694 Loc Loss: 0.0000 Cls Loss: 2.7624 Tot Loss: 2.7624 Acc: 0.3722
1718/1718 [==============================] - 1998.0s
1720/1718 [==============================] - 2000.1supdate:
train
826/1718 [=============>................] - 968.5s Epoch:34 train steps: 57553 Loc Loss: 0.0000 Cls Loss: 2.6530 Tot Loss: 2.6530 Acc: 0.3735
1685/1718 [============================>.] - 1957.8s Epoch:34 train steps: 58412 Loc Loss: 0.0000 Cls Loss: 2.7581 Tot Loss: 2.7581 Acc: 0.3726
1718/1718 [==============================] - 1995.7s
1720/1718 [==============================] - 1997.8supdate:
train
825/1718 [=============>................] - 965.1s Epoch:35 train steps: 59271 Loc Loss: 0.0000 Cls Loss: 2.6484 Tot Loss: 2.6484 Acc: 0.3737
1684/1718 [============================>.] - 1955.3s Epoch:35 train steps: 60130 Loc Loss: 0.0000 Cls Loss: 2.7563 Tot Loss: 2.7563 Acc: 0.3734
1718/1718 [==============================] - 1994.6s
1720/1718 [==============================] - 1997.6supdate:
train
824/1718 [=============>................] - 965.6s Epoch:36 train steps: 60989 Loc Loss: 0.0000 Cls Loss: 2.6464 Tot Loss: 2.6464 Acc: 0.3726
844/1718 [=============>................] - 988.3s ***** LR 0.2 Frames 80/10 BS (512,256,128) W/H (112,158,224) BN_splits 4 long_ind 3 *****
1683/1718 [============================>.] - 2006.1s Epoch:36 train steps: 61848 Loc Loss: 0.0000 Cls Loss: 2.5861 Tot Loss: 2.5861 Acc: 0.4018
1718/1718 [==============================] - 2052.5s
1720/1718 [==============================] - 2054.5supdate:
val
135/136 [============================>.] - 267.5s Epoch:36 val Loc Loss: 0.0000 Cls Loss: 2.5043 Tot Loss: 2.5043 Acc: 0.4233
Step 61884 Epoch 36
----------
update:
train
823/1718 [=============>................] - 1032.0s Epoch:37 train steps: 62707 Loc Loss: 0.0000 Cls Loss: 2.4549 Tot Loss: 2.4549 Acc: 0.4061
1682/1718 [============================>.] - 2006.2s Epoch:37 train steps: 63566 Loc Loss: 0.0000 Cls Loss: 2.5619 Tot Loss: 2.5619 Acc: 0.4069
1718/1718 [==============================] - 2043.7s
1720/1718 [==============================] - 2045.7supdate:
train
822/1718 [=============>................] - 908.9s Epoch:38 train steps: 64425 Loc Loss: 0.0000 Cls Loss: 2.4463 Tot Loss: 2.4463 Acc: 0.4086
1681/1718 [============================>.] - 1859.4s Epoch:38 train steps: 65284 Loc Loss: 0.0000 Cls Loss: 2.5479 Tot Loss: 2.5479 Acc: 0.4091
1718/1718 [==============================] - 1901.8s
1720/1718 [==============================] - 1903.9supdate:
train
821/1718 [=============>................] - 918.4s Epoch:39 train steps: 66143 Loc Loss: 0.0000 Cls Loss: 2.4425 Tot Loss: 2.4425 Acc: 0.4081
1680/1718 [============================>.] - 1877.9s Epoch:39 train steps: 67002 Loc Loss: 0.0000 Cls Loss: 2.5394 Tot Loss: 2.5394 Acc: 0.4105
1718/1718 [==============================] - 1920.6s
1720/1718 [==============================] - 1922.6supdate:
train
820/1718 [=============>................] - 925.6s Epoch:40 train steps: 67861 Loc Loss: 0.0000 Cls Loss: 2.4256 Tot Loss: 2.4256 Acc: 0.4097
1679/1718 [============================>.] - 1888.2s Epoch:40 train steps: 68720 Loc Loss: 0.0000 Cls Loss: 2.5448 Tot Loss: 2.5448 Acc: 0.4102
1718/1718 [==============================] - 1932.5s
1720/1718 [==============================] - 1934.6supdate:
val
135/136 [============================>.] - 206.9s Epoch:40 val Loc Loss: 0.0000 Cls Loss: 2.3883 Tot Loss: 2.3883 Acc: 0.4450
Step 68760 Epoch 40
----------
update:
train
819/1718 [=============>................] - 928.6s Epoch:41 train steps: 69579 Loc Loss: 0.0000 Cls Loss: 2.4237 Tot Loss: 2.4237 Acc: 0.4092
1678/1718 [============================>.] - 1889.1s Epoch:41 train steps: 70438 Loc Loss: 0.0000 Cls Loss: 2.5413 Tot Loss: 2.5413 Acc: 0.4109
1718/1718 [==============================] - 1933.7s
1720/1718 [==============================] - 1936.4supdate:
train
818/1718 [=============>................] - 938.4s Epoch:42 train steps: 71297 Loc Loss: 0.0000 Cls Loss: 2.4204 Tot Loss: 2.4204 Acc: 0.4114
1677/1718 [============================>.] - 1906.3s Epoch:42 train steps: 72156 Loc Loss: 0.0000 Cls Loss: 2.5449 Tot Loss: 2.5449 Acc: 0.4105
1718/1718 [==============================] - 1951.7s
1720/1718 [==============================] - 1953.7supdate:
train
817/1718 [=============>................] - 919.9s Epoch:43 train steps: 73015 Loc Loss: 0.0000 Cls Loss: 2.4177 Tot Loss: 2.4177 Acc: 0.4112
1676/1718 [============================>.] - 1900.1s Epoch:43 train steps: 73874 Loc Loss: 0.0000 Cls Loss: 2.5290 Tot Loss: 2.5290 Acc: 0.4125
1718/1718 [==============================] - 1949.1s
1720/1718 [==============================] - 1952.1supdate:
train
816/1718 [=============>................] - 914.0s Epoch:44 train steps: 74733 Loc Loss: 0.0000 Cls Loss: 2.4061 Tot Loss: 2.4061 Acc: 0.4120
1675/1718 [============================>.] - 1852.5s Epoch:44 train steps: 75592 Loc Loss: 0.0000 Cls Loss: 2.5300 Tot Loss: 2.5300 Acc: 0.4131
1718/1718 [==============================] - 1899.7s
1720/1718 [==============================] - 1902.4supdate:
val
135/136 [============================>.] - 216.3s Epoch:44 val Loc Loss: 0.0000 Cls Loss: 2.3896 Tot Loss: 2.3896 Acc: 0.4529
Step 75636 Epoch 44
----------
update:
train
815/1718 [=============>................] - 897.0s Epoch:45 train steps: 76451 Loc Loss: 0.0000 Cls Loss: 2.4042 Tot Loss: 2.4042 Acc: 0.4114
1674/1718 [============================>.] - 1838.4s Epoch:45 train steps: 77310 Loc Loss: 0.0000 Cls Loss: 2.5256 Tot Loss: 2.5256 Acc: 0.4131
1718/1718 [==============================] - 1885.6s
1720/1718 [==============================] - 1887.6supdate:
train
814/1718 [=============>................] - 909.1s Epoch:46 train steps: 78169 Loc Loss: 0.0000 Cls Loss: 2.3928 Tot Loss: 2.3928 Acc: 0.4121
1673/1718 [============================>.] - 1856.7s Epoch:46 train steps: 79028 Loc Loss: 0.0000 Cls Loss: 2.5282 Tot Loss: 2.5282 Acc: 0.4136
1718/1718 [==============================] - 1906.0s
1720/1718 [==============================] - 1908.1supdate:
train
813/1718 [=============>................] - 902.9s Epoch:47 train steps: 79887 Loc Loss: 0.0000 Cls Loss: 2.3920 Tot Loss: 2.3920 Acc: 0.4130
1672/1718 [============================>.] - 1837.7s Epoch:47 train steps: 80746 Loc Loss: 0.0000 Cls Loss: 2.5250 Tot Loss: 2.5250 Acc: 0.4135
1718/1718 [==============================] - 1885.8s
1720/1718 [==============================] - 1889.5supdate:
train
545/1718 [========>.....................] - 610.0s ***** LR 1.6 Frames 20/10 BS (2048,1024) W/H (111,158) BN_splits 32 long_ind 0 *****
812/1718 [=============>................] - 944.4s Epoch:48 train steps: 81605 Loc Loss: 0.0000 Cls Loss: 2.6388 Tot Loss: 2.6388 Acc: 0.3175
1671/1718 [============================>.] - 2015.7s Epoch:48 train steps: 82464 Loc Loss: 0.0000 Cls Loss: 3.1986 Tot Loss: 3.1986 Acc: 0.3004
1718/1718 [==============================] - 2073.6s
1720/1718 [==============================] - 2076.7supdate:
val
135/136 [============================>.] - 259.4s Epoch:48 val Loc Loss: 0.0000 Cls Loss: 3.6465 Tot Loss: 3.6465 Acc: 0.2788
Step 82512 Epoch 48
----------
update:
train
811/1718 [=============>................] - 1027.4s Epoch:49 train steps: 83323 Loc Loss: 0.0000 Cls Loss: 2.5380 Tot Loss: 2.5380 Acc: 0.3887
1670/1718 [============================>.] - 2103.9s Epoch:49 train steps: 84182 Loc Loss: 0.0000 Cls Loss: 2.5712 Tot Loss: 2.5712 Acc: 0.4108
1718/1718 [==============================] - 2163.2s
1720/1718 [==============================] - 2165.6supdate:
train
810/1718 [=============>................] - 1077.9s Epoch:50 train steps: 85041 Loc Loss: 0.0000 Cls Loss: 2.3778 Tot Loss: 2.3778 Acc: 0.4200
913/1718 [==============>...............] - 1286.2sERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
python train_x3d_kinetics_multigrid.py -gpu 4,5,6,7
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/train_frames/labeldata_80.npy exists
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/valid_frames/labeldata_80.npy exists
train 215901 val 17227
Total iterations: 206160 Total steps: 206160
datasets created
model loaded
LR:0.200000
Step 84000 Epoch 49
----------
update:
train
1/1718 [..............................] - 22.1s ***** LR 0.16000000000000003 Frames 20/10 BS (2048,1024) W/H (111,158) BN_splits 32 long_ind 0 *****
859/1718 [==============>...............] - 1112.0s Epoch:50 train steps: 84859 Loc Loss: 0.0000 Cls Loss: 2.5299 Tot Loss: 2.5299 Acc: 0.4187
1718/1718 [==============================] - 2192.0s
Epoch:50 train steps: 85718 Loc Loss: 0.0000 Cls Loss: 2.4986 Tot Loss: 2.4986 Acc: 0.4246
1720/1718 [==============================] - 2194.4supdate:
train
858/1718 [=============>................] - 1091.8s Epoch:51 train steps: 86577 Loc Loss: 0.0000 Cls Loss: 2.4774 Tot Loss: 2.4774 Acc: 0.4283
1717/1718 [============================>.] - 2163.1s Epoch:51 train steps: 87436 Loc Loss: 0.0000 Cls Loss: 2.4687 Tot Loss: 2.4687 Acc: 0.4298
1718/1718 [==============================] - 2164.1s
1720/1718 [==============================] - 2166.6supdate:
train
857/1718 [=============>................] - 1091.7s Epoch:52 train steps: 88295 Loc Loss: 0.0000 Cls Loss: 2.4575 Tot Loss: 2.4575 Acc: 0.4311
1716/1718 [============================>.] - 2166.7s Epoch:52 train steps: 89154 Loc Loss: 0.0000 Cls Loss: 2.4563 Tot Loss: 2.4563 Acc: 0.4317
1718/1718 [==============================] - 2169.1s
1720/1718 [==============================] - 2171.5supdate:
train
856/1718 [=============>................] - 1086.2s Epoch:53 train steps: 90013 Loc Loss: 0.0000 Cls Loss: 2.4447 Tot Loss: 2.4447 Acc: 0.4321
1715/1718 [============================>.] - 2153.7s Epoch:53 train steps: 90872 Loc Loss: 0.0000 Cls Loss: 2.4490 Tot Loss: 2.4490 Acc: 0.4329
1718/1718 [==============================] - 2157.2s
1720/1718 [==============================] - 2159.6supdate:
val
135/136 [============================>.] - 214.0s Epoch:53 val Loc Loss: 0.0000 Cls Loss: 4.0196 Tot Loss: 4.0196 Acc: 0.2204
Step 90876 Epoch 53
----------
update:
train
855/1718 [=============>................] - 1081.5s Epoch:54 train steps: 91731 Loc Loss: 0.0000 Cls Loss: 2.4359 Tot Loss: 2.4359 Acc: 0.4335
1714/1718 [============================>.] - 2153.1s Epoch:54 train steps: 92590 Loc Loss: 0.0000 Cls Loss: 2.4454 Tot Loss: 2.4454 Acc: 0.4336
1718/1718 [==============================] - 2157.8s
1720/1718 [==============================] - 2161.1supdate:
train
854/1718 [=============>................] - 1082.8s Epoch:55 train steps: 93449 Loc Loss: 0.0000 Cls Loss: 2.4278 Tot Loss: 2.4278 Acc: 0.4344
1713/1718 [============================>.] - 2155.3s Epoch:55 train steps: 94308 Loc Loss: 0.0000 Cls Loss: 2.4383 Tot Loss: 2.4383 Acc: 0.4350
1718/1718 [==============================] - 2161.3s
1720/1718 [==============================] - 2163.7supdate:
train
853/1718 [=============>................] - 1079.5s Epoch:56 train steps: 95167 Loc Loss: 0.0000 Cls Loss: 2.4183 Tot Loss: 2.4183 Acc: 0.4351
892/1718 [==============>...............] - 1127.6s ***** LR 0.08000000000000002 Frames 40/10 BS (1024,512) W/H (111,158) BN_splits 16 long_ind 1 *****
1712/1718 [============================>.] - 2147.5s Epoch:56 train steps: 96026 Loc Loss: 0.0000 Cls Loss: 2.2739 Tot Loss: 2.2739 Acc: 0.4641
1718/1718 [==============================] - 2154.8s
1720/1718 [==============================] - 2157.3supdate:
train
852/1718 [=============>................] - 1078.3s Epoch:57 train steps: 96885 Loc Loss: 0.0000 Cls Loss: 2.2175 Tot Loss: 2.2175 Acc: 0.4711
1711/1718 [============================>.] - 2145.0s Epoch:57 train steps: 97744 Loc Loss: 0.0000 Cls Loss: 2.2254 Tot Loss: 2.2254 Acc: 0.4735
1718/1718 [==============================] - 2153.3s
1720/1718 [==============================] - 2156.2supdate:
val
135/136 [============================>.] - 213.2s Epoch:57 val Loc Loss: 0.0000 Cls Loss: 2.5305 Tot Loss: 2.5305 Acc: 0.4500
Step 97752 Epoch 57
----------
update:
train
851/1718 [=============>................] - 1076.2s Epoch:58 train steps: 98603 Loc Loss: 0.0000 Cls Loss: 2.2009 Tot Loss: 2.2009 Acc: 0.4748
1710/1718 [============================>.] - 2138.2s Epoch:58 train steps: 99462 Loc Loss: 0.0000 Cls Loss: 2.2177 Tot Loss: 2.2177 Acc: 0.4746
1718/1718 [==============================] - 2148.3s
1720/1718 [==============================] - 2150.7supdate:
train
850/1718 [=============>................] - 1076.5s Epoch:59 train steps: 100321 Loc Loss: 0.0000 Cls Loss: 2.1875 Tot Loss: 2.1875 Acc: 0.4762
1709/1718 [============================>.] - 2141.8s Epoch:59 train steps: 101180 Loc Loss: 0.0000 Cls Loss: 2.2104 Tot Loss: 2.2104 Acc: 0.4767
1718/1718 [==============================] - 2152.7s
1720/1718 [==============================] - 2155.2supdate:
train
849/1718 [=============>................] - 1073.6s Epoch:60 train steps: 102039 Loc Loss: 0.0000 Cls Loss: 2.1813 Tot Loss: 2.1813 Acc: 0.4775
1708/1718 [============================>.] - 2139.8s Epoch:60 train steps: 102898 Loc Loss: 0.0000 Cls Loss: 2.1996 Tot Loss: 2.1996 Acc: 0.4785
1718/1718 [==============================] - 2152.5s
1720/1718 [==============================] - 2154.9supdate:
train
848/1718 [=============>................] - 1071.7s Epoch:61 train steps: 103757 Loc Loss: 0.0000 Cls Loss: 2.1701 Tot Loss: 2.1701 Acc: 0.4787
1707/1718 [============================>.] - 2137.7s Epoch:61 train steps: 104616 Loc Loss: 0.0000 Cls Loss: 2.1923 Tot Loss: 2.1923 Acc: 0.4787
1718/1718 [==============================] - 2150.9s
1720/1718 [==============================] - 2153.3supdate:
val
135/136 [============================>.] - 222.1s Epoch:61 val Loc Loss: 0.0000 Cls Loss: 2.4023 Tot Loss: 2.4023 Acc: 0.4651
Step 104628 Epoch 61
----------
update:
train
847/1718 [=============>................] - 1067.7s Epoch:62 train steps: 105475 Loc Loss: 0.0000 Cls Loss: 2.1634 Tot Loss: 2.1634 Acc: 0.4799
1706/1718 [============================>.] - 2133.5s Epoch:62 train steps: 106334 Loc Loss: 0.0000 Cls Loss: 2.1863 Tot Loss: 2.1863 Acc: 0.4807
1718/1718 [==============================] - 2147.9s
1720/1718 [==============================] - 2150.9supdate:
train
846/1718 [=============>................] - 1066.6s Epoch:63 train steps: 107193 Loc Loss: 0.0000 Cls Loss: 2.1542 Tot Loss: 2.1542 Acc: 0.4816
1576/1718 [==========================>...] - 1973.4s ***** LR 0.04000000000000001 Frames 40/10 BS (1024,512,256) W/H (112,158,224) BN_splits 8 long_ind 2 *****
1705/1718 [============================>.] - 2121.3s Epoch:63 train steps: 108052 Loc Loss: 0.0000 Cls Loss: 2.1763 Tot Loss: 2.1763 Acc: 0.4828
1718/1718 [==============================] - 2136.5s
1720/1718 [==============================] - 2138.5supdate:
train
845/1718 [=============>................] - 983.6s Epoch:64 train steps: 108911 Loc Loss: 0.0000 Cls Loss: 2.0648 Tot Loss: 2.0648 Acc: 0.5012
1704/1718 [============================>.] - 1967.4s Epoch:64 train steps: 109770 Loc Loss: 0.0000 Cls Loss: 2.0879 Tot Loss: 2.0879 Acc: 0.5021
1718/1718 [==============================] - 1982.9s
1720/1718 [==============================] - 1985.8supdate:
train
844/1718 [=============>................] - 983.3s Epoch:65 train steps: 110629 Loc Loss: 0.0000 Cls Loss: 2.0418 Tot Loss: 2.0418 Acc: 0.5036
1703/1718 [============================>.] - 1969.3s Epoch:65 train steps: 111488 Loc Loss: 0.0000 Cls Loss: 2.0852 Tot Loss: 2.0852 Acc: 0.5029
1718/1718 [==============================] - 1986.7s
1720/1718 [==============================] - 1988.7supdate:
val
135/136 [============================>.] - 206.5s Epoch:65 val Loc Loss: 0.0000 Cls Loss: 2.0731 Tot Loss: 2.0731 Acc: 0.5284
Step 111504 Epoch 65
----------
update:
train
843/1718 [=============>................] - 989.5s Epoch:66 train steps: 112347 Loc Loss: 0.0000 Cls Loss: 2.0326 Tot Loss: 2.0326 Acc: 0.5041
1702/1718 [============================>.] - 1979.3s Epoch:66 train steps: 113206 Loc Loss: 0.0000 Cls Loss: 2.0773 Tot Loss: 2.0773 Acc: 0.5034
1718/1718 [==============================] - 1997.9s
1720/1718 [==============================] - 2000.0supdate:
train
842/1718 [=============>................] - 984.6s Epoch:67 train steps: 114065 Loc Loss: 0.0000 Cls Loss: 2.0341 Tot Loss: 2.0341 Acc: 0.5039
1701/1718 [============================>.] - 1974.8s Epoch:67 train steps: 114924 Loc Loss: 0.0000 Cls Loss: 2.0791 Tot Loss: 2.0791 Acc: 0.5037
1718/1718 [==============================] - 1994.3s
1720/1718 [==============================] - 1996.4supdate:
train
841/1718 [=============>................] - 986.0s Epoch:68 train steps: 115783 Loc Loss: 0.0000 Cls Loss: 2.0281 Tot Loss: 2.0281 Acc: 0.5042
1700/1718 [============================>.] - 1974.9s Epoch:68 train steps: 116642 Loc Loss: 0.0000 Cls Loss: 2.0725 Tot Loss: 2.0725 Acc: 0.5038
1718/1718 [==============================] - 1995.2s
1720/1718 [==============================] - 1997.2supdate:
train
840/1718 [=============>................] - 979.9s Epoch:69 train steps: 117501 Loc Loss: 0.0000 Cls Loss: 2.0203 Tot Loss: 2.0203 Acc: 0.5051
1699/1718 [============================>.] - 1963.0s Epoch:69 train steps: 118360 Loc Loss: 0.0000 Cls Loss: 2.0706 Tot Loss: 2.0706 Acc: 0.5041
1718/1718 [==============================] - 1985.2s
1720/1718 [==============================] - 1987.2supdate:
val
135/136 [============================>.] - 211.6s Epoch:69 val Loc Loss: 0.0000 Cls Loss: 2.0978 Tot Loss: 2.0978 Acc: 0.5177
Step 118380 Epoch 69
----------
update:
train
839/1718 [=============>................] - 980.0s Epoch:70 train steps: 119219 Loc Loss: 0.0000 Cls Loss: 2.0182 Tot Loss: 2.0182 Acc: 0.5048
1698/1718 [============================>.] - 1963.2s Epoch:70 train steps: 120078 Loc Loss: 0.0000 Cls Loss: 2.0670 Tot Loss: 2.0670 Acc: 0.5041
1718/1718 [==============================] - 1986.0s
1720/1718 [==============================] - 1988.0supdate:
train
517/1718 [========>.....................] - 608.8s ***** LR 0.020000000000000004 Frames 80/10 BS (512,256,128) W/H (112,158,224) BN_splits 4 long_ind 3 *****
838/1718 [=============>................] - 950.5s Epoch:71 train steps: 120937 Loc Loss: 0.0000 Cls Loss: 1.9605 Tot Loss: 1.9605 Acc: 0.5122
1697/1718 [============================>.] - 1859.0s Epoch:71 train steps: 121796 Loc Loss: 0.0000 Cls Loss: 1.8933 Tot Loss: 1.8933 Acc: 0.5400
1718/1718 [==============================] - 1881.0s
1720/1718 [==============================] - 1883.0supdate:
train
837/1718 [=============>................] - 901.7s Epoch:72 train steps: 122655 Loc Loss: 0.0000 Cls Loss: 1.8309 Tot Loss: 1.8309 Acc: 0.5420
1696/1718 [============================>.] - 1818.8s Epoch:72 train steps: 123514 Loc Loss: 0.0000 Cls Loss: 1.8891 Tot Loss: 1.8891 Acc: 0.5409
1718/1718 [==============================] - 1842.8s
1720/1718 [==============================] - 1845.8supdate:
train
836/1718 [=============>................] - 908.3s Epoch:73 train steps: 124373 Loc Loss: 0.0000 Cls Loss: 1.8242 Tot Loss: 1.8242 Acc: 0.5425
1695/1718 [============================>.] - 1829.6s Epoch:73 train steps: 125232 Loc Loss: 0.0000 Cls Loss: 1.8711 Tot Loss: 1.8711 Acc: 0.5422
1718/1718 [==============================] - 1854.3s
1720/1718 [==============================] - 1856.3supdate:
val
135/136 [============================>.] - 215.5s Epoch:73 val Loc Loss: 0.0000 Cls Loss: 1.8247 Tot Loss: 1.8247 Acc: 0.5599
Step 125256 Epoch 73
----------
update:
train
835/1718 [=============>................] - 901.7s Epoch:74 train steps: 126091 Loc Loss: 0.0000 Cls Loss: 1.8154 Tot Loss: 1.8154 Acc: 0.5422
1694/1718 [============================>.] - 1822.8s Epoch:74 train steps: 126950 Loc Loss: 0.0000 Cls Loss: 1.8666 Tot Loss: 1.8666 Acc: 0.5434
1718/1718 [==============================] - 1847.8s
1720/1718 [==============================] - 1849.8supdate:
train
834/1718 [=============>................] - 905.4s Epoch:75 train steps: 127809 Loc Loss: 0.0000 Cls Loss: 1.8168 Tot Loss: 1.8168 Acc: 0.5431
1693/1718 [============================>.] - 1818.8s Epoch:75 train steps: 128668 Loc Loss: 0.0000 Cls Loss: 1.8743 Tot Loss: 1.8743 Acc: 0.5427
1718/1718 [==============================] - 1846.4s
1720/1718 [==============================] - 1848.5supdate:
train
833/1718 [=============>................] - 913.7s Epoch:76 train steps: 129527 Loc Loss: 0.0000 Cls Loss: 1.8109 Tot Loss: 1.8109 Acc: 0.5445
1692/1718 [============================>.] - 1895.1s Epoch:76 train steps: 130386 Loc Loss: 0.0000 Cls Loss: 1.8687 Tot Loss: 1.8687 Acc: 0.5452
1718/1718 [==============================] - 1923.1s
1720/1718 [==============================] - 1925.1supdate:
train
832/1718 [=============>................] - 907.4s Epoch:77 train steps: 131245 Loc Loss: 0.0000 Cls Loss: 1.8070 Tot Loss: 1.8070 Acc: 0.5435
1691/1718 [============================>.] - 1824.8s Epoch:77 train steps: 132104 Loc Loss: 0.0000 Cls Loss: 1.8640 Tot Loss: 1.8640 Acc: 0.5438
1718/1718 [==============================] - 1855.2s
1720/1718 [==============================] - 1857.2supdate:
val
135/136 [============================>.] - 208.2s Epoch:77 val Loc Loss: 0.0000 Cls Loss: 1.7728 Tot Loss: 1.7728 Acc: 0.5656
Step 132132 Epoch 77
----------
update:
train
831/1718 [=============>................] - 908.5s Epoch:78 train steps: 132963 Loc Loss: 0.0000 Cls Loss: 1.8054 Tot Loss: 1.8054 Acc: 0.5444
1202/1718 [===================>..........] - 1311.3s ***** LR 0.16000000000000003 Frames 20/10 BS (2048,1024) W/H (111,158) BN_splits 32 long_ind 0 *****
1690/1718 [============================>.] - 1921.2s Epoch:78 train steps: 133822 Loc Loss: 0.0000 Cls Loss: 2.2178 Tot Loss: 2.2178 Acc: 0.4392
1718/1718 [==============================] - 1955.4s
1720/1718 [==============================] - 1957.8supdate:
train
830/1718 [=============>................] - 1055.5s Epoch:79 train steps: 134681 Loc Loss: 0.0000 Cls Loss: 2.1957 Tot Loss: 2.1957 Acc: 0.4654
1689/1718 [============================>.] - 2131.3s Epoch:79 train steps: 135540 Loc Loss: 0.0000 Cls Loss: 2.1804 Tot Loss: 2.1804 Acc: 0.4840
1718/1718 [==============================] - 2167.7s
1720/1718 [==============================] - 2170.1supdate:
train
829/1718 [=============>................] - 1056.8s Epoch:80 train steps: 136399 Loc Loss: 0.0000 Cls Loss: 2.0797 Tot Loss: 2.0797 Acc: 0.4891
1688/1718 [============================>.] - 2132.6s Epoch:80 train steps: 137258 Loc Loss: 0.0000 Cls Loss: 2.1409 Tot Loss: 2.1409 Acc: 0.4919
1718/1718 [==============================] - 2170.3s
1720/1718 [==============================] - 2172.7supdate:
train
828/1718 [=============>................] - 1049.1s Epoch:81 train steps: 138117 Loc Loss: 0.0000 Cls Loss: 2.0531 Tot Loss: 2.0531 Acc: 0.4943
1687/1718 [============================>.] - 2123.7s Epoch:81 train steps: 138976 Loc Loss: 0.0000 Cls Loss: 2.1211 Tot Loss: 2.1211 Acc: 0.4960
1718/1718 [==============================] - 2162.6s
1720/1718 [==============================] - 2165.1supdate:
val
135/136 [============================>.] - 220.2s Epoch:81 val Loc Loss: 0.0000 Cls Loss: 3.1346 Tot Loss: 3.1346 Acc: 0.3822
Step 139008 Epoch 81
----------
update:
train
827/1718 [=============>................] - 1054.0s Epoch:82 train steps: 139835 Loc Loss: 0.0000 Cls Loss: 2.0353 Tot Loss: 2.0353 Acc: 0.4971
1686/1718 [============================>.] - 2128.3s Epoch:82 train steps: 140694 Loc Loss: 0.0000 Cls Loss: 2.1093 Tot Loss: 2.1093 Acc: 0.4987
1718/1718 [==============================] - 2169.0s
1720/1718 [==============================] - 2171.4supdate:
train
826/1718 [=============>................] - 1055.1s Epoch:83 train steps: 141553 Loc Loss: 0.0000 Cls Loss: 2.0233 Tot Loss: 2.0233 Acc: 0.4992
1685/1718 [============================>.] - 2129.5s Epoch:83 train steps: 142412 Loc Loss: 0.0000 Cls Loss: 2.0975 Tot Loss: 2.0975 Acc: 0.5007
1718/1718 [==============================] - 2171.0s
1720/1718 [==============================] - 2173.5supdate:
train
825/1718 [=============>................] - 1062.9s Epoch:84 train steps: 143271 Loc Loss: 0.0000 Cls Loss: 2.0119 Tot Loss: 2.0119 Acc: 0.5013
1684/1718 [============================>.] - 2137.3s Epoch:84 train steps: 144130 Loc Loss: 0.0000 Cls Loss: 2.0909 Tot Loss: 2.0909 Acc: 0.5023
1718/1718 [==============================] - 2179.2s
1720/1718 [==============================] - 2182.4supdate:
train
824/1718 [=============>................] - 1044.2s Epoch:85 train steps: 144989 Loc Loss: 0.0000 Cls Loss: 2.0020 Tot Loss: 2.0020 Acc: 0.5031
1683/1718 [============================>.] - 2117.3s Epoch:85 train steps: 145848 Loc Loss: 0.0000 Cls Loss: 2.0842 Tot Loss: 2.0842 Acc: 0.5037
1718/1718 [==============================] - 2161.1s
1720/1718 [==============================] - 2163.5supdate:
val
135/136 [============================>.] - 249.0s Epoch:85 val Loc Loss: 0.0000 Cls Loss: 3.3365 Tot Loss: 3.3365 Acc: 0.3494
Step 145884 Epoch 85
----------
update:
train
823/1718 [=============>................] - 1048.5s Epoch:86 train steps: 146707 Loc Loss: 0.0000 Cls Loss: 1.9927 Tot Loss: 1.9927 Acc: 0.5046
1431/1718 [=======================>......] - 1810.5s ***** LR 0.008000000000000002 Frames 40/10 BS (1024,512) W/H (111,158) BN_splits 16 long_ind 1 *****
1682/1718 [============================>.] - 2137.1s Epoch:86 train steps: 147566 Loc Loss: 0.0000 Cls Loss: 2.0357 Tot Loss: 2.0357 Acc: 0.5098
1718/1718 [==============================] - 2182.0s
1720/1718 [==============================] - 2184.5supdate:
train
822/1718 [=============>................] - 1039.3s Epoch:87 train steps: 148425 Loc Loss: 0.0000 Cls Loss: 1.8045 Tot Loss: 1.8045 Acc: 0.5423
1681/1718 [============================>.] - 2107.7s Epoch:87 train steps: 149284 Loc Loss: 0.0000 Cls Loss: 1.8732 Tot Loss: 1.8732 Acc: 0.5455
1718/1718 [==============================] - 2154.0s
1720/1718 [==============================] - 2156.4supdate:
train
821/1718 [=============>................] - 1041.7s Epoch:88 train steps: 150143 Loc Loss: 0.0000 Cls Loss: 1.7832 Tot Loss: 1.7832 Acc: 0.5460
1680/1718 [============================>.] - 2112.0s Epoch:88 train steps: 151002 Loc Loss: 0.0000 Cls Loss: 1.8620 Tot Loss: 1.8620 Acc: 0.5469
1718/1718 [==============================] - 2159.4s
1720/1718 [==============================] - 2161.8supdate:
train
820/1718 [=============>................] - 1042.5s Epoch:89 train steps: 151861 Loc Loss: 0.0000 Cls Loss: 1.7720 Tot Loss: 1.7720 Acc: 0.5485
1679/1718 [============================>.] - 2112.8s Epoch:89 train steps: 152720 Loc Loss: 0.0000 Cls Loss: 1.8535 Tot Loss: 1.8535 Acc: 0.5487
1718/1718 [==============================] - 2161.7s
1720/1718 [==============================] - 2164.2supdate:
val
135/136 [============================>.] - 242.8s Epoch:89 val Loc Loss: 0.0000 Cls Loss: 2.0892 Tot Loss: 2.0892 Acc: 0.5313
Step 152760 Epoch 89
----------
update:
train
819/1718 [=============>................] - 1036.0s Epoch:90 train steps: 153579 Loc Loss: 0.0000 Cls Loss: 1.7684 Tot Loss: 1.7684 Acc: 0.5487
1678/1718 [============================>.] - 2104.6s Epoch:90 train steps: 154438 Loc Loss: 0.0000 Cls Loss: 1.8528 Tot Loss: 1.8528 Acc: 0.5495
1718/1718 [==============================] - 2154.3s
1720/1718 [==============================] - 2156.7supdate:
train
818/1718 [=============>................] - 1040.3s Epoch:91 train steps: 155297 Loc Loss: 0.0000 Cls Loss: 1.7606 Tot Loss: 1.7606 Acc: 0.5499
1677/1718 [============================>.] - 2111.0s Epoch:91 train steps: 156156 Loc Loss: 0.0000 Cls Loss: 1.8451 Tot Loss: 1.8451 Acc: 0.5499
1718/1718 [==============================] - 2162.3s
1720/1718 [==============================] - 2164.6supdate:
train
817/1718 [=============>................] - 1034.9s Epoch:92 train steps: 157015 Loc Loss: 0.0000 Cls Loss: 1.7531 Tot Loss: 1.7531 Acc: 0.5506
1676/1718 [============================>.] - 2101.3s Epoch:92 train steps: 157874 Loc Loss: 0.0000 Cls Loss: 1.8454 Tot Loss: 1.8454 Acc: 0.5510
1718/1718 [==============================] - 2153.0s
1720/1718 [==============================] - 2156.0supdate:
train
816/1718 [=============>................] - 1034.5s Epoch:93 train steps: 158733 Loc Loss: 0.0000 Cls Loss: 1.7511 Tot Loss: 1.7511 Acc: 0.5497
1675/1718 [============================>.] - 2102.7s Epoch:93 train steps: 159592 Loc Loss: 0.0000 Cls Loss: 1.8430 Tot Loss: 1.8430 Acc: 0.5509
1718/1718 [==============================] - 2156.0s
1720/1718 [==============================] - 2158.3supdate:
val
135/136 [============================>.] - 218.3s Epoch:93 val Loc Loss: 0.0000 Cls Loss: 2.0720 Tot Loss: 2.0720 Acc: 0.5307
Step 159636 Epoch 93
----------
update:
train
815/1718 [=============>................] - 1033.0s Epoch:94 train steps: 160451 Loc Loss: 0.0000 Cls Loss: 1.7451 Tot Loss: 1.7451 Acc: 0.5514
1660/1718 [===========================>..] - 2082.4s ***** LR 0.004000000000000001 Frames 40/10 BS (1024,512,256) W/H (112,158,224) BN_splits 8 long_ind 2 *****
1674/1718 [============================>.] - 2099.1s Epoch:94 train steps: 161310 Loc Loss: 0.0000 Cls Loss: 1.8359 Tot Loss: 1.8359 Acc: 0.5512
1718/1718 [==============================] - 2149.3s
1720/1718 [==============================] - 2151.4supdate:
train
814/1718 [=============>................] - 953.3s Epoch:95 train steps: 162169 Loc Loss: 0.0000 Cls Loss: 1.7183 Tot Loss: 1.7183 Acc: 0.5631
1673/1718 [============================>.] - 1937.2s Epoch:95 train steps: 163028 Loc Loss: 0.0000 Cls Loss: 1.8001 Tot Loss: 1.8001 Acc: 0.5633
1718/1718 [==============================] - 1988.7s
1720/1718 [==============================] - 1990.7supdate:
train
813/1718 [=============>................] - 957.8s Epoch:96 train steps: 163887 Loc Loss: 0.0000 Cls Loss: 1.6988 Tot Loss: 1.6988 Acc: 0.5633
1672/1718 [============================>.] - 1946.0s Epoch:96 train steps: 164746 Loc Loss: 0.0000 Cls Loss: 1.7992 Tot Loss: 1.7992 Acc: 0.5623
1718/1718 [==============================] - 1999.0s
1720/1718 [==============================] - 2001.3supdate:
train
812/1718 [=============>................] - 947.1s Epoch:97 train steps: 165605 Loc Loss: 0.0000 Cls Loss: 1.6972 Tot Loss: 1.6972 Acc: 0.5628
1671/1718 [============================>.] - 1934.9s Epoch:97 train steps: 166464 Loc Loss: 0.0000 Cls Loss: 1.7921 Tot Loss: 1.7921 Acc: 0.5636
1718/1718 [==============================] - 1988.7s
1720/1718 [==============================] - 1990.7supdate:
val
135/136 [============================>.] - 224.8s Epoch:97 val Loc Loss: 0.0000 Cls Loss: 1.8751 Tot Loss: 1.8751 Acc: 0.5692
Step 166512 Epoch 97
----------
update:
train
811/1718 [=============>................] - 943.2s Epoch:98 train steps: 167323 Loc Loss: 0.0000 Cls Loss: 1.6887 Tot Loss: 1.6887 Acc: 0.5632
1670/1718 [============================>.] - 1925.2s Epoch:98 train steps: 168182 Loc Loss: 0.0000 Cls Loss: 1.7891 Tot Loss: 1.7891 Acc: 0.5633
1718/1718 [==============================] - 1980.9s
1720/1718 [==============================] - 1982.9supdate:
train
810/1718 [=============>................] - 945.6s Epoch:99 train steps: 169041 Loc Loss: 0.0000 Cls Loss: 1.6879 Tot Loss: 1.6879 Acc: 0.5637
1669/1718 [============================>.] - 1932.0s Epoch:99 train steps: 169900 Loc Loss: 0.0000 Cls Loss: 1.7896 Tot Loss: 1.7896 Acc: 0.5646
1718/1718 [==============================] - 1987.7s
1720/1718 [==============================] - 1989.8supdate:
train
809/1718 [=============>................] - 941.5s Epoch:100 train steps: 170759 Loc Loss: 0.0000 Cls Loss: 1.6850 Tot Loss: 1.6850 Acc: 0.5639
1668/1718 [============================>.] - 1925.9s Epoch:100 train steps: 171618 Loc Loss: 0.0000 Cls Loss: 1.7881 Tot Loss: 1.7881 Acc: 0.5634
1718/1718 [==============================] - 1982.8s
1720/1718 [==============================] - 1984.8supdate:
python train_x3d_kinetics_multigrid.py -gpu 4,5,6,7
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/train_frames/labeldata_80.npy exists
/data/add_disk0/kumarak/Kinetics/kinetics-downloader/dataset/valid_frames/labeldata_80.npy exists
train 215901 val 17227
Total iterations: 206160 Total steps: 206160
datasets created
model loaded
LR:0.200000
Step 172000 Epoch 100
----------
update:
val
135/136 [============================>.] - 211.3s Epoch:100 val Loc Loss: 0.0000 Cls Loss: 1.8572 Tot Loss: 1.8572 Acc: 0.5672
client_loop: send disconnect: Broken pipe
train
1/1718 [..............................] - 14.9s ***** LR 0.004000000000000001 Frames 40/10 BS (1024,512,256) W/H (112,158,224) BN_splits 8 long_ind 2 *****
859/1718 [==============>...............] - 1028.3s Epoch:101 train steps: 172859 Loc Loss: 0.0000 Cls Loss: 1.7839 Tot Loss: 1.7839 Acc: 0.5639
1718/1718 [==============================] - 2025.9s
Epoch:101 train steps: 173718 Loc Loss: 0.0000 Cls Loss: 1.7864 Tot Loss: 1.7864 Acc: 0.5639
1720/1718 [==============================] - 2027.9supdate:
train
858/1718 [=============>................] - 1011.6s Epoch:102 train steps: 174577 Loc Loss: 0.0000 Cls Loss: 1.7865 Tot Loss: 1.7865 Acc: 0.5644
1717/1718 [============================>.] - 2007.7s Epoch:102 train steps: 175436 Loc Loss: 0.0000 Cls Loss: 1.7819 Tot Loss: 1.7819 Acc: 0.5648
1718/1718 [==============================] - 2009.0s
1720/1718 [==============================] - 2011.1supdate:
train
857/1718 [=============>................] - 1004.2s Epoch:103 train steps: 176295 Loc Loss: 0.0000 Cls Loss: 1.7794 Tot Loss: 1.7794 Acc: 0.5648
1040/1718 [=================>............] - 1213.4s ***** LR 0.0020000000000000005 Frames 80/10 BS (512,256,128) W/H (112,158,224) BN_splits 4 long_ind 3 *****
1716/1718 [============================>.] - 1952.1s Epoch:103 train steps: 177154 Loc Loss: 0.0000 Cls Loss: 1.6655 Tot Loss: 1.6655 Acc: 0.5855
1718/1718 [==============================] - 1956.3s
1720/1718 [==============================] - 1958.4supdate:
train
856/1718 [=============>................] - 970.6s Epoch:104 train steps: 178013 Loc Loss: 0.0000 Cls Loss: 1.6214 Tot Loss: 1.6214 Acc: 0.5972
1715/1718 [============================>.] - 1926.6s Epoch:104 train steps: 178872 Loc Loss: 0.0000 Cls Loss: 1.6166 Tot Loss: 1.6166 Acc: 0.5988
1718/1718 [==============================] - 1930.0s
1720/1718 [==============================] - 1932.1supdate:
val
135/136 [============================>.] - 206.1s Epoch:104 val Loc Loss: 0.0000 Cls Loss: 1.6255 Tot Loss: 1.6255 Acc: 0.5999
Step 178876 Epoch 104
----------
update:
train
855/1718 [=============>................] - 958.5s Epoch:105 train steps: 179731 Loc Loss: 0.0000 Cls Loss: 1.6106 Tot Loss: 1.6106 Acc: 0.5993
1714/1718 [============================>.] - 1925.1s Epoch:105 train steps: 180590 Loc Loss: 0.0000 Cls Loss: 1.6139 Tot Loss: 1.6139 Acc: 0.5992
1718/1718 [==============================] - 1930.0s
1720/1718 [==============================] - 1932.0supdate:
train
854/1718 [=============>................] - 951.4s Epoch:106 train steps: 181449 Loc Loss: 0.0000 Cls Loss: 1.6057 Tot Loss: 1.6057 Acc: 0.5995
1713/1718 [============================>.] - 1893.2s Epoch:106 train steps: 182308 Loc Loss: 0.0000 Cls Loss: 1.6062 Tot Loss: 1.6062 Acc: 0.6007
1718/1718 [==============================] - 1898.2s
1720/1718 [==============================] - 1900.8supdate:
train
853/1718 [=============>................] - 950.7s Epoch:107 train steps: 183167 Loc Loss: 0.0000 Cls Loss: 1.5983 Tot Loss: 1.5983 Acc: 0.6003
1712/1718 [============================>.] - 1907.9s Epoch:107 train steps: 184026 Loc Loss: 0.0000 Cls Loss: 1.6050 Tot Loss: 1.6050 Acc: 0.5999
1718/1718 [==============================] - 1914.7s
1720/1718 [==============================] - 1916.7supdate:
train
852/1718 [=============>................] - 982.3s Epoch:108 train steps: 184885 Loc Loss: 0.0000 Cls Loss: 1.5959 Tot Loss: 1.5959 Acc: 0.6006
1711/1718 [============================>.] - 1952.5s Epoch:108 train steps: 185744 Loc Loss: 0.0000 Cls Loss: 1.6059 Tot Loss: 1.6059 Acc: 0.6009
1718/1718 [==============================] - 1959.6s
1720/1718 [==============================] - 1961.7supdate:
val
135/136 [============================>.] - 220.5s Epoch:108 val Loc Loss: 0.0000 Cls Loss: 1.6333 Tot Loss: 1.6333 Acc: 0.6027
Step 185752 Epoch 108
----------
update:
train
851/1718 [=============>................] - 966.4s Epoch:109 train steps: 186603 Loc Loss: 0.0000 Cls Loss: 1.5896 Tot Loss: 1.5896 Acc: 0.6002
1710/1718 [============================>.] - 1932.3s Epoch:109 train steps: 187462 Loc Loss: 0.0000 Cls Loss: 1.5998 Tot Loss: 1.5998 Acc: 0.6011
1718/1718 [==============================] - 1941.6s
1720/1718 [==============================] - 1943.7supdate:
train
850/1718 [=============>................] - 975.6s Epoch:110 train steps: 188321 Loc Loss: 0.0000 Cls Loss: 1.5813 Tot Loss: 1.5813 Acc: 0.6028
1709/1718 [============================>.] - 1946.4s Epoch:110 train steps: 189180 Loc Loss: 0.0000 Cls Loss: 1.6030 Tot Loss: 1.6030 Acc: 0.6024
1718/1718 [==============================] - 1958.1s
1720/1718 [==============================] - 1960.1supdate:
train
849/1718 [=============>................] - 967.4s Epoch:111 train steps: 190039 Loc Loss: 0.0000 Cls Loss: 1.5765 Tot Loss: 1.5765 Acc: 0.6024
1270/1718 [=====================>........] - 1429.2s ***** LR 0.0020000000000000005 Frames 80/10 BS (512,256,128) W/H (112,158,224) BN_splits 4 long_ind -1 *****
1708/1718 [============================>.] - 1914.8s Epoch:111 train steps: 190898 Loc Loss: 0.0000 Cls Loss: 1.5960 Tot Loss: 1.5960 Acc: 0.6029
1718/1718 [==============================] - 1926.0s
1720/1718 [==============================] - 1928.0supdate:
train
848/1718 [=============>................] - 957.1s Epoch:112 train steps: 191757 Loc Loss: 0.0000 Cls Loss: 1.5585 Tot Loss: 1.5585 Acc: 0.6072
1707/1718 [============================>.] - 1921.6s Epoch:112 train steps: 192616 Loc Loss: 0.0000 Cls Loss: 1.5615 Tot Loss: 1.5615 Acc: 0.6099
1718/1718 [==============================] - 1933.7s
1720/1718 [==============================] - 1935.7supdate:
val
135/136 [============================>.] - 222.6s Epoch:112 val Loc Loss: 0.0000 Cls Loss: 1.5964 Tot Loss: 1.5964 Acc: 0.6081
Step 192628 Epoch 112
----------
update:
train
847/1718 [=============>................] - 964.2s Epoch:113 train steps: 193475 Loc Loss: 0.0000 Cls Loss: 1.5520 Tot Loss: 1.5520 Acc: 0.6088
1706/1718 [============================>.] - 1920.4s Epoch:113 train steps: 194334 Loc Loss: 0.0000 Cls Loss: 1.5707 Tot Loss: 1.5707 Acc: 0.6080
1718/1718 [==============================] - 1932.4s
1720/1718 [==============================] - 1935.8supdate:
train
846/1718 [=============>................] - 956.9s Epoch:114 train steps: 195193 Loc Loss: 0.0000 Cls Loss: 1.5462 Tot Loss: 1.5462 Acc: 0.6092
1705/1718 [============================>.] - 1897.7s Epoch:114 train steps: 196052 Loc Loss: 0.0000 Cls Loss: 1.5716 Tot Loss: 1.5716 Acc: 0.6089
1718/1718 [==============================] - 1912.9s
1720/1718 [==============================] - 1917.0supdate:
train
845/1718 [=============>................] - 947.5s Epoch:115 train steps: 196911 Loc Loss: 0.0000 Cls Loss: 1.5399 Tot Loss: 1.5399 Acc: 0.6091
1704/1718 [============================>.] - 1886.0s Epoch:115 train steps: 197770 Loc Loss: 0.0000 Cls Loss: 1.5595 Tot Loss: 1.5595 Acc: 0.6102
1718/1718 [==============================] - 1902.3s
1720/1718 [==============================] - 1904.3supdate:
train
844/1718 [=============>................] - 938.7s Epoch:116 train steps: 198629 Loc Loss: 0.0000 Cls Loss: 1.5389 Tot Loss: 1.5389 Acc: 0.6103
1703/1718 [============================>.] - 1883.1s Epoch:116 train steps: 199488 Loc Loss: 0.0000 Cls Loss: 1.5642 Tot Loss: 1.5642 Acc: 0.6089
1718/1718 [==============================] - 1899.2s
1720/1718 [==============================] - 1901.4supdate:
val
135/136 [============================>.] - 225.2s Epoch:116 val Loc Loss: 0.0000 Cls Loss: 1.5904 Tot Loss: 1.5904 Acc: 0.6095
Step 199504 Epoch 116
----------
update:
train
843/1718 [=============>................] - 939.1s Epoch:117 train steps: 200347 Loc Loss: 0.0000 Cls Loss: 1.5350 Tot Loss: 1.5350 Acc: 0.6098
1702/1718 [============================>.] - 1883.7s Epoch:117 train steps: 201206 Loc Loss: 0.0000 Cls Loss: 1.5584 Tot Loss: 1.5584 Acc: 0.6111
1718/1718 [==============================] - 1903.3s
1720/1718 [==============================] - 1905.3supdate:
train
842/1718 [=============>................] - 931.3s Epoch:118 train steps: 202065 Loc Loss: 0.0000 Cls Loss: 1.5300 Tot Loss: 1.5300 Acc: 0.6103
1701/1718 [============================>.] - 2017.4s Epoch:118 train steps: 202924 Loc Loss: 0.0000 Cls Loss: 1.5649 Tot Loss: 1.5649 Acc: 0.6097
1718/1718 [==============================] - 2040.1s
1720/1718 [==============================] - 2054.4supdate:
train
841/1718 [=============>................] - 1232.6s Epoch:119 train steps: 203783 Loc Loss: 0.0000 Cls Loss: 1.5284 Tot Loss: 1.5284 Acc: 0.6101
1700/1718 [============================>.] - 2231.3s Epoch:119 train steps: 204642 Loc Loss: 0.0000 Cls Loss: 1.5680 Tot Loss: 1.5680 Acc: 0.6084
1718/1718 [==============================] - 2252.5s
1720/1718 [==============================] - 2254.4supdate:
train
840/1718 [=============>................] - 963.4s Epoch:120 train steps: 205501 Loc Loss: 0.0000 Cls Loss: 1.5282 Tot Loss: 1.5282 Acc: 0.6102
1020/1718 [================>.............] - 1172.9sTraceback (most recent call last):
corrected val spatial cropping, same 3-temporal crop
update:
val
135/136 [============================>.] - 207.2s Epoch:118 val Loc Loss: 0.0000 Cls Loss: 1.5335 Tot Loss: 1.5335 Acc: 0.6262
Step 204000 Epoch 118
----------