-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCT_MSRVTT_full.txt
3154 lines (3154 loc) · 183 KB
/
HCT_MSRVTT_full.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 524.4001622200012 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 42.76755213737488 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 261.7604694366455 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 135.27866768836975 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch0.pth ...
Done in 8.306s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch0.pth ...
Done in 9.853s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.2012072434607646
MSRVTT_full_val/t2v_metrics/R5: 0.8048289738430584
MSRVTT_full_val/t2v_metrics/R10: 1.408450704225352
MSRVTT_full_val/t2v_metrics/R50: 10.060362173038229
MSRVTT_full_val/t2v_metrics/MedR: 255.0
MSRVTT_full_val/t2v_metrics/MeanR: 254.12474849094568
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.6109836965544593
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.6036217303822937
MSRVTT_full_val/v2t_metrics/R10: 1.8108651911468814
MSRVTT_full_val/v2t_metrics/R50: 9.859154929577464
MSRVTT_full_val/v2t_metrics/MedR: 260.0
MSRVTT_full_val/v2t_metrics/MeanR: 254.46277665995976
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.23411371237458195
MSRVTT_full_test/t2v_metrics/R10: 0.43478260869565216
MSRVTT_full_test/t2v_metrics/R50: 2.1070234113712374
MSRVTT_full_test/t2v_metrics/MedR: 1498.0
MSRVTT_full_test/t2v_metrics/MeanR: 1497.0429765886288
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.15043282425670285
MSRVTT_full_test/v2t_metrics/R1: 0.033444816053511704
MSRVTT_full_test/v2t_metrics/R5: 0.23411371237458195
MSRVTT_full_test/v2t_metrics/R10: 0.4013377926421405
MSRVTT_full_test/v2t_metrics/R50: 1.8729096989966556
MSRVTT_full_test/v2t_metrics/MedR: 1495.5
MSRVTT_full_test/v2t_metrics/MeanR: 1494.5735785953177
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14647221203638425
mnt_best : 0.15043282425670285
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.83392 batch_time=25.54276
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.66514 batch_time=0.37019
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.11823 batch_time=0.37618
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.76084 batch_time=0.36244
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.27419 batch_time=0.40594
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.03600 batch_time=0.39431
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.60832 batch_time=0.72059
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.66158 batch_time=0.37427
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.51408 batch_time=0.34899
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.25779 batch_time=0.36765
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.14447 batch_time=0.39699
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.76389 batch_time=0.39443
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.17941 batch_time=0.84647
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.28411 batch_time=0.38403
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.90672 batch_time=0.35921
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.08412 batch_time=0.36070
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.23589 batch_time=0.45413
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.01460 batch_time=0.35908
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 3.92534 batch_time=0.37346
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 3.98102 batch_time=0.37416
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.02772 batch_time=0.36025
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.23773 batch_time=0.35787
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.02721 batch_time=1.08338
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch1.pth ...
Done in 3.956s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch1.pth ...
Done in 7.766s
epoch : 1
loss : 5.2643980731964115
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 22.132796780684103
MSRVTT_full_val/t2v_metrics/R5: 59.758551307847085
MSRVTT_full_val/t2v_metrics/R10: 72.83702213279678
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.865191146881287
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 45.841924997640476
MSRVTT_full_val/v2t_metrics/R1: 26.358148893360163
MSRVTT_full_val/v2t_metrics/R5: 60.96579476861167
MSRVTT_full_val/v2t_metrics/R10: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R50: 96.17706237424547
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.38430583501006
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.40655014057821
MSRVTT_full_test/t2v_metrics/R1: 8.060200668896321
MSRVTT_full_test/t2v_metrics/R5: 26.421404682274247
MSRVTT_full_test/t2v_metrics/R10: 37.85953177257525
MSRVTT_full_test/t2v_metrics/R50: 71.67224080267559
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 59.51036789297659
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.052060144868815
MSRVTT_full_test/v2t_metrics/R1: 8.762541806020067
MSRVTT_full_test/v2t_metrics/R5: 28.59531772575251
MSRVTT_full_test/v2t_metrics/R10: 41.77257525083612
MSRVTT_full_test/v2t_metrics/R50: 75.21739130434783
MSRVTT_full_test/v2t_metrics/MedR: 16.0
MSRVTT_full_test/v2t_metrics/MeanR: 54.37926421404682
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.874531478264746
mnt_best : 20.052060144868815
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.88483 batch_time=29.40067
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.35911 batch_time=0.36314
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.66350 batch_time=0.36299
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.54970 batch_time=0.39018
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.82460 batch_time=0.35993
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.06415 batch_time=0.36512
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.28008 batch_time=0.35797
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.00756 batch_time=0.36097
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.40755 batch_time=0.38418
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.75737 batch_time=0.35476
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.39285 batch_time=0.36657
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.69634 batch_time=0.41308
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.17955 batch_time=0.35992
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.40728 batch_time=0.40881
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.32397 batch_time=0.37261
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.20936 batch_time=0.36350
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.36102 batch_time=0.38369
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.37944 batch_time=0.36222
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.23789 batch_time=0.37030
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.16372 batch_time=0.36782
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.22821 batch_time=0.40170
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.31194 batch_time=0.37425
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 2.92923 batch_time=0.34873
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch2.pth ...
Done in 3.791s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch2.pth ...
Done in 7.451s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.515139163017273
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.311871227364186
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.77814908955263
MSRVTT_full_val/v2t_metrics/R1: 33.199195171026155
MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.5875251509054324
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.94018259670176
MSRVTT_full_test/t2v_metrics/R1: 11.33779264214047
MSRVTT_full_test/t2v_metrics/R5: 31.304347826086957
MSRVTT_full_test/t2v_metrics/R10: 44.5819397993311
MSRVTT_full_test/t2v_metrics/R50: 77.29096989966555
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.444147157190635
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.10522071063057
MSRVTT_full_test/v2t_metrics/R1: 12.909698996655518
MSRVTT_full_test/v2t_metrics/R5: 38.42809364548495
MSRVTT_full_test/v2t_metrics/R10: 52.30769230769231
MSRVTT_full_test/v2t_metrics/R50: 83.4113712374582
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.85585284280936
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.605802595301167
mnt_best : 25.10522071063057
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.01603 batch_time=31.25110
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.20618 batch_time=0.36146
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 2.95613 batch_time=0.35872
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.22847 batch_time=0.34931
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.05845 batch_time=0.37597
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.20742 batch_time=0.35998
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.38822 batch_time=1.04090
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 2.70416 batch_time=0.37159
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.21189 batch_time=0.37033
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.11823 batch_time=0.41689
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.79997 batch_time=0.35959
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.23310 batch_time=0.36618
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.83298 batch_time=2.02240
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.28606 batch_time=0.35841
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.18147 batch_time=0.38000
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.38945 batch_time=0.36258
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.64079 batch_time=0.35941
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.77779 batch_time=0.40110
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.56822 batch_time=0.41823
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.90983 batch_time=0.38444
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.54650 batch_time=0.35539
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.85640 batch_time=0.35931
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.83408 batch_time=0.42269
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch3.pth ...
Done in 4.002s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch3.pth ...
Done in 8.132s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 2.9744157733917236
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
MSRVTT_full_val/t2v_metrics/R5: 68.61167002012073
MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.01609657947686
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.80134141252314
MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.257545271629779
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.50151515891983
MSRVTT_full_test/t2v_metrics/R1: 11.404682274247492
MSRVTT_full_test/t2v_metrics/R5: 33.04347826086956
MSRVTT_full_test/t2v_metrics/R10: 45.25083612040134
MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.03846153846154
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.739405889117798
MSRVTT_full_test/v2t_metrics/R1: 14.214046822742475
MSRVTT_full_test/v2t_metrics/R5: 39.163879598662206
MSRVTT_full_test/v2t_metrics/R10: 54.5819397993311
MSRVTT_full_test/v2t_metrics/R50: 85.55183946488295
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.433779264214046
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.20451748807841
mnt_best : 25.739405889117798
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.15366 batch_time=31.78218
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.09364 batch_time=0.40159
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.56596 batch_time=0.37915
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.94183 batch_time=0.35773
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.89290 batch_time=0.38580
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.95769 batch_time=0.37590
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.72887 batch_time=5.02948
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.84479 batch_time=0.37435
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.54744 batch_time=0.37796
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.20970 batch_time=0.37468
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.65284 batch_time=0.36754
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.98661 batch_time=0.35929
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.13287 batch_time=0.36340
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.61452 batch_time=0.35169
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.41783 batch_time=0.36864
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.52549 batch_time=0.35553
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.38596 batch_time=0.37817
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.24482 batch_time=0.61566
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.46445 batch_time=0.37159
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.14744 batch_time=0.38256
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.51517 batch_time=0.39236
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.32343 batch_time=0.40083
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.41788 batch_time=0.35284
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch4.pth ...
Done in 4.112s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch4.pth ...
Done in 8.541s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 2.582158268451691
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
MSRVTT_full_val/t2v_metrics/R5: 69.81891348088531
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.275653923541247
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.41270028541397
MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
MSRVTT_full_val/v2t_metrics/R5: 76.65995975855131
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.907444668008048
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.76779849835815
MSRVTT_full_test/t2v_metrics/R1: 11.471571906354516
MSRVTT_full_test/t2v_metrics/R5: 34.54849498327759
MSRVTT_full_test/t2v_metrics/R10: 47.357859531772576
MSRVTT_full_test/t2v_metrics/R50: 79.63210702341136
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.52876254180602
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.575495864586124
MSRVTT_full_test/v2t_metrics/R1: 15.68561872909699
MSRVTT_full_test/v2t_metrics/R5: 41.87290969899666
MSRVTT_full_test/v2t_metrics/R10: 55.919732441471574
MSRVTT_full_test/v2t_metrics/R50: 85.41806020066889
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.293979933110364
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.24042949237502
mnt_best : 26.575495864586124
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.71422 batch_time=43.66518
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.29561 batch_time=0.37469
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.50321 batch_time=0.37993
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.05000 batch_time=0.37682
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.46746 batch_time=0.41121
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.84647 batch_time=0.38375
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.23944 batch_time=0.37610
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.10276 batch_time=0.40899
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.18144 batch_time=0.39011
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.54196 batch_time=0.36239
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.50962 batch_time=0.37838
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.55358 batch_time=0.35539
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.33393 batch_time=0.37085
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 1.96934 batch_time=0.35135
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.66767 batch_time=0.35630
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.40499 batch_time=0.38841
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.86920 batch_time=0.44884
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 1.94783 batch_time=0.37213
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 1.91988 batch_time=0.37497
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 1.84097 batch_time=0.61171
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.29389 batch_time=0.35721
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.54803 batch_time=0.35570
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 1.84853 batch_time=0.35950
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch5.pth ...
Done in 12.947s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch5.pth ...
Done in 16.602s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.3305803365707396
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 36.41851106639839
MSRVTT_full_val/t2v_metrics/R5: 71.42857142857143
MSRVTT_full_val/t2v_metrics/R10: 83.29979879275653
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.744466800804829
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.06378563254013
MSRVTT_full_val/v2t_metrics/R1: 39.23541247484909
MSRVTT_full_val/v2t_metrics/R5: 77.2635814889336
MSRVTT_full_val/v2t_metrics/R10: 88.53118712273641
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.553319919517103
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 64.50347548979175
MSRVTT_full_test/t2v_metrics/R1: 13.478260869565217
MSRVTT_full_test/t2v_metrics/R5: 37.625418060200666
MSRVTT_full_test/t2v_metrics/R10: 51.27090301003344
MSRVTT_full_test/t2v_metrics/R50: 81.50501672240803
MSRVTT_full_test/t2v_metrics/MedR: 10.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.91371237458194
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.625251970595997
MSRVTT_full_test/v2t_metrics/R1: 17.4247491638796
MSRVTT_full_test/v2t_metrics/R5: 45.18394648829432
MSRVTT_full_test/v2t_metrics/R10: 59.86622073578595
MSRVTT_full_test/v2t_metrics/R50: 87.35785953177258
MSRVTT_full_test/v2t_metrics/MedR: 7.0
MSRVTT_full_test/v2t_metrics/MeanR: 29.76789297658863
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.122476131482735
mnt_best : 29.625251970595997
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.37590 batch_time=34.86299
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 1.70622 batch_time=0.35416
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.22492 batch_time=0.34869
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.38516 batch_time=0.35475
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.27712 batch_time=0.34823
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.19187 batch_time=0.39372
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.33084 batch_time=0.39922
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.20422 batch_time=0.37421
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 1.86274 batch_time=0.36103
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.35544 batch_time=0.36658
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.28767 batch_time=0.37540
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 1.78660 batch_time=0.35446
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.01348 batch_time=0.37627
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.20323 batch_time=3.22938
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.03228 batch_time=0.39946
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 1.84828 batch_time=0.34938
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 1.89684 batch_time=0.36523
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.27636 batch_time=0.36105
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.82129 batch_time=0.47148
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.26908 batch_time=0.36427
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.24469 batch_time=0.40621
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.15095 batch_time=0.39681
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 1.68710 batch_time=0.48963
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch6.pth ...
Done in 3.847s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch6.pth ...
Done in 7.623s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 2.088572820186615
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 36.82092555331992
MSRVTT_full_val/t2v_metrics/R5: 70.62374245472837
MSRVTT_full_val/t2v_metrics/R10: 83.70221327967806
MSRVTT_full_val/t2v_metrics/R50: 98.39034205231388
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.712273641851106
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.15348456999551
MSRVTT_full_val/v2t_metrics/R1: 39.23541247484909
MSRVTT_full_val/v2t_metrics/R5: 78.47082494969818
MSRVTT_full_val/v2t_metrics/R10: 88.53118712273641
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.9959758551307845
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 64.83769635508094
MSRVTT_full_test/t2v_metrics/R1: 13.946488294314381
MSRVTT_full_test/t2v_metrics/R5: 37.92642140468227
MSRVTT_full_test/t2v_metrics/R10: 52.240802675585286
MSRVTT_full_test/t2v_metrics/R50: 81.97324414715719
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.78929765886288
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.23236981172158
MSRVTT_full_test/v2t_metrics/R1: 18.093645484949832
MSRVTT_full_test/v2t_metrics/R5: 45.25083612040134
MSRVTT_full_test/v2t_metrics/R10: 60.06688963210702
MSRVTT_full_test/v2t_metrics/R50: 87.72575250836121
MSRVTT_full_test/v2t_metrics/MedR: 7.0
MSRVTT_full_test/v2t_metrics/MeanR: 28.152173913043477
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.63779061686705
mnt_best : 30.23236981172158
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.11550 batch_time=31.54674
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.11066 batch_time=2.70931
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 1.91858 batch_time=0.35028
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.77661 batch_time=0.38506
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.37423 batch_time=0.38701
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.12317 batch_time=0.35063
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 1.92828 batch_time=1.04357
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 1.67286 batch_time=0.38271
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.59698 batch_time=0.37752
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.03885 batch_time=0.35355
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.14052 batch_time=0.38885
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.69351 batch_time=0.38638
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.68904 batch_time=0.36475
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.14512 batch_time=0.36461
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.38644 batch_time=0.35416
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 1.69872 batch_time=0.35657
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.55079 batch_time=0.39561
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 1.60286 batch_time=0.69318
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.83033 batch_time=0.35185
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.17490 batch_time=0.39161
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 1.99896 batch_time=0.37449
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.18048 batch_time=0.42145
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.96821 batch_time=0.42080
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch7.pth ...
Done in 4.115s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 1.9583240246772766
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 33.80281690140845
MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
MSRVTT_full_val/t2v_metrics/R10: 83.70221327967806
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.665995975855131
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.84566095051338
MSRVTT_full_val/v2t_metrics/R1: 40.84507042253521
MSRVTT_full_val/v2t_metrics/R5: 79.67806841046277
MSRVTT_full_val/v2t_metrics/R10: 89.33601609657947
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.537223340040241
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 66.24732438886666
MSRVTT_full_test/t2v_metrics/R1: 14.013377926421406
MSRVTT_full_test/t2v_metrics/R5: 37.290969899665555
MSRVTT_full_test/t2v_metrics/R10: 50.60200668896321
MSRVTT_full_test/t2v_metrics/R50: 81.77257525083613
MSRVTT_full_test/t2v_metrics/MedR: 10.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.02073578595318
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.792349294685433
MSRVTT_full_test/v2t_metrics/R1: 17.59197324414716
MSRVTT_full_test/v2t_metrics/R5: 45.752508361204015
MSRVTT_full_test/v2t_metrics/R10: 60.23411371237458
MSRVTT_full_test/v2t_metrics/R50: 87.95986622073579
MSRVTT_full_test/v2t_metrics/MedR: 7.0
MSRVTT_full_test/v2t_metrics/MeanR: 27.412207357859533
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.46341411812585
mnt_best : 30.23236981172158
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.87247 batch_time=34.79672
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 1.70943 batch_time=0.37300
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 1.94298 batch_time=0.46475
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.40946 batch_time=0.39879
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 1.88665 batch_time=0.39183
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.00947 batch_time=0.36061
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.49282 batch_time=0.35554
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.11854 batch_time=0.40468
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.86345 batch_time=0.38546
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.39765 batch_time=0.36896
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.99972 batch_time=0.42507
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.67331 batch_time=0.41982
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.03791 batch_time=0.36772
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.40719 batch_time=0.36200
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.19736 batch_time=0.37100
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.84562 batch_time=0.35705
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.86179 batch_time=0.41227
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.75618 batch_time=0.38209
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.54124 batch_time=0.42388
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.06887 batch_time=3.95208
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.86048 batch_time=0.40914
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.73168 batch_time=0.36993
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.71883 batch_time=0.35205
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch8.pth ...
Done in 3.849s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch8.pth ...
Done in 7.793s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 1.8325197048187256
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 34.00402414486921
MSRVTT_full_val/t2v_metrics/R5: 70.4225352112676
MSRVTT_full_val/t2v_metrics/R10: 85.51307847082495
MSRVTT_full_val/t2v_metrics/R50: 98.39034205231388
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.261569416498994
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 58.941995861578484
MSRVTT_full_val/v2t_metrics/R1: 43.05835010060362
MSRVTT_full_val/v2t_metrics/R5: 78.47082494969818
MSRVTT_full_val/v2t_metrics/R10: 88.93360160965794
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.154929577464789
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 66.97979713573632
MSRVTT_full_test/t2v_metrics/R1: 13.779264214046822
MSRVTT_full_test/t2v_metrics/R5: 38.79598662207358
MSRVTT_full_test/t2v_metrics/R10: 51.73913043478261
MSRVTT_full_test/t2v_metrics/R50: 82.20735785953177
MSRVTT_full_test/t2v_metrics/MedR: 10.0
MSRVTT_full_test/t2v_metrics/MeanR: 39.73076923076923
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.24200990689421
MSRVTT_full_test/v2t_metrics/R1: 19.264214046822744
MSRVTT_full_test/v2t_metrics/R5: 46.92307692307692
MSRVTT_full_test/v2t_metrics/R10: 62.0066889632107
MSRVTT_full_test/v2t_metrics/R50: 88.79598662207358
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 25.9438127090301
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.27002739394878
mnt_best : 30.24200990689421
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.63468 batch_time=30.69445
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.73555 batch_time=0.38863
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.29613 batch_time=0.40474
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.15619 batch_time=0.41523
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.85515 batch_time=0.34684
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.89603 batch_time=0.35170
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.71830 batch_time=2.19735
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.75111 batch_time=5.27463
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.55126 batch_time=0.37952
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.13304 batch_time=0.42250
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.40645 batch_time=0.35751
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.46882 batch_time=0.36376
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.63914 batch_time=0.37661
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.34586 batch_time=0.35821
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.60071 batch_time=0.36327
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.56036 batch_time=0.35960
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.41830 batch_time=0.38211
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.88409 batch_time=0.38464
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.22811 batch_time=0.35455
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.57723 batch_time=0.35621
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.47230 batch_time=0.92106
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.78853 batch_time=0.40701
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.90090 batch_time=0.40382
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch9.pth ...
Done in 4.031s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch9.pth ...
Done in 7.945s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.7225992321968078
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 34.80885311871227
MSRVTT_full_val/t2v_metrics/R5: 70.82494969818913
MSRVTT_full_val/t2v_metrics/R10: 84.50704225352112
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 6.754527162977867
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 59.28201591161986
MSRVTT_full_val/v2t_metrics/R1: 40.241448692152915
MSRVTT_full_val/v2t_metrics/R5: 78.06841046277665
MSRVTT_full_val/v2t_metrics/R10: 87.92756539235413
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.406438631790745
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 65.12653872048928
MSRVTT_full_test/t2v_metrics/R1: 15.351170568561873
MSRVTT_full_test/t2v_metrics/R5: 39.76588628762542
MSRVTT_full_test/t2v_metrics/R10: 54.48160535117057
MSRVTT_full_test/t2v_metrics/R50: 83.84615384615384
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 35.6314381270903
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.15886307009498
MSRVTT_full_test/v2t_metrics/R1: 18.561872909698998
MSRVTT_full_test/v2t_metrics/R5: 46.75585284280937
MSRVTT_full_test/v2t_metrics/R10: 61.30434782608695
MSRVTT_full_test/v2t_metrics/R50: 88.56187290969899
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 24.64448160535117
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.61112738596449
mnt_best : 32.15886307009498
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.40119 batch_time=32.14188
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.60046 batch_time=0.40072
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.76645 batch_time=0.37879
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.69591 batch_time=0.35674
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.66719 batch_time=0.41708
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.63197 batch_time=0.36167
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.45371 batch_time=0.40689
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.29562 batch_time=0.39095
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.18794 batch_time=0.36542
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.78989 batch_time=0.36009
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.64119 batch_time=0.35639
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.64678 batch_time=0.36194
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.60628 batch_time=0.35515
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.31029 batch_time=0.35490
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.79088 batch_time=0.38105
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.63864 batch_time=0.35723
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.30031 batch_time=0.39185
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.36162 batch_time=1.08815
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.55194 batch_time=0.41408
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.55091 batch_time=0.42895
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.35951 batch_time=0.81750
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.41010 batch_time=1.50350
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.30594 batch_time=0.38959
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch10.pth ...
Done in 4.047s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch10.pth ...
Done in 7.888s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.5956752858161927
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 36.82092555331992
MSRVTT_full_val/t2v_metrics/R5: 71.83098591549296
MSRVTT_full_val/t2v_metrics/R10: 83.70221327967806
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.160965794768612
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.49430478691261
MSRVTT_full_val/v2t_metrics/R1: 41.24748490945674
MSRVTT_full_val/v2t_metrics/R5: 78.67203219315896
MSRVTT_full_val/v2t_metrics/R10: 88.32997987927565
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.094567404426559
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 65.9338836660389
MSRVTT_full_test/t2v_metrics/R1: 15.986622073578594
MSRVTT_full_test/t2v_metrics/R5: 39.7324414715719
MSRVTT_full_test/t2v_metrics/R10: 53.21070234113712
MSRVTT_full_test/t2v_metrics/R50: 82.64214046822742
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 39.79397993311037
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.332080441546125
MSRVTT_full_test/v2t_metrics/R1: 18.862876254180602
MSRVTT_full_test/v2t_metrics/R5: 47.491638795986624
MSRVTT_full_test/v2t_metrics/R10: 61.87290969899666
MSRVTT_full_test/v2t_metrics/R50: 88.56187290969899
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 27.125752508361202
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.12781085898411
mnt_best : 32.332080441546125
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.76209 batch_time=32.88425
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.31438 batch_time=2.76682
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.62585 batch_time=0.41104
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.66302 batch_time=0.38365
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.62137 batch_time=0.38321
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.31944 batch_time=0.36052
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.54318 batch_time=0.85517
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.53832 batch_time=0.40528
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.32413 batch_time=0.40436
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.31531 batch_time=0.36486
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.62942 batch_time=0.36696
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.16876 batch_time=0.35503
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.40140 batch_time=0.97396
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.55030 batch_time=0.38372
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.29842 batch_time=0.37462
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 0.97424 batch_time=0.36152
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.42961 batch_time=0.38847
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.60989 batch_time=0.41531
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.52428 batch_time=0.38458
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.90706 batch_time=0.38116
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.21470 batch_time=0.60190
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.24966 batch_time=0.38635
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.66542 batch_time=0.37349
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch11.pth ...
Done in 4.089s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 1.5169617681503296
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 36.82092555331992
MSRVTT_full_val/t2v_metrics/R5: 72.83702213279678
MSRVTT_full_val/t2v_metrics/R10: 83.09859154929578
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.020120724346077
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.62896883498361
MSRVTT_full_val/v2t_metrics/R1: 40.44265593561368
MSRVTT_full_val/v2t_metrics/R5: 80.6841046277666
MSRVTT_full_val/v2t_metrics/R10: 89.13480885311871
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.462776659959759
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 66.25596697526895
MSRVTT_full_test/t2v_metrics/R1: 15.250836120401338
MSRVTT_full_test/t2v_metrics/R5: 39.4314381270903
MSRVTT_full_test/t2v_metrics/R10: 53.24414715719063
MSRVTT_full_test/t2v_metrics/R50: 82.64214046822742
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.48160535117057
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.7543125699226
MSRVTT_full_test/v2t_metrics/R1: 20.334448160535118
MSRVTT_full_test/v2t_metrics/R5: 49.66555183946488
MSRVTT_full_test/v2t_metrics/R10: 63.24414715719063
MSRVTT_full_test/v2t_metrics/R50: 88.66220735785953
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 26.070234113712374
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.973238267436734
mnt_best : 32.332080441546125
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 0.88442 batch_time=35.60072
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.40199 batch_time=0.38931
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.77905 batch_time=0.40598
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.47758 batch_time=0.45859
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.54267 batch_time=0.36572
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.25188 batch_time=0.36956
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.21964 batch_time=0.36604
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.50344 batch_time=0.35799
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.72205 batch_time=0.37245
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.55445 batch_time=0.40102
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.51163 batch_time=0.44707
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.55920 batch_time=0.35681
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.85176 batch_time=0.41125
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.34485 batch_time=0.35379
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.50081 batch_time=0.41399
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.34676 batch_time=0.37511
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.37122 batch_time=0.39812
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.73737 batch_time=0.39382
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.41220 batch_time=0.41338
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.03015 batch_time=1.07811
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.38835 batch_time=0.36425
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.31658 batch_time=0.35917
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.65438 batch_time=0.36630
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch12.pth ...
Done in 4.287s
removing stale ckpt [epoch 11] [took 1.50s]
epoch : 12
loss : 1.451134897708893
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 37.42454728370221
MSRVTT_full_val/t2v_metrics/R5: 72.43460764587525
MSRVTT_full_val/t2v_metrics/R10: 84.90945674044265
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.158953722334004
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 61.28481930132889
MSRVTT_full_val/v2t_metrics/R1: 41.4486921529175
MSRVTT_full_val/v2t_metrics/R5: 79.87927565392354
MSRVTT_full_val/v2t_metrics/R10: 90.14084507042253
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.706237424547283
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 66.82754776263303
MSRVTT_full_test/t2v_metrics/R1: 15.418060200668897
MSRVTT_full_test/t2v_metrics/R5: 40.13377926421405
MSRVTT_full_test/t2v_metrics/R10: 53.979933110367895
MSRVTT_full_test/t2v_metrics/R50: 82.70903010033445
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 39.4752508361204
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.20505503212437
MSRVTT_full_test/v2t_metrics/R1: 20.301003344481604
MSRVTT_full_test/v2t_metrics/R5: 49.163879598662206
MSRVTT_full_test/v2t_metrics/R10: 62.14046822742475
MSRVTT_full_test/v2t_metrics/R50: 88.69565217391305
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 26.922408026755853
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.58336613838038
mnt_best : 32.332080441546125
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.35328 batch_time=35.89511
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.59799 batch_time=0.42938
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.12061 batch_time=0.40093
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.05670 batch_time=0.37411
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.26798 batch_time=0.35111
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.36505 batch_time=0.36105
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 0.97422 batch_time=0.36775
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.56999 batch_time=0.42042
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.58823 batch_time=0.38438
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.19759 batch_time=0.35867
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.67454 batch_time=0.41389
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.15682 batch_time=0.39503
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.28166 batch_time=0.38200
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.35829 batch_time=0.37342
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.18326 batch_time=0.36676
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.76092 batch_time=0.39913
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.26697 batch_time=0.36494
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.64678 batch_time=0.45218
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.57070 batch_time=0.36463
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.51337 batch_time=0.35888
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.62247 batch_time=0.37147
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.21415 batch_time=0.40067
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.22662 batch_time=1.15101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch13.pth ...
Done in 4.825s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch13.pth ...
Done in 9.801s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 1.3926355829238892
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 35.814889336016094
MSRVTT_full_val/t2v_metrics/R5: 72.63581488933602
MSRVTT_full_val/t2v_metrics/R10: 83.90342052313883
MSRVTT_full_val/t2v_metrics/R50: 98.59154929577464
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.040241448692153
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.20945521574318
MSRVTT_full_val/v2t_metrics/R1: 42.25352112676056
MSRVTT_full_val/v2t_metrics/R5: 78.47082494969818
MSRVTT_full_val/v2t_metrics/R10: 88.93360160965794
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.778672032193159
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 66.55985006388225
MSRVTT_full_test/t2v_metrics/R1: 15.217391304347826
MSRVTT_full_test/t2v_metrics/R5: 41.438127090301
MSRVTT_full_test/t2v_metrics/R10: 54.94983277591973
MSRVTT_full_test/t2v_metrics/R50: 82.90969899665552
MSRVTT_full_test/t2v_metrics/MedR: 8.0
MSRVTT_full_test/t2v_metrics/MeanR: 37.58361204013378
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.60134871499456
MSRVTT_full_test/v2t_metrics/R1: 20.367892976588628
MSRVTT_full_test/v2t_metrics/R5: 48.49498327759197
MSRVTT_full_test/v2t_metrics/R10: 62.90969899665552
MSRVTT_full_test/v2t_metrics/R50: 89.09698996655518
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 25.55819397993311
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.60835828315498
mnt_best : 32.60134871499456
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.45926 batch_time=43.44571
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.53937 batch_time=0.36063
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.65829 batch_time=0.37637
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.37833 batch_time=0.35284
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.07854 batch_time=0.36126
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.07684 batch_time=0.37896
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.34913 batch_time=0.35511
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.18798 batch_time=0.36592
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.27401 batch_time=0.35476
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.24507 batch_time=0.35114
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.58575 batch_time=0.34968
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.32005 batch_time=0.35383
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.17010 batch_time=0.37289
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.29823 batch_time=0.38685
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.32567 batch_time=0.35738
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.26082 batch_time=0.35215
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.16027 batch_time=0.35745
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.18403 batch_time=0.35392
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.66848 batch_time=1.83327
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.13396 batch_time=0.81061
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.28424 batch_time=0.39815
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.54613 batch_time=1.15458
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.58163 batch_time=0.38883
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch14.pth ...
Done in 4.683s
removing stale ckpt [epoch 13] [took 0.20s]
epoch : 14
loss : 1.3318395845890045
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 37.223340040241446
MSRVTT_full_val/t2v_metrics/R5: 71.62977867203219
MSRVTT_full_val/t2v_metrics/R10: 84.30583501006036
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.45271629778672
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 60.80260611835894
MSRVTT_full_val/v2t_metrics/R1: 42.65593561368209
MSRVTT_full_val/v2t_metrics/R5: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R10: 89.53722334004024
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.716297786720322
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 67.43207654248786
MSRVTT_full_test/t2v_metrics/R1: 15.016722408026755
MSRVTT_full_test/t2v_metrics/R5: 39.89966555183946
MSRVTT_full_test/t2v_metrics/R10: 53.54515050167224
MSRVTT_full_test/t2v_metrics/R50: 82.47491638795987
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.018729096989965
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.775192031949775
MSRVTT_full_test/v2t_metrics/R1: 20.066889632107024
MSRVTT_full_test/v2t_metrics/R5: 47.759197324414714
MSRVTT_full_test/v2t_metrics/R10: 62.709030100334445
MSRVTT_full_test/v2t_metrics/R50: 88.72909698996655
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 26.54515050167224
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.17019392197179
mnt_best : 32.60134871499456
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.34414 batch_time=32.33859
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 0.93068 batch_time=0.37299
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.24263 batch_time=0.38640
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.70051 batch_time=0.35793
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.00212 batch_time=0.39998
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.47439 batch_time=0.36297
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.61806 batch_time=0.40339
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.30145 batch_time=0.40609
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.19730 batch_time=0.39498
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.30213 batch_time=0.41900
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.39225 batch_time=0.43263
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.28720 batch_time=0.38353
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.60629 batch_time=0.37390
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.05778 batch_time=0.37077
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 0.98036 batch_time=0.37004
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.23869 batch_time=0.36829
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.44247 batch_time=0.37480
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 0.98237 batch_time=0.37343
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.35543 batch_time=0.35154
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.16847 batch_time=1.45196
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.36788 batch_time=0.64206
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.62596 batch_time=0.35407
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.58023 batch_time=0.38443
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch15.pth ...
Done in 19.628s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_MSRVTT_full/checkpoint-epoch15.pth ...
Done in 25.418s
removing stale ckpt [epoch 14] [took 0.17s]
epoch : 15
loss : 1.2811736540794372
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 37.625754527162975
MSRVTT_full_val/t2v_metrics/R5: 71.62977867203219
MSRVTT_full_val/t2v_metrics/R10: 84.70824949698189
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 2.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.315895372233401
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 61.117865451601816
MSRVTT_full_val/v2t_metrics/R1: 39.436619718309856
MSRVTT_full_val/v2t_metrics/R5: 79.47686116700201
MSRVTT_full_val/v2t_metrics/R10: 89.13480885311871
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.885311871227364
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 65.37262271830608
MSRVTT_full_test/t2v_metrics/R1: 15.852842809364548
MSRVTT_full_test/t2v_metrics/R5: 41.438127090301
MSRVTT_full_test/t2v_metrics/R10: 53.812709030100336
MSRVTT_full_test/t2v_metrics/R50: 82.80936454849498
MSRVTT_full_test/t2v_metrics/MedR: 9.0
MSRVTT_full_test/t2v_metrics/MeanR: 39.352842809364546
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.81940543980636
MSRVTT_full_test/v2t_metrics/R1: 20.401337792642142
MSRVTT_full_test/v2t_metrics/R5: 49.66555183946488
MSRVTT_full_test/v2t_metrics/R10: 64.14715719063545
MSRVTT_full_test/v2t_metrics/R50: 89.36454849498328
MSRVTT_full_test/v2t_metrics/MedR: 6.0
MSRVTT_full_test/v2t_metrics/MeanR: 25.67792642140468
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.206577737571564
mnt_best : 32.81940543980636
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.26340 batch_time=34.68544
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.42872 batch_time=0.35594
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.48197 batch_time=0.40606
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.38361 batch_time=0.41927
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.23062 batch_time=0.40597
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.21022 batch_time=0.38590
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.00411 batch_time=0.35231
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.51722 batch_time=0.37654
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.34313 batch_time=0.36079
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.08166 batch_time=0.35064
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.37715 batch_time=0.37620
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.35942 batch_time=0.38572
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.15153 batch_time=0.52264