-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_bs64.txt
5315 lines (5315 loc) · 464 KB
/
HCQ_MSRVTT_full_bs64.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 649.5171322822571 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 34.05401158332825 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 182.97837901115417 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 163.49362683296204 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch0.pth ...
Done in 15.029s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch0.pth ...
Done in 16.602s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5290969899665
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/500 64/32000 (0%)] Loss: 8.43812 (QuantReg: 22.43759) QuantErr: 22.43759 batch_time=27.87996
Train Epoch: 1 [9/500 576/32000 (2%)] Loss: 7.86671 (QuantReg: 22.45715) QuantErr: 22.45715 batch_time=0.45899
Train Epoch: 1 [17/500 1088/32000 (3%)] Loss: 7.17673 (QuantReg: 22.54086) QuantErr: 22.54086 batch_time=0.38768
Train Epoch: 1 [25/500 1600/32000 (5%)] Loss: 6.34603 (QuantReg: 22.56167) QuantErr: 22.56167 batch_time=0.39327
Train Epoch: 1 [33/500 2112/32000 (7%)] Loss: 6.04188 (QuantReg: 22.62214) QuantErr: 22.62214 batch_time=2.43885
Train Epoch: 1 [41/500 2624/32000 (8%)] Loss: 5.58680 (QuantReg: 22.61850) QuantErr: 22.61850 batch_time=0.38928
Train Epoch: 1 [49/500 3136/32000 (10%)] Loss: 5.05479 (QuantReg: 22.61581) QuantErr: 22.61581 batch_time=0.38589
Train Epoch: 1 [57/500 3648/32000 (11%)] Loss: 5.06972 (QuantReg: 22.63078) QuantErr: 22.63078 batch_time=0.40591
Train Epoch: 1 [65/500 4160/32000 (13%)] Loss: 4.80396 (QuantReg: 22.68736) QuantErr: 22.68736 batch_time=0.42898
Train Epoch: 1 [73/500 4672/32000 (15%)] Loss: 5.03449 (QuantReg: 22.61640) QuantErr: 22.61640 batch_time=0.49692
Train Epoch: 1 [81/500 5184/32000 (16%)] Loss: 4.07466 (QuantReg: 22.64266) QuantErr: 22.64266 batch_time=0.38338
Train Epoch: 1 [89/500 5696/32000 (18%)] Loss: 4.56333 (QuantReg: 22.64845) QuantErr: 22.64845 batch_time=0.38644
Train Epoch: 1 [97/500 6208/32000 (19%)] Loss: 4.37364 (QuantReg: 22.65076) QuantErr: 22.65076 batch_time=1.90739
Train Epoch: 1 [105/500 6720/32000 (21%)] Loss: 4.25968 (QuantReg: 22.68264) QuantErr: 22.68264 batch_time=0.43072
Train Epoch: 1 [113/500 7232/32000 (23%)] Loss: 4.44467 (QuantReg: 22.69199) QuantErr: 22.69199 batch_time=0.39206
Train Epoch: 1 [121/500 7744/32000 (24%)] Loss: 4.70315 (QuantReg: 22.66682) QuantErr: 22.66682 batch_time=0.40118
Train Epoch: 1 [129/500 8256/32000 (26%)] Loss: 3.73010 (QuantReg: 22.64540) QuantErr: 22.64540 batch_time=0.39252
Train Epoch: 1 [137/500 8768/32000 (27%)] Loss: 4.18199 (QuantReg: 22.67181) QuantErr: 22.67181 batch_time=0.39048
Train Epoch: 1 [145/500 9280/32000 (29%)] Loss: 4.09839 (QuantReg: 22.64741) QuantErr: 22.64741 batch_time=0.43396
Train Epoch: 1 [153/500 9792/32000 (31%)] Loss: 3.72685 (QuantReg: 22.61679) QuantErr: 22.61679 batch_time=0.39160
Train Epoch: 1 [161/500 10304/32000 (32%)] Loss: 4.04095 (QuantReg: 22.66866) QuantErr: 22.66866 batch_time=0.66094
Train Epoch: 1 [169/500 10816/32000 (34%)] Loss: 3.90190 (QuantReg: 22.62306) QuantErr: 22.62306 batch_time=0.39225
Train Epoch: 1 [177/500 11328/32000 (35%)] Loss: 3.13213 (QuantReg: 22.63277) QuantErr: 22.63277 batch_time=0.39455
Train Epoch: 1 [185/500 11840/32000 (37%)] Loss: 4.12965 (QuantReg: 22.66827) QuantErr: 22.66827 batch_time=0.38693
Train Epoch: 1 [193/500 12352/32000 (39%)] Loss: 3.46408 (QuantReg: 22.69338) QuantErr: 22.69338 batch_time=0.49677
Train Epoch: 1 [201/500 12864/32000 (40%)] Loss: 3.73195 (QuantReg: 22.67654) QuantErr: 22.67654 batch_time=0.38712
Train Epoch: 1 [209/500 13376/32000 (42%)] Loss: 4.35894 (QuantReg: 22.64442) QuantErr: 22.64442 batch_time=0.40131
Train Epoch: 1 [217/500 13888/32000 (43%)] Loss: 3.63338 (QuantReg: 22.69431) QuantErr: 22.69431 batch_time=0.42981
Train Epoch: 1 [225/500 14400/32000 (45%)] Loss: 3.52637 (QuantReg: 22.68792) QuantErr: 22.68792 batch_time=0.40458
Train Epoch: 1 [233/500 14912/32000 (47%)] Loss: 3.89610 (QuantReg: 22.65108) QuantErr: 22.65108 batch_time=0.43784
Train Epoch: 1 [241/500 15424/32000 (48%)] Loss: 3.17873 (QuantReg: 22.68122) QuantErr: 22.68122 batch_time=0.40678
Train Epoch: 1 [249/500 15936/32000 (50%)] Loss: 3.41487 (QuantReg: 22.65787) QuantErr: 22.65787 batch_time=0.39149
Train Epoch: 1 [257/500 16448/32000 (51%)] Loss: 3.65602 (QuantReg: 22.71299) QuantErr: 22.71299 batch_time=0.40272
Train Epoch: 1 [265/500 16960/32000 (53%)] Loss: 3.29337 (QuantReg: 22.69943) QuantErr: 22.69943 batch_time=0.40049
Train Epoch: 1 [273/500 17472/32000 (55%)] Loss: 2.98783 (QuantReg: 22.60728) QuantErr: 22.60728 batch_time=0.50104
Train Epoch: 1 [281/500 17984/32000 (56%)] Loss: 3.19987 (QuantReg: 22.72689) QuantErr: 22.72689 batch_time=0.40222
Train Epoch: 1 [289/500 18496/32000 (58%)] Loss: 2.75551 (QuantReg: 22.61244) QuantErr: 22.61244 batch_time=0.40087
Train Epoch: 1 [297/500 19008/32000 (59%)] Loss: 3.88758 (QuantReg: 22.68275) QuantErr: 22.68275 batch_time=0.39273
Train Epoch: 1 [305/500 19520/32000 (61%)] Loss: 2.78415 (QuantReg: 22.66384) QuantErr: 22.66384 batch_time=0.41084
Train Epoch: 1 [313/500 20032/32000 (63%)] Loss: 3.10222 (QuantReg: 22.70513) QuantErr: 22.70513 batch_time=0.39195
Train Epoch: 1 [321/500 20544/32000 (64%)] Loss: 3.43994 (QuantReg: 22.67940) QuantErr: 22.67940 batch_time=0.39041
Train Epoch: 1 [329/500 21056/32000 (66%)] Loss: 3.06995 (QuantReg: 22.73015) QuantErr: 22.73015 batch_time=0.40382
Train Epoch: 1 [337/500 21568/32000 (67%)] Loss: 4.04827 (QuantReg: 22.66801) QuantErr: 22.66801 batch_time=0.43552
Train Epoch: 1 [345/500 22080/32000 (69%)] Loss: 3.56386 (QuantReg: 22.63121) QuantErr: 22.63121 batch_time=0.39125
Train Epoch: 1 [353/500 22592/32000 (71%)] Loss: 3.06678 (QuantReg: 22.68566) QuantErr: 22.68566 batch_time=0.39495
Train Epoch: 1 [361/500 23104/32000 (72%)] Loss: 2.81924 (QuantReg: 22.70088) QuantErr: 22.70088 batch_time=0.40698
Train Epoch: 1 [369/500 23616/32000 (74%)] Loss: 3.25984 (QuantReg: 22.64950) QuantErr: 22.64950 batch_time=0.39665
Train Epoch: 1 [377/500 24128/32000 (75%)] Loss: 3.10946 (QuantReg: 22.67654) QuantErr: 22.67654 batch_time=0.39715
Train Epoch: 1 [385/500 24640/32000 (77%)] Loss: 3.22354 (QuantReg: 22.67923) QuantErr: 22.67923 batch_time=0.43401
Train Epoch: 1 [393/500 25152/32000 (79%)] Loss: 3.39545 (QuantReg: 22.71891) QuantErr: 22.71891 batch_time=0.39396
Train Epoch: 1 [401/500 25664/32000 (80%)] Loss: 3.14739 (QuantReg: 22.70868) QuantErr: 22.70868 batch_time=0.38997
Train Epoch: 1 [409/500 26176/32000 (82%)] Loss: 2.59390 (QuantReg: 22.63985) QuantErr: 22.63985 batch_time=0.40841
Train Epoch: 1 [417/500 26688/32000 (83%)] Loss: 3.91234 (QuantReg: 22.68909) QuantErr: 22.68909 batch_time=0.40390
Train Epoch: 1 [425/500 27200/32000 (85%)] Loss: 2.61363 (QuantReg: 22.64343) QuantErr: 22.64343 batch_time=0.41446
Train Epoch: 1 [433/500 27712/32000 (87%)] Loss: 2.49760 (QuantReg: 22.68137) QuantErr: 22.68137 batch_time=0.40178
Train Epoch: 1 [441/500 28224/32000 (88%)] Loss: 2.83490 (QuantReg: 22.62788) QuantErr: 22.62788 batch_time=0.39304
Train Epoch: 1 [449/500 28736/32000 (90%)] Loss: 2.76408 (QuantReg: 22.65774) QuantErr: 22.65774 batch_time=0.39722
Train Epoch: 1 [457/500 29248/32000 (91%)] Loss: 2.57579 (QuantReg: 22.61464) QuantErr: 22.61464 batch_time=0.39048
Train Epoch: 1 [465/500 29760/32000 (93%)] Loss: 3.44080 (QuantReg: 22.59327) QuantErr: 22.59327 batch_time=0.40418
Train Epoch: 1 [473/500 30272/32000 (95%)] Loss: 2.35722 (QuantReg: 22.71026) QuantErr: 22.71026 batch_time=0.41946
Train Epoch: 1 [481/500 30784/32000 (96%)] Loss: 2.91950 (QuantReg: 22.67534) QuantErr: 22.67534 batch_time=0.39290
Train Epoch: 1 [489/500 31296/32000 (98%)] Loss: 2.89775 (QuantReg: 22.61932) QuantErr: 22.61932 batch_time=0.40013
Train Epoch: 1 [497/500 31808/32000 (99%)] Loss: 2.72798 (QuantReg: 22.67437) QuantErr: 22.67437 batch_time=0.39619
Train Epoch: 1 codebook_update_time=2.75309
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch1.pth ...
Done in 4.361s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch1.pth ...
Done in 9.485s
epoch : 1
loss : 3.849457883834839
quant_reg : 22.65250672531128
quant_err : 22.65250672531128
learning_rate : 5e-05
n_samples : 32000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 19.3158953722334
MSRVTT_full_val/t2v_metrics/R5: 54.32595573440644
MSRVTT_full_val/t2v_metrics/R10: 68.81287726358148
MSRVTT_full_val/t2v_metrics/R50: 94.96981891348088
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 13.195171026156942
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 41.64191043098871
MSRVTT_full_val/v2t_metrics/R1: 20.120724346076457
MSRVTT_full_val/v2t_metrics/R5: 58.14889336016097
MSRVTT_full_val/v2t_metrics/R10: 74.84909456740442
MSRVTT_full_val/v2t_metrics/R50: 94.76861167002012
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 12.587525150905433
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 44.407590084736555
MSRVTT_full_test/t2v_metrics/R1: 6.722408026755853
MSRVTT_full_test/t2v_metrics/R5: 22.34113712374582
MSRVTT_full_test/t2v_metrics/R10: 33.17725752508361
MSRVTT_full_test/t2v_metrics/R50: 67.85953177257525
MSRVTT_full_test/t2v_metrics/MedR: 22.0
MSRVTT_full_test/t2v_metrics/MeanR: 67.31605351170569
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.08009213323801
MSRVTT_full_test/v2t_metrics/R1: 7.792642140468227
MSRVTT_full_test/v2t_metrics/R5: 25.016722408026755
MSRVTT_full_test/v2t_metrics/R10: 37.69230769230769
MSRVTT_full_test/v2t_metrics/R50: 71.87290969899665
MSRVTT_full_test/v2t_metrics/MedR: 19.0
MSRVTT_full_test/v2t_metrics/MeanR: 61.93277591973244
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.44118009107646
mnt_best : 17.08009213323801
not_improved_count: 0
Train Epoch: 2 [1/500 64/32000 (0%)] Loss: 2.53605 (QuantReg: 11.63172) QuantErr: 11.63172 batch_time=29.86650
Train Epoch: 2 [9/500 576/32000 (2%)] Loss: 2.65572 (QuantReg: 11.63410) QuantErr: 11.63410 batch_time=0.39064
Train Epoch: 2 [17/500 1088/32000 (3%)] Loss: 2.62144 (QuantReg: 12.11064) QuantErr: 12.11064 batch_time=0.40844
Train Epoch: 2 [25/500 1600/32000 (5%)] Loss: 2.90449 (QuantReg: 11.86863) QuantErr: 11.86863 batch_time=0.39697
Train Epoch: 2 [33/500 2112/32000 (7%)] Loss: 2.87888 (QuantReg: 11.74395) QuantErr: 11.74395 batch_time=0.40302
Train Epoch: 2 [41/500 2624/32000 (8%)] Loss: 3.07237 (QuantReg: 12.29843) QuantErr: 12.29843 batch_time=0.39783
Train Epoch: 2 [49/500 3136/32000 (10%)] Loss: 3.59839 (QuantReg: 12.63230) QuantErr: 12.63230 batch_time=0.39187
Train Epoch: 2 [57/500 3648/32000 (11%)] Loss: 2.03922 (QuantReg: 12.33534) QuantErr: 12.33534 batch_time=0.38698
Train Epoch: 2 [65/500 4160/32000 (13%)] Loss: 2.90109 (QuantReg: 12.04460) QuantErr: 12.04460 batch_time=0.40366
Train Epoch: 2 [73/500 4672/32000 (15%)] Loss: 2.64196 (QuantReg: 12.23916) QuantErr: 12.23916 batch_time=0.42171
Train Epoch: 2 [81/500 5184/32000 (16%)] Loss: 2.43274 (QuantReg: 11.86896) QuantErr: 11.86896 batch_time=0.39732
Train Epoch: 2 [89/500 5696/32000 (18%)] Loss: 3.05322 (QuantReg: 12.15937) QuantErr: 12.15937 batch_time=0.39606
Train Epoch: 2 [97/500 6208/32000 (19%)] Loss: 2.94253 (QuantReg: 12.30458) QuantErr: 12.30458 batch_time=0.39852
Train Epoch: 2 [105/500 6720/32000 (21%)] Loss: 2.73018 (QuantReg: 12.05891) QuantErr: 12.05891 batch_time=0.39027
Train Epoch: 2 [113/500 7232/32000 (23%)] Loss: 2.41126 (QuantReg: 12.72122) QuantErr: 12.72122 batch_time=0.43747
Train Epoch: 2 [121/500 7744/32000 (24%)] Loss: 2.63221 (QuantReg: 12.45748) QuantErr: 12.45748 batch_time=0.39216
Train Epoch: 2 [129/500 8256/32000 (26%)] Loss: 2.51456 (QuantReg: 12.31284) QuantErr: 12.31284 batch_time=0.39730
Train Epoch: 2 [137/500 8768/32000 (27%)] Loss: 2.95940 (QuantReg: 12.79674) QuantErr: 12.79674 batch_time=0.41175
Train Epoch: 2 [145/500 9280/32000 (29%)] Loss: 2.21811 (QuantReg: 12.76750) QuantErr: 12.76750 batch_time=0.44208
Train Epoch: 2 [153/500 9792/32000 (31%)] Loss: 2.70480 (QuantReg: 12.50456) QuantErr: 12.50456 batch_time=0.38894
Train Epoch: 2 [161/500 10304/32000 (32%)] Loss: 1.76672 (QuantReg: 12.85682) QuantErr: 12.85682 batch_time=0.39970
Train Epoch: 2 [169/500 10816/32000 (34%)] Loss: 3.08564 (QuantReg: 13.06790) QuantErr: 13.06790 batch_time=0.39517
Train Epoch: 2 [177/500 11328/32000 (35%)] Loss: 2.73130 (QuantReg: 13.19134) QuantErr: 13.19134 batch_time=0.39277
Train Epoch: 2 [185/500 11840/32000 (37%)] Loss: 2.36597 (QuantReg: 13.34442) QuantErr: 13.34442 batch_time=0.39032
Train Epoch: 2 [193/500 12352/32000 (39%)] Loss: 2.02484 (QuantReg: 13.20122) QuantErr: 13.20122 batch_time=0.38541
Train Epoch: 2 [201/500 12864/32000 (40%)] Loss: 2.49385 (QuantReg: 14.01521) QuantErr: 14.01521 batch_time=0.39422
Train Epoch: 2 [209/500 13376/32000 (42%)] Loss: 2.32300 (QuantReg: 13.56448) QuantErr: 13.56448 batch_time=0.38705
Train Epoch: 2 [217/500 13888/32000 (43%)] Loss: 2.85885 (QuantReg: 13.30222) QuantErr: 13.30222 batch_time=0.39559
Train Epoch: 2 [225/500 14400/32000 (45%)] Loss: 2.49944 (QuantReg: 13.31543) QuantErr: 13.31543 batch_time=0.42000
Train Epoch: 2 [233/500 14912/32000 (47%)] Loss: 2.39026 (QuantReg: 13.56131) QuantErr: 13.56131 batch_time=0.63560
Train Epoch: 2 [241/500 15424/32000 (48%)] Loss: 3.05074 (QuantReg: 13.24188) QuantErr: 13.24188 batch_time=0.38901
Train Epoch: 2 [249/500 15936/32000 (50%)] Loss: 3.23537 (QuantReg: 12.98922) QuantErr: 12.98922 batch_time=0.39870
Train Epoch: 2 [257/500 16448/32000 (51%)] Loss: 2.72793 (QuantReg: 13.34555) QuantErr: 13.34555 batch_time=0.40146
Train Epoch: 2 [265/500 16960/32000 (53%)] Loss: 2.15982 (QuantReg: 13.60440) QuantErr: 13.60440 batch_time=0.40925
Train Epoch: 2 [273/500 17472/32000 (55%)] Loss: 2.23662 (QuantReg: 13.60143) QuantErr: 13.60143 batch_time=0.39668
Train Epoch: 2 [281/500 17984/32000 (56%)] Loss: 2.97562 (QuantReg: 13.46586) QuantErr: 13.46586 batch_time=0.39510
Train Epoch: 2 [289/500 18496/32000 (58%)] Loss: 2.10177 (QuantReg: 13.60669) QuantErr: 13.60669 batch_time=0.40311
Train Epoch: 2 [297/500 19008/32000 (59%)] Loss: 2.34748 (QuantReg: 13.60603) QuantErr: 13.60603 batch_time=0.39944
Train Epoch: 2 [305/500 19520/32000 (61%)] Loss: 2.29499 (QuantReg: 13.35366) QuantErr: 13.35366 batch_time=0.42528
Train Epoch: 2 [313/500 20032/32000 (63%)] Loss: 2.88125 (QuantReg: 13.33046) QuantErr: 13.33046 batch_time=0.42345
Train Epoch: 2 [321/500 20544/32000 (64%)] Loss: 2.26759 (QuantReg: 13.92188) QuantErr: 13.92188 batch_time=0.40381
Train Epoch: 2 [329/500 21056/32000 (66%)] Loss: 2.04968 (QuantReg: 13.61777) QuantErr: 13.61777 batch_time=0.40090
Train Epoch: 2 [337/500 21568/32000 (67%)] Loss: 2.24272 (QuantReg: 14.08984) QuantErr: 14.08984 batch_time=0.43528
Train Epoch: 2 [345/500 22080/32000 (69%)] Loss: 2.45271 (QuantReg: 13.60779) QuantErr: 13.60779 batch_time=0.39856
Train Epoch: 2 [353/500 22592/32000 (71%)] Loss: 2.54965 (QuantReg: 14.12436) QuantErr: 14.12436 batch_time=0.40204
Train Epoch: 2 [361/500 23104/32000 (72%)] Loss: 2.31628 (QuantReg: 14.35352) QuantErr: 14.35352 batch_time=0.39221
Train Epoch: 2 [369/500 23616/32000 (74%)] Loss: 2.22895 (QuantReg: 13.74874) QuantErr: 13.74874 batch_time=0.38267
Train Epoch: 2 [377/500 24128/32000 (75%)] Loss: 2.44795 (QuantReg: 14.07456) QuantErr: 14.07456 batch_time=0.39238
Train Epoch: 2 [385/500 24640/32000 (77%)] Loss: 2.52951 (QuantReg: 13.97346) QuantErr: 13.97346 batch_time=0.40466
Train Epoch: 2 [393/500 25152/32000 (79%)] Loss: 2.02496 (QuantReg: 13.97087) QuantErr: 13.97087 batch_time=0.40363
Train Epoch: 2 [401/500 25664/32000 (80%)] Loss: 2.53389 (QuantReg: 13.51935) QuantErr: 13.51935 batch_time=0.47465
Train Epoch: 2 [409/500 26176/32000 (82%)] Loss: 2.55643 (QuantReg: 14.07141) QuantErr: 14.07141 batch_time=0.43868
Train Epoch: 2 [417/500 26688/32000 (83%)] Loss: 2.21421 (QuantReg: 14.07287) QuantErr: 14.07287 batch_time=0.41080
Train Epoch: 2 [425/500 27200/32000 (85%)] Loss: 2.49372 (QuantReg: 13.97935) QuantErr: 13.97935 batch_time=0.40241
Train Epoch: 2 [433/500 27712/32000 (87%)] Loss: 1.98306 (QuantReg: 14.05542) QuantErr: 14.05542 batch_time=0.39267
Train Epoch: 2 [441/500 28224/32000 (88%)] Loss: 2.95604 (QuantReg: 13.92814) QuantErr: 13.92814 batch_time=0.40508
Train Epoch: 2 [449/500 28736/32000 (90%)] Loss: 2.12576 (QuantReg: 14.23887) QuantErr: 14.23887 batch_time=0.40700
Train Epoch: 2 [457/500 29248/32000 (91%)] Loss: 2.25334 (QuantReg: 14.17091) QuantErr: 14.17091 batch_time=0.43724
Train Epoch: 2 [465/500 29760/32000 (93%)] Loss: 2.74878 (QuantReg: 14.38042) QuantErr: 14.38042 batch_time=0.39757
Train Epoch: 2 [473/500 30272/32000 (95%)] Loss: 2.00447 (QuantReg: 14.20713) QuantErr: 14.20713 batch_time=0.41819
Train Epoch: 2 [481/500 30784/32000 (96%)] Loss: 2.22458 (QuantReg: 14.07179) QuantErr: 14.07179 batch_time=0.39057
Train Epoch: 2 [489/500 31296/32000 (98%)] Loss: 2.14161 (QuantReg: 14.71571) QuantErr: 14.71571 batch_time=0.44881
Train Epoch: 2 [497/500 31808/32000 (99%)] Loss: 1.81142 (QuantReg: 14.52993) QuantErr: 14.52993 batch_time=0.40740
Train Epoch: 2 codebook_update_time=2.41499
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch2.pth ...
Done in 21.615s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch2.pth ...
Done in 25.614s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 2.5091844520568847
quant_reg : 13.311342796325684
quant_err : 13.311342796325684
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 19.718309859154928
MSRVTT_full_val/t2v_metrics/R5: 57.142857142857146
MSRVTT_full_val/t2v_metrics/R10: 72.03219315895372
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.849094567404427
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 43.296496852473936
MSRVTT_full_val/v2t_metrics/R1: 25.35211267605634
MSRVTT_full_val/v2t_metrics/R5: 62.17303822937626
MSRVTT_full_val/v2t_metrics/R10: 76.25754527162978
MSRVTT_full_val/v2t_metrics/R50: 95.97585513078471
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.509054325955734
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.351424112964175
MSRVTT_full_test/t2v_metrics/R1: 8.361204013377927
MSRVTT_full_test/t2v_metrics/R5: 25.016722408026755
MSRVTT_full_test/t2v_metrics/R10: 37.25752508361204
MSRVTT_full_test/t2v_metrics/R50: 73.17725752508362
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.824414715719065
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.82612062374619
MSRVTT_full_test/v2t_metrics/R1: 9.297658862876254
MSRVTT_full_test/v2t_metrics/R5: 28.762541806020067
MSRVTT_full_test/v2t_metrics/R10: 41.77257525083612
MSRVTT_full_test/v2t_metrics/R50: 77.123745819398
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 49.81973244147157
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.35445262825209
mnt_best : 19.82612062374619
not_improved_count: 0
Train Epoch: 3 [1/500 64/32000 (0%)] Loss: 2.25649 (QuantReg: 11.82912) QuantErr: 11.82912 batch_time=28.47623
Train Epoch: 3 [9/500 576/32000 (2%)] Loss: 2.02672 (QuantReg: 11.74582) QuantErr: 11.74582 batch_time=0.39523
Train Epoch: 3 [17/500 1088/32000 (3%)] Loss: 2.04886 (QuantReg: 11.94289) QuantErr: 11.94289 batch_time=0.39664
Train Epoch: 3 [25/500 1600/32000 (5%)] Loss: 1.58702 (QuantReg: 12.15829) QuantErr: 12.15829 batch_time=0.40124
Train Epoch: 3 [33/500 2112/32000 (7%)] Loss: 2.03333 (QuantReg: 12.12302) QuantErr: 12.12302 batch_time=0.43743
Train Epoch: 3 [41/500 2624/32000 (8%)] Loss: 2.11203 (QuantReg: 12.22991) QuantErr: 12.22991 batch_time=0.39017
Train Epoch: 3 [49/500 3136/32000 (10%)] Loss: 2.19472 (QuantReg: 12.49193) QuantErr: 12.49193 batch_time=0.40780
Train Epoch: 3 [57/500 3648/32000 (11%)] Loss: 1.87017 (QuantReg: 11.79900) QuantErr: 11.79900 batch_time=0.38439
Train Epoch: 3 [65/500 4160/32000 (13%)] Loss: 2.26375 (QuantReg: 12.29614) QuantErr: 12.29614 batch_time=0.73775
Train Epoch: 3 [73/500 4672/32000 (15%)] Loss: 1.49701 (QuantReg: 11.49531) QuantErr: 11.49531 batch_time=0.43055
Train Epoch: 3 [81/500 5184/32000 (16%)] Loss: 2.67945 (QuantReg: 12.21346) QuantErr: 12.21346 batch_time=0.40031
Train Epoch: 3 [89/500 5696/32000 (18%)] Loss: 2.14983 (QuantReg: 12.39651) QuantErr: 12.39651 batch_time=0.38607
Train Epoch: 3 [97/500 6208/32000 (19%)] Loss: 2.10773 (QuantReg: 12.17273) QuantErr: 12.17273 batch_time=0.42984
Train Epoch: 3 [105/500 6720/32000 (21%)] Loss: 2.13538 (QuantReg: 12.10429) QuantErr: 12.10429 batch_time=0.43298
Train Epoch: 3 [113/500 7232/32000 (23%)] Loss: 1.91849 (QuantReg: 12.23803) QuantErr: 12.23803 batch_time=0.39522
Train Epoch: 3 [121/500 7744/32000 (24%)] Loss: 2.16711 (QuantReg: 12.49025) QuantErr: 12.49025 batch_time=0.41638
Train Epoch: 3 [129/500 8256/32000 (26%)] Loss: 2.19637 (QuantReg: 12.55766) QuantErr: 12.55766 batch_time=0.75136
Train Epoch: 3 [137/500 8768/32000 (27%)] Loss: 1.66317 (QuantReg: 12.30215) QuantErr: 12.30215 batch_time=0.39192
Train Epoch: 3 [145/500 9280/32000 (29%)] Loss: 2.66310 (QuantReg: 12.43555) QuantErr: 12.43555 batch_time=0.38893
Train Epoch: 3 [153/500 9792/32000 (31%)] Loss: 2.22288 (QuantReg: 12.37793) QuantErr: 12.37793 batch_time=0.39342
Train Epoch: 3 [161/500 10304/32000 (32%)] Loss: 2.33422 (QuantReg: 12.41948) QuantErr: 12.41948 batch_time=0.40362
Train Epoch: 3 [169/500 10816/32000 (34%)] Loss: 2.75066 (QuantReg: 12.85815) QuantErr: 12.85815 batch_time=0.40331
Train Epoch: 3 [177/500 11328/32000 (35%)] Loss: 2.19557 (QuantReg: 12.33459) QuantErr: 12.33459 batch_time=0.41632
Train Epoch: 3 [185/500 11840/32000 (37%)] Loss: 2.46144 (QuantReg: 12.64014) QuantErr: 12.64014 batch_time=0.39043
Train Epoch: 3 [193/500 12352/32000 (39%)] Loss: 2.00284 (QuantReg: 12.50689) QuantErr: 12.50689 batch_time=0.39243
Train Epoch: 3 [201/500 12864/32000 (40%)] Loss: 3.04268 (QuantReg: 12.63590) QuantErr: 12.63590 batch_time=0.40001
Train Epoch: 3 [209/500 13376/32000 (42%)] Loss: 1.99925 (QuantReg: 12.50186) QuantErr: 12.50186 batch_time=0.39684
Train Epoch: 3 [217/500 13888/32000 (43%)] Loss: 1.76716 (QuantReg: 12.50002) QuantErr: 12.50002 batch_time=0.42863
Train Epoch: 3 [225/500 14400/32000 (45%)] Loss: 1.82578 (QuantReg: 12.37822) QuantErr: 12.37822 batch_time=0.43315
Train Epoch: 3 [233/500 14912/32000 (47%)] Loss: 2.12395 (QuantReg: 12.65936) QuantErr: 12.65936 batch_time=0.40304
Train Epoch: 3 [241/500 15424/32000 (48%)] Loss: 1.87409 (QuantReg: 12.52458) QuantErr: 12.52458 batch_time=0.38502
Train Epoch: 3 [249/500 15936/32000 (50%)] Loss: 1.55831 (QuantReg: 12.55218) QuantErr: 12.55218 batch_time=0.38903
Train Epoch: 3 [257/500 16448/32000 (51%)] Loss: 1.98979 (QuantReg: 12.68942) QuantErr: 12.68942 batch_time=0.40500
Train Epoch: 3 [265/500 16960/32000 (53%)] Loss: 2.74933 (QuantReg: 12.63388) QuantErr: 12.63388 batch_time=0.40053
Train Epoch: 3 [273/500 17472/32000 (55%)] Loss: 1.97571 (QuantReg: 12.79188) QuantErr: 12.79188 batch_time=0.39243
Train Epoch: 3 [281/500 17984/32000 (56%)] Loss: 1.82893 (QuantReg: 12.81462) QuantErr: 12.81462 batch_time=0.39274
Train Epoch: 3 [289/500 18496/32000 (58%)] Loss: 1.38237 (QuantReg: 13.03939) QuantErr: 13.03939 batch_time=0.39666
Train Epoch: 3 [297/500 19008/32000 (59%)] Loss: 2.17999 (QuantReg: 13.21073) QuantErr: 13.21073 batch_time=0.39726
Train Epoch: 3 [305/500 19520/32000 (61%)] Loss: 1.83722 (QuantReg: 12.79434) QuantErr: 12.79434 batch_time=0.39872
Train Epoch: 3 [313/500 20032/32000 (63%)] Loss: 2.57090 (QuantReg: 12.82205) QuantErr: 12.82205 batch_time=0.39420
Train Epoch: 3 [321/500 20544/32000 (64%)] Loss: 2.29722 (QuantReg: 12.69314) QuantErr: 12.69314 batch_time=0.39870
Train Epoch: 3 [329/500 21056/32000 (66%)] Loss: 2.03968 (QuantReg: 12.81554) QuantErr: 12.81554 batch_time=0.42395
Train Epoch: 3 [337/500 21568/32000 (67%)] Loss: 2.26037 (QuantReg: 12.97324) QuantErr: 12.97324 batch_time=0.39173
Train Epoch: 3 [345/500 22080/32000 (69%)] Loss: 2.65350 (QuantReg: 13.11128) QuantErr: 13.11128 batch_time=0.39462
Train Epoch: 3 [353/500 22592/32000 (71%)] Loss: 1.90708 (QuantReg: 12.92290) QuantErr: 12.92290 batch_time=0.40037
Train Epoch: 3 [361/500 23104/32000 (72%)] Loss: 1.88691 (QuantReg: 13.13088) QuantErr: 13.13088 batch_time=0.39127
Train Epoch: 3 [369/500 23616/32000 (74%)] Loss: 1.66741 (QuantReg: 13.54965) QuantErr: 13.54965 batch_time=0.39292
Train Epoch: 3 [377/500 24128/32000 (75%)] Loss: 2.17326 (QuantReg: 12.83577) QuantErr: 12.83577 batch_time=0.40860
Train Epoch: 3 [385/500 24640/32000 (77%)] Loss: 1.53834 (QuantReg: 13.42226) QuantErr: 13.42226 batch_time=0.39369
Train Epoch: 3 [393/500 25152/32000 (79%)] Loss: 1.55599 (QuantReg: 13.71889) QuantErr: 13.71889 batch_time=0.39962
Train Epoch: 3 [401/500 25664/32000 (80%)] Loss: 2.16720 (QuantReg: 12.94922) QuantErr: 12.94922 batch_time=0.41815
Train Epoch: 3 [409/500 26176/32000 (82%)] Loss: 1.84963 (QuantReg: 12.96148) QuantErr: 12.96148 batch_time=0.46212
Train Epoch: 3 [417/500 26688/32000 (83%)] Loss: 2.41941 (QuantReg: 13.45395) QuantErr: 13.45395 batch_time=0.39260
Train Epoch: 3 [425/500 27200/32000 (85%)] Loss: 1.93016 (QuantReg: 13.23715) QuantErr: 13.23715 batch_time=0.39962
Train Epoch: 3 [433/500 27712/32000 (87%)] Loss: 1.72406 (QuantReg: 13.14350) QuantErr: 13.14350 batch_time=0.41512
Train Epoch: 3 [441/500 28224/32000 (88%)] Loss: 2.34867 (QuantReg: 13.04842) QuantErr: 13.04842 batch_time=0.40287
Train Epoch: 3 [449/500 28736/32000 (90%)] Loss: 1.73906 (QuantReg: 13.45788) QuantErr: 13.45788 batch_time=0.39435
Train Epoch: 3 [457/500 29248/32000 (91%)] Loss: 1.62633 (QuantReg: 13.25621) QuantErr: 13.25621 batch_time=0.38605
Train Epoch: 3 [465/500 29760/32000 (93%)] Loss: 1.73384 (QuantReg: 13.39483) QuantErr: 13.39483 batch_time=0.39267
Train Epoch: 3 [473/500 30272/32000 (95%)] Loss: 1.84466 (QuantReg: 13.41760) QuantErr: 13.41760 batch_time=0.41552
Train Epoch: 3 [481/500 30784/32000 (96%)] Loss: 1.52981 (QuantReg: 13.11084) QuantErr: 13.11084 batch_time=0.42812
Train Epoch: 3 [489/500 31296/32000 (98%)] Loss: 1.38983 (QuantReg: 13.54920) QuantErr: 13.54920 batch_time=0.39776
Train Epoch: 3 [497/500 31808/32000 (99%)] Loss: 1.69949 (QuantReg: 13.82477) QuantErr: 13.82477 batch_time=0.39960
Train Epoch: 3 codebook_update_time=2.38272
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch3.pth ...
Done in 5.490s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch3.pth ...
Done in 10.518s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 2.053936599731445
quant_reg : 12.738394086837769
quant_err : 12.738394086837769
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 24.14486921529175
MSRVTT_full_val/t2v_metrics/R5: 61.3682092555332
MSRVTT_full_val/t2v_metrics/R10: 74.64788732394366
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.619718309859154
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.00228851771536
MSRVTT_full_val/v2t_metrics/R1: 31.187122736418512
MSRVTT_full_val/v2t_metrics/R5: 66.39839034205231
MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.837022132796781
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.985703916158485
MSRVTT_full_test/t2v_metrics/R1: 9.397993311036789
MSRVTT_full_test/t2v_metrics/R5: 28.896321070234112
MSRVTT_full_test/t2v_metrics/R10: 40.60200668896321
MSRVTT_full_test/t2v_metrics/R50: 75.18394648829431
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 54.65183946488294
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.257432316308563
MSRVTT_full_test/v2t_metrics/R1: 10.501672240802675
MSRVTT_full_test/v2t_metrics/R5: 32.675585284280935
MSRVTT_full_test/v2t_metrics/R10: 46.35451505016722
MSRVTT_full_test/v2t_metrics/R50: 80.80267558528428
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.80434782608695
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.149226120197923
mnt_best : 22.257432316308563
not_improved_count: 0
Train Epoch: 4 [1/500 64/32000 (0%)] Loss: 1.75184 (QuantReg: 12.32118) QuantErr: 12.32118 batch_time=31.22708
Train Epoch: 4 [9/500 576/32000 (2%)] Loss: 2.78752 (QuantReg: 12.10580) QuantErr: 12.10580 batch_time=0.40614
Train Epoch: 4 [17/500 1088/32000 (3%)] Loss: 2.30530 (QuantReg: 12.06200) QuantErr: 12.06200 batch_time=0.40101
Train Epoch: 4 [25/500 1600/32000 (5%)] Loss: 1.87462 (QuantReg: 12.04039) QuantErr: 12.04039 batch_time=0.44210
Train Epoch: 4 [33/500 2112/32000 (7%)] Loss: 1.93882 (QuantReg: 12.28127) QuantErr: 12.28127 batch_time=0.38623
Train Epoch: 4 [41/500 2624/32000 (8%)] Loss: 2.08572 (QuantReg: 12.16413) QuantErr: 12.16413 batch_time=0.85170
Train Epoch: 4 [49/500 3136/32000 (10%)] Loss: 1.40285 (QuantReg: 12.51955) QuantErr: 12.51955 batch_time=0.39086
Train Epoch: 4 [57/500 3648/32000 (11%)] Loss: 2.75910 (QuantReg: 11.76770) QuantErr: 11.76770 batch_time=0.40046
Train Epoch: 4 [65/500 4160/32000 (13%)] Loss: 1.58656 (QuantReg: 12.27555) QuantErr: 12.27555 batch_time=1.00272
Train Epoch: 4 [73/500 4672/32000 (15%)] Loss: 1.84999 (QuantReg: 12.26431) QuantErr: 12.26431 batch_time=0.39449
Train Epoch: 4 [81/500 5184/32000 (16%)] Loss: 1.85780 (QuantReg: 12.29288) QuantErr: 12.29288 batch_time=0.40724
Train Epoch: 4 [89/500 5696/32000 (18%)] Loss: 1.65569 (QuantReg: 12.29228) QuantErr: 12.29228 batch_time=0.39394
Train Epoch: 4 [97/500 6208/32000 (19%)] Loss: 1.77152 (QuantReg: 11.76648) QuantErr: 11.76648 batch_time=0.38227
Train Epoch: 4 [105/500 6720/32000 (21%)] Loss: 1.22797 (QuantReg: 12.19668) QuantErr: 12.19668 batch_time=1.02913
Train Epoch: 4 [113/500 7232/32000 (23%)] Loss: 1.69009 (QuantReg: 12.70263) QuantErr: 12.70263 batch_time=0.38684
Train Epoch: 4 [121/500 7744/32000 (24%)] Loss: 1.68382 (QuantReg: 12.12011) QuantErr: 12.12011 batch_time=0.42795
Train Epoch: 4 [129/500 8256/32000 (26%)] Loss: 1.57485 (QuantReg: 12.67229) QuantErr: 12.67229 batch_time=1.04039
Train Epoch: 4 [137/500 8768/32000 (27%)] Loss: 1.92329 (QuantReg: 12.43384) QuantErr: 12.43384 batch_time=0.40283
Train Epoch: 4 [145/500 9280/32000 (29%)] Loss: 1.53394 (QuantReg: 12.70564) QuantErr: 12.70564 batch_time=0.39843
Train Epoch: 4 [153/500 9792/32000 (31%)] Loss: 1.79072 (QuantReg: 12.96934) QuantErr: 12.96934 batch_time=0.40073
Train Epoch: 4 [161/500 10304/32000 (32%)] Loss: 1.66194 (QuantReg: 12.50966) QuantErr: 12.50966 batch_time=0.40184
Train Epoch: 4 [169/500 10816/32000 (34%)] Loss: 1.62762 (QuantReg: 13.23319) QuantErr: 13.23319 batch_time=1.04397
Train Epoch: 4 [177/500 11328/32000 (35%)] Loss: 1.80753 (QuantReg: 12.92026) QuantErr: 12.92026 batch_time=0.38684
Train Epoch: 4 [185/500 11840/32000 (37%)] Loss: 1.83017 (QuantReg: 12.58231) QuantErr: 12.58231 batch_time=0.38970
Train Epoch: 4 [193/500 12352/32000 (39%)] Loss: 1.21169 (QuantReg: 12.59854) QuantErr: 12.59854 batch_time=1.00929
Train Epoch: 4 [201/500 12864/32000 (40%)] Loss: 1.25136 (QuantReg: 12.52450) QuantErr: 12.52450 batch_time=0.39078
Train Epoch: 4 [209/500 13376/32000 (42%)] Loss: 2.02713 (QuantReg: 12.98659) QuantErr: 12.98659 batch_time=0.39078
Train Epoch: 4 [217/500 13888/32000 (43%)] Loss: 2.43001 (QuantReg: 12.93306) QuantErr: 12.93306 batch_time=0.39495
Train Epoch: 4 [225/500 14400/32000 (45%)] Loss: 1.70389 (QuantReg: 12.92437) QuantErr: 12.92437 batch_time=0.39738
Train Epoch: 4 [233/500 14912/32000 (47%)] Loss: 1.30379 (QuantReg: 12.66633) QuantErr: 12.66633 batch_time=0.38119
Train Epoch: 4 [241/500 15424/32000 (48%)] Loss: 1.44864 (QuantReg: 12.48967) QuantErr: 12.48967 batch_time=0.38705
Train Epoch: 4 [249/500 15936/32000 (50%)] Loss: 1.86175 (QuantReg: 13.20412) QuantErr: 13.20412 batch_time=0.41614
Train Epoch: 4 [257/500 16448/32000 (51%)] Loss: 1.43100 (QuantReg: 12.19050) QuantErr: 12.19050 batch_time=0.39876
Train Epoch: 4 [265/500 16960/32000 (53%)] Loss: 1.77790 (QuantReg: 13.02192) QuantErr: 13.02192 batch_time=0.39474
Train Epoch: 4 [273/500 17472/32000 (55%)] Loss: 1.83683 (QuantReg: 12.84354) QuantErr: 12.84354 batch_time=0.38687
Train Epoch: 4 [281/500 17984/32000 (56%)] Loss: 1.24931 (QuantReg: 12.39553) QuantErr: 12.39553 batch_time=0.40043
Train Epoch: 4 [289/500 18496/32000 (58%)] Loss: 1.21358 (QuantReg: 12.81881) QuantErr: 12.81881 batch_time=0.39482
Train Epoch: 4 [297/500 19008/32000 (59%)] Loss: 1.81615 (QuantReg: 12.09757) QuantErr: 12.09757 batch_time=0.39930
Train Epoch: 4 [305/500 19520/32000 (61%)] Loss: 2.09199 (QuantReg: 12.79320) QuantErr: 12.79320 batch_time=0.45359
Train Epoch: 4 [313/500 20032/32000 (63%)] Loss: 1.40940 (QuantReg: 12.69559) QuantErr: 12.69559 batch_time=0.41887
Train Epoch: 4 [321/500 20544/32000 (64%)] Loss: 2.15685 (QuantReg: 12.63464) QuantErr: 12.63464 batch_time=0.39324
Train Epoch: 4 [329/500 21056/32000 (66%)] Loss: 1.64204 (QuantReg: 12.83083) QuantErr: 12.83083 batch_time=0.40159
Train Epoch: 4 [337/500 21568/32000 (67%)] Loss: 1.94875 (QuantReg: 12.84053) QuantErr: 12.84053 batch_time=0.39665
Train Epoch: 4 [345/500 22080/32000 (69%)] Loss: 1.81361 (QuantReg: 12.86080) QuantErr: 12.86080 batch_time=0.38974
Train Epoch: 4 [353/500 22592/32000 (71%)] Loss: 1.54206 (QuantReg: 13.29385) QuantErr: 13.29385 batch_time=0.41419
Train Epoch: 4 [361/500 23104/32000 (72%)] Loss: 2.28765 (QuantReg: 12.66079) QuantErr: 12.66079 batch_time=0.38732
Train Epoch: 4 [369/500 23616/32000 (74%)] Loss: 1.83458 (QuantReg: 12.92764) QuantErr: 12.92764 batch_time=0.38579
Train Epoch: 4 [377/500 24128/32000 (75%)] Loss: 1.43909 (QuantReg: 12.98123) QuantErr: 12.98123 batch_time=0.40155
Train Epoch: 4 [385/500 24640/32000 (77%)] Loss: 2.12607 (QuantReg: 12.89286) QuantErr: 12.89286 batch_time=0.40138
Train Epoch: 4 [393/500 25152/32000 (79%)] Loss: 1.60292 (QuantReg: 13.03639) QuantErr: 13.03639 batch_time=0.39648
Train Epoch: 4 [401/500 25664/32000 (80%)] Loss: 1.96240 (QuantReg: 12.74891) QuantErr: 12.74891 batch_time=0.39606
Train Epoch: 4 [409/500 26176/32000 (82%)] Loss: 2.30991 (QuantReg: 12.74133) QuantErr: 12.74133 batch_time=0.40137
Train Epoch: 4 [417/500 26688/32000 (83%)] Loss: 2.03576 (QuantReg: 12.70645) QuantErr: 12.70645 batch_time=0.41194
Train Epoch: 4 [425/500 27200/32000 (85%)] Loss: 1.91951 (QuantReg: 13.22574) QuantErr: 13.22574 batch_time=0.39774
Train Epoch: 4 [433/500 27712/32000 (87%)] Loss: 1.59066 (QuantReg: 13.48866) QuantErr: 13.48866 batch_time=0.39251
Train Epoch: 4 [441/500 28224/32000 (88%)] Loss: 1.76940 (QuantReg: 13.07471) QuantErr: 13.07471 batch_time=0.39513
Train Epoch: 4 [449/500 28736/32000 (90%)] Loss: 1.86777 (QuantReg: 12.90453) QuantErr: 12.90453 batch_time=0.40047
Train Epoch: 4 [457/500 29248/32000 (91%)] Loss: 1.60647 (QuantReg: 13.74910) QuantErr: 13.74910 batch_time=0.39574
Train Epoch: 4 [465/500 29760/32000 (93%)] Loss: 2.24671 (QuantReg: 13.35341) QuantErr: 13.35341 batch_time=0.44267
Train Epoch: 4 [473/500 30272/32000 (95%)] Loss: 2.25806 (QuantReg: 13.01092) QuantErr: 13.01092 batch_time=0.44701
Train Epoch: 4 [481/500 30784/32000 (96%)] Loss: 1.60337 (QuantReg: 12.96256) QuantErr: 12.96256 batch_time=0.38697
Train Epoch: 4 [489/500 31296/32000 (98%)] Loss: 1.04834 (QuantReg: 13.27271) QuantErr: 13.27271 batch_time=0.41083
Train Epoch: 4 [497/500 31808/32000 (99%)] Loss: 1.59139 (QuantReg: 13.00679) QuantErr: 13.00679 batch_time=0.39177
Train Epoch: 4 codebook_update_time=2.12345
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch4.pth ...
Done in 5.467s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch4.pth ...
Done in 10.261s
removing stale ckpt [epoch 3] [took 0.08s]
epoch : 4
loss : 1.7582367053031922
quant_reg : 12.69219441986084
quant_err : 12.69219441986084
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 26.760563380281692
MSRVTT_full_val/t2v_metrics/R5: 63.17907444668008
MSRVTT_full_val/t2v_metrics/R10: 75.0503018108652
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.637826961770624
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.25049199397202
MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
MSRVTT_full_val/v2t_metrics/R5: 66.59959758551308
MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.927565392354125
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.32161383741219
MSRVTT_full_test/t2v_metrics/R1: 9.59866220735786
MSRVTT_full_test/t2v_metrics/R5: 29.19732441471572
MSRVTT_full_test/t2v_metrics/R10: 42.207357859531776
MSRVTT_full_test/t2v_metrics/R50: 76.5886287625418
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.66321070234114
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.784909614840664
MSRVTT_full_test/v2t_metrics/R1: 11.036789297658864
MSRVTT_full_test/v2t_metrics/R5: 33.31103678929766
MSRVTT_full_test/v2t_metrics/R10: 47.324414715719065
MSRVTT_full_test/v2t_metrics/R50: 81.73913043478261
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 40.854682274247494
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.912265472610954
mnt_best : 22.784909614840664
not_improved_count: 0
Train Epoch: 5 [1/500 64/32000 (0%)] Loss: 1.54472 (QuantReg: 12.56585) QuantErr: 12.56585 batch_time=31.63661
Train Epoch: 5 [9/500 576/32000 (2%)] Loss: 2.01757 (QuantReg: 12.06950) QuantErr: 12.06950 batch_time=0.40030
Train Epoch: 5 [17/500 1088/32000 (3%)] Loss: 1.79857 (QuantReg: 12.33059) QuantErr: 12.33059 batch_time=0.41243
Train Epoch: 5 [25/500 1600/32000 (5%)] Loss: 1.94354 (QuantReg: 12.34549) QuantErr: 12.34549 batch_time=0.39584
Train Epoch: 5 [33/500 2112/32000 (7%)] Loss: 1.58160 (QuantReg: 12.06917) QuantErr: 12.06917 batch_time=0.51270
Train Epoch: 5 [41/500 2624/32000 (8%)] Loss: 1.71365 (QuantReg: 11.97016) QuantErr: 11.97016 batch_time=0.39403
Train Epoch: 5 [49/500 3136/32000 (10%)] Loss: 1.63804 (QuantReg: 12.71265) QuantErr: 12.71265 batch_time=0.39636
Train Epoch: 5 [57/500 3648/32000 (11%)] Loss: 1.73117 (QuantReg: 12.19431) QuantErr: 12.19431 batch_time=0.43102
Train Epoch: 5 [65/500 4160/32000 (13%)] Loss: 1.53462 (QuantReg: 12.41392) QuantErr: 12.41392 batch_time=0.40349
Train Epoch: 5 [73/500 4672/32000 (15%)] Loss: 2.27222 (QuantReg: 12.20629) QuantErr: 12.20629 batch_time=0.40093
Train Epoch: 5 [81/500 5184/32000 (16%)] Loss: 1.59031 (QuantReg: 12.32838) QuantErr: 12.32838 batch_time=0.39347
Train Epoch: 5 [89/500 5696/32000 (18%)] Loss: 1.53942 (QuantReg: 12.32615) QuantErr: 12.32615 batch_time=0.40769
Train Epoch: 5 [97/500 6208/32000 (19%)] Loss: 2.34045 (QuantReg: 12.11192) QuantErr: 12.11192 batch_time=0.50168
Train Epoch: 5 [105/500 6720/32000 (21%)] Loss: 1.66288 (QuantReg: 12.07535) QuantErr: 12.07535 batch_time=0.39078
Train Epoch: 5 [113/500 7232/32000 (23%)] Loss: 1.70220 (QuantReg: 12.36147) QuantErr: 12.36147 batch_time=0.38958
Train Epoch: 5 [121/500 7744/32000 (24%)] Loss: 1.41827 (QuantReg: 12.35462) QuantErr: 12.35462 batch_time=0.38860
Train Epoch: 5 [129/500 8256/32000 (26%)] Loss: 1.92247 (QuantReg: 13.01104) QuantErr: 13.01104 batch_time=0.37775
Train Epoch: 5 [137/500 8768/32000 (27%)] Loss: 1.93224 (QuantReg: 12.75348) QuantErr: 12.75348 batch_time=0.39471
Train Epoch: 5 [145/500 9280/32000 (29%)] Loss: 1.41281 (QuantReg: 12.34961) QuantErr: 12.34961 batch_time=0.39790
Train Epoch: 5 [153/500 9792/32000 (31%)] Loss: 1.45909 (QuantReg: 12.87001) QuantErr: 12.87001 batch_time=0.41847
Train Epoch: 5 [161/500 10304/32000 (32%)] Loss: 1.74232 (QuantReg: 12.79114) QuantErr: 12.79114 batch_time=0.51224
Train Epoch: 5 [169/500 10816/32000 (34%)] Loss: 1.30749 (QuantReg: 12.94249) QuantErr: 12.94249 batch_time=0.39317
Train Epoch: 5 [177/500 11328/32000 (35%)] Loss: 1.58138 (QuantReg: 12.72949) QuantErr: 12.72949 batch_time=0.40861
Train Epoch: 5 [185/500 11840/32000 (37%)] Loss: 1.71015 (QuantReg: 13.22083) QuantErr: 13.22083 batch_time=0.38978
Train Epoch: 5 [193/500 12352/32000 (39%)] Loss: 1.57528 (QuantReg: 12.46153) QuantErr: 12.46153 batch_time=0.39401
Train Epoch: 5 [201/500 12864/32000 (40%)] Loss: 1.61467 (QuantReg: 12.15421) QuantErr: 12.15421 batch_time=0.41229
Train Epoch: 5 [209/500 13376/32000 (42%)] Loss: 1.27286 (QuantReg: 12.25346) QuantErr: 12.25346 batch_time=0.39676
Train Epoch: 5 [217/500 13888/32000 (43%)] Loss: 1.47115 (QuantReg: 12.58697) QuantErr: 12.58697 batch_time=0.39760
Train Epoch: 5 [225/500 14400/32000 (45%)] Loss: 1.88333 (QuantReg: 12.86080) QuantErr: 12.86080 batch_time=0.54761
Train Epoch: 5 [233/500 14912/32000 (47%)] Loss: 1.65079 (QuantReg: 12.87507) QuantErr: 12.87507 batch_time=0.39215
Train Epoch: 5 [241/500 15424/32000 (48%)] Loss: 2.21671 (QuantReg: 12.35644) QuantErr: 12.35644 batch_time=0.38905
Train Epoch: 5 [249/500 15936/32000 (50%)] Loss: 1.58451 (QuantReg: 12.32329) QuantErr: 12.32329 batch_time=0.39310
Train Epoch: 5 [257/500 16448/32000 (51%)] Loss: 1.25012 (QuantReg: 12.71306) QuantErr: 12.71306 batch_time=0.38682
Train Epoch: 5 [265/500 16960/32000 (53%)] Loss: 1.31599 (QuantReg: 12.48547) QuantErr: 12.48547 batch_time=0.40943
Train Epoch: 5 [273/500 17472/32000 (55%)] Loss: 1.44974 (QuantReg: 13.04136) QuantErr: 13.04136 batch_time=0.39887
Train Epoch: 5 [281/500 17984/32000 (56%)] Loss: 1.63951 (QuantReg: 12.57378) QuantErr: 12.57378 batch_time=0.39446
Train Epoch: 5 [289/500 18496/32000 (58%)] Loss: 1.76328 (QuantReg: 12.54737) QuantErr: 12.54737 batch_time=0.56687
Train Epoch: 5 [297/500 19008/32000 (59%)] Loss: 1.72582 (QuantReg: 12.81276) QuantErr: 12.81276 batch_time=0.39603
Train Epoch: 5 [305/500 19520/32000 (61%)] Loss: 1.64066 (QuantReg: 13.04384) QuantErr: 13.04384 batch_time=0.39534
Train Epoch: 5 [313/500 20032/32000 (63%)] Loss: 1.19077 (QuantReg: 13.08715) QuantErr: 13.08715 batch_time=0.39617
Train Epoch: 5 [321/500 20544/32000 (64%)] Loss: 1.22293 (QuantReg: 13.20170) QuantErr: 13.20170 batch_time=0.40959
Train Epoch: 5 [329/500 21056/32000 (66%)] Loss: 1.58775 (QuantReg: 12.85840) QuantErr: 12.85840 batch_time=0.39562
Train Epoch: 5 [337/500 21568/32000 (67%)] Loss: 1.26907 (QuantReg: 12.69888) QuantErr: 12.69888 batch_time=0.41482
Train Epoch: 5 [345/500 22080/32000 (69%)] Loss: 1.69866 (QuantReg: 13.07562) QuantErr: 13.07562 batch_time=0.39221
Train Epoch: 5 [353/500 22592/32000 (71%)] Loss: 1.24814 (QuantReg: 13.03347) QuantErr: 13.03347 batch_time=0.41402
Train Epoch: 5 [361/500 23104/32000 (72%)] Loss: 1.47351 (QuantReg: 12.87604) QuantErr: 12.87604 batch_time=0.39685
Train Epoch: 5 [369/500 23616/32000 (74%)] Loss: 1.66804 (QuantReg: 12.98231) QuantErr: 12.98231 batch_time=0.38507
Train Epoch: 5 [377/500 24128/32000 (75%)] Loss: 1.44861 (QuantReg: 12.82968) QuantErr: 12.82968 batch_time=0.38729
Train Epoch: 5 [385/500 24640/32000 (77%)] Loss: 1.39330 (QuantReg: 12.54931) QuantErr: 12.54931 batch_time=0.39913
Train Epoch: 5 [393/500 25152/32000 (79%)] Loss: 1.47478 (QuantReg: 12.92687) QuantErr: 12.92687 batch_time=0.39962
Train Epoch: 5 [401/500 25664/32000 (80%)] Loss: 1.97498 (QuantReg: 12.55503) QuantErr: 12.55503 batch_time=0.42971
Train Epoch: 5 [409/500 26176/32000 (82%)] Loss: 1.80582 (QuantReg: 12.86993) QuantErr: 12.86993 batch_time=0.39031
Train Epoch: 5 [417/500 26688/32000 (83%)] Loss: 1.68756 (QuantReg: 12.61598) QuantErr: 12.61598 batch_time=0.39408
Train Epoch: 5 [425/500 27200/32000 (85%)] Loss: 1.23612 (QuantReg: 13.11189) QuantErr: 13.11189 batch_time=0.39341
Train Epoch: 5 [433/500 27712/32000 (87%)] Loss: 1.29950 (QuantReg: 13.15410) QuantErr: 13.15410 batch_time=0.39605
Train Epoch: 5 [441/500 28224/32000 (88%)] Loss: 1.58980 (QuantReg: 12.90549) QuantErr: 12.90549 batch_time=0.39955
Train Epoch: 5 [449/500 28736/32000 (90%)] Loss: 1.98252 (QuantReg: 13.01122) QuantErr: 13.01122 batch_time=0.41687
Train Epoch: 5 [457/500 29248/32000 (91%)] Loss: 1.44212 (QuantReg: 12.96486) QuantErr: 12.96486 batch_time=0.39180
Train Epoch: 5 [465/500 29760/32000 (93%)] Loss: 1.68168 (QuantReg: 12.79189) QuantErr: 12.79189 batch_time=0.39877
Train Epoch: 5 [473/500 30272/32000 (95%)] Loss: 1.77197 (QuantReg: 12.63530) QuantErr: 12.63530 batch_time=0.39598
Train Epoch: 5 [481/500 30784/32000 (96%)] Loss: 2.02312 (QuantReg: 12.64193) QuantErr: 12.64193 batch_time=0.39276
Train Epoch: 5 [489/500 31296/32000 (98%)] Loss: 1.97414 (QuantReg: 12.47764) QuantErr: 12.47764 batch_time=0.39013
Train Epoch: 5 [497/500 31808/32000 (99%)] Loss: 1.97256 (QuantReg: 12.83946) QuantErr: 12.83946 batch_time=0.39449
Train Epoch: 5 codebook_update_time=2.21408
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch5.pth ...
Done in 4.267s
removing stale ckpt [epoch 4] [took 2.23s]
epoch : 5
loss : 1.5685114523172379
quant_reg : 12.716559984207153
quant_err : 12.716559984207153
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 26.961770623742456
MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
MSRVTT_full_val/t2v_metrics/R10: 75.25150905432595
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.231388329979879
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.043599136847476
MSRVTT_full_val/v2t_metrics/R1: 29.77867203219316
MSRVTT_full_val/v2t_metrics/R5: 68.00804828973843
MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.132796780684105
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.57922715214068
MSRVTT_full_test/t2v_metrics/R1: 8.66220735785953
MSRVTT_full_test/t2v_metrics/R5: 26.889632107023413
MSRVTT_full_test/t2v_metrics/R10: 40.23411371237458
MSRVTT_full_test/t2v_metrics/R50: 74.94983277591973
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.159866220735786
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.083172319440084
MSRVTT_full_test/v2t_metrics/R1: 11.33779264214047
MSRVTT_full_test/v2t_metrics/R5: 32.608695652173914
MSRVTT_full_test/v2t_metrics/R10: 46.68896321070234
MSRVTT_full_test/v2t_metrics/R50: 80.06688963210702
MSRVTT_full_test/v2t_metrics/MedR: 12.5
MSRVTT_full_test/v2t_metrics/MeanR: 47.37692307692308
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.843939899184004
mnt_best : 22.784909614840664
not_improved_count: 1
Train Epoch: 6 [1/500 64/32000 (0%)] Loss: 1.96780 (QuantReg: 12.45331) QuantErr: 12.45331 batch_time=30.23567
Train Epoch: 6 [9/500 576/32000 (2%)] Loss: 1.42974 (QuantReg: 12.84266) QuantErr: 12.84266 batch_time=0.39367
Train Epoch: 6 [17/500 1088/32000 (3%)] Loss: 1.11814 (QuantReg: 12.55186) QuantErr: 12.55186 batch_time=0.43467
Train Epoch: 6 [25/500 1600/32000 (5%)] Loss: 1.05518 (QuantReg: 12.45961) QuantErr: 12.45961 batch_time=0.40822
Train Epoch: 6 [33/500 2112/32000 (7%)] Loss: 1.97150 (QuantReg: 12.56135) QuantErr: 12.56135 batch_time=0.39436
Train Epoch: 6 [41/500 2624/32000 (8%)] Loss: 1.36753 (QuantReg: 12.32276) QuantErr: 12.32276 batch_time=0.39861
Train Epoch: 6 [49/500 3136/32000 (10%)] Loss: 1.56254 (QuantReg: 12.79762) QuantErr: 12.79762 batch_time=0.38634
Train Epoch: 6 [57/500 3648/32000 (11%)] Loss: 1.58998 (QuantReg: 12.52922) QuantErr: 12.52922 batch_time=0.39916
Train Epoch: 6 [65/500 4160/32000 (13%)] Loss: 1.62367 (QuantReg: 12.69595) QuantErr: 12.69595 batch_time=0.40552
Train Epoch: 6 [73/500 4672/32000 (15%)] Loss: 1.09168 (QuantReg: 12.47095) QuantErr: 12.47095 batch_time=0.41953
Train Epoch: 6 [81/500 5184/32000 (16%)] Loss: 1.11758 (QuantReg: 13.32632) QuantErr: 13.32632 batch_time=0.38772
Train Epoch: 6 [89/500 5696/32000 (18%)] Loss: 1.35721 (QuantReg: 12.38876) QuantErr: 12.38876 batch_time=0.39849
Train Epoch: 6 [97/500 6208/32000 (19%)] Loss: 1.51466 (QuantReg: 12.10286) QuantErr: 12.10286 batch_time=0.40087
Train Epoch: 6 [105/500 6720/32000 (21%)] Loss: 1.62664 (QuantReg: 12.45674) QuantErr: 12.45674 batch_time=0.39853
Train Epoch: 6 [113/500 7232/32000 (23%)] Loss: 1.07988 (QuantReg: 12.03989) QuantErr: 12.03989 batch_time=0.43750
Train Epoch: 6 [121/500 7744/32000 (24%)] Loss: 0.95002 (QuantReg: 13.19120) QuantErr: 13.19120 batch_time=0.43408
Train Epoch: 6 [129/500 8256/32000 (26%)] Loss: 2.16467 (QuantReg: 12.64594) QuantErr: 12.64594 batch_time=0.39942
Train Epoch: 6 [137/500 8768/32000 (27%)] Loss: 1.08272 (QuantReg: 12.70597) QuantErr: 12.70597 batch_time=0.39444
Train Epoch: 6 [145/500 9280/32000 (29%)] Loss: 1.49941 (QuantReg: 12.45223) QuantErr: 12.45223 batch_time=0.41354
Train Epoch: 6 [153/500 9792/32000 (31%)] Loss: 0.97053 (QuantReg: 12.77758) QuantErr: 12.77758 batch_time=0.38569
Train Epoch: 6 [161/500 10304/32000 (32%)] Loss: 1.08872 (QuantReg: 12.55771) QuantErr: 12.55771 batch_time=0.39600
Train Epoch: 6 [169/500 10816/32000 (34%)] Loss: 1.44907 (QuantReg: 12.97076) QuantErr: 12.97076 batch_time=0.41320
Train Epoch: 6 [177/500 11328/32000 (35%)] Loss: 1.12895 (QuantReg: 12.83212) QuantErr: 12.83212 batch_time=0.39629
Train Epoch: 6 [185/500 11840/32000 (37%)] Loss: 1.69368 (QuantReg: 12.15770) QuantErr: 12.15770 batch_time=0.39453
Train Epoch: 6 [193/500 12352/32000 (39%)] Loss: 1.32147 (QuantReg: 13.08154) QuantErr: 13.08154 batch_time=0.41142
Train Epoch: 6 [201/500 12864/32000 (40%)] Loss: 1.54563 (QuantReg: 12.35089) QuantErr: 12.35089 batch_time=0.40083
Train Epoch: 6 [209/500 13376/32000 (42%)] Loss: 1.51472 (QuantReg: 12.57879) QuantErr: 12.57879 batch_time=0.39329
Train Epoch: 6 [217/500 13888/32000 (43%)] Loss: 1.93960 (QuantReg: 12.65104) QuantErr: 12.65104 batch_time=0.44187
Train Epoch: 6 [225/500 14400/32000 (45%)] Loss: 1.62859 (QuantReg: 12.87815) QuantErr: 12.87815 batch_time=0.39267
Train Epoch: 6 [233/500 14912/32000 (47%)] Loss: 1.80977 (QuantReg: 12.54914) QuantErr: 12.54914 batch_time=0.38425
Train Epoch: 6 [241/500 15424/32000 (48%)] Loss: 1.96786 (QuantReg: 12.33850) QuantErr: 12.33850 batch_time=0.44606
Train Epoch: 6 [249/500 15936/32000 (50%)] Loss: 1.75273 (QuantReg: 13.05940) QuantErr: 13.05940 batch_time=0.39930
Train Epoch: 6 [257/500 16448/32000 (51%)] Loss: 1.82761 (QuantReg: 12.53507) QuantErr: 12.53507 batch_time=0.40037
Train Epoch: 6 [265/500 16960/32000 (53%)] Loss: 0.75841 (QuantReg: 12.98775) QuantErr: 12.98775 batch_time=0.39057
Train Epoch: 6 [273/500 17472/32000 (55%)] Loss: 1.82650 (QuantReg: 12.98372) QuantErr: 12.98372 batch_time=0.38874
Train Epoch: 6 [281/500 17984/32000 (56%)] Loss: 1.25147 (QuantReg: 12.95396) QuantErr: 12.95396 batch_time=0.39258
Train Epoch: 6 [289/500 18496/32000 (58%)] Loss: 2.10690 (QuantReg: 12.65615) QuantErr: 12.65615 batch_time=0.38898
Train Epoch: 6 [297/500 19008/32000 (59%)] Loss: 1.27318 (QuantReg: 13.07700) QuantErr: 13.07700 batch_time=0.40113
Train Epoch: 6 [305/500 19520/32000 (61%)] Loss: 1.39506 (QuantReg: 12.75171) QuantErr: 12.75171 batch_time=0.45175
Train Epoch: 6 [313/500 20032/32000 (63%)] Loss: 1.58306 (QuantReg: 12.72298) QuantErr: 12.72298 batch_time=0.40408
Train Epoch: 6 [321/500 20544/32000 (64%)] Loss: 1.41836 (QuantReg: 13.16352) QuantErr: 13.16352 batch_time=0.39070
Train Epoch: 6 [329/500 21056/32000 (66%)] Loss: 1.46001 (QuantReg: 12.53345) QuantErr: 12.53345 batch_time=0.39749
Train Epoch: 6 [337/500 21568/32000 (67%)] Loss: 1.49733 (QuantReg: 12.97483) QuantErr: 12.97483 batch_time=0.43273
Train Epoch: 6 [345/500 22080/32000 (69%)] Loss: 1.41370 (QuantReg: 13.47442) QuantErr: 13.47442 batch_time=0.40838
Train Epoch: 6 [353/500 22592/32000 (71%)] Loss: 0.94807 (QuantReg: 13.03864) QuantErr: 13.03864 batch_time=0.39236
Train Epoch: 6 [361/500 23104/32000 (72%)] Loss: 1.37744 (QuantReg: 13.00916) QuantErr: 13.00916 batch_time=0.39504
Train Epoch: 6 [369/500 23616/32000 (74%)] Loss: 0.94484 (QuantReg: 12.49626) QuantErr: 12.49626 batch_time=0.64860
Train Epoch: 6 [377/500 24128/32000 (75%)] Loss: 1.19804 (QuantReg: 12.78401) QuantErr: 12.78401 batch_time=0.39823
Train Epoch: 6 [385/500 24640/32000 (77%)] Loss: 1.29649 (QuantReg: 12.84787) QuantErr: 12.84787 batch_time=0.40245
Train Epoch: 6 [393/500 25152/32000 (79%)] Loss: 1.30465 (QuantReg: 12.87922) QuantErr: 12.87922 batch_time=0.40091
Train Epoch: 6 [401/500 25664/32000 (80%)] Loss: 1.00381 (QuantReg: 13.07053) QuantErr: 13.07053 batch_time=0.39036
Train Epoch: 6 [409/500 26176/32000 (82%)] Loss: 1.25852 (QuantReg: 12.79083) QuantErr: 12.79083 batch_time=0.39168
Train Epoch: 6 [417/500 26688/32000 (83%)] Loss: 1.09798 (QuantReg: 12.40524) QuantErr: 12.40524 batch_time=0.38707
Train Epoch: 6 [425/500 27200/32000 (85%)] Loss: 1.64692 (QuantReg: 12.80969) QuantErr: 12.80969 batch_time=0.39027
Train Epoch: 6 [433/500 27712/32000 (87%)] Loss: 1.52631 (QuantReg: 12.80992) QuantErr: 12.80992 batch_time=0.43780
Train Epoch: 6 [441/500 28224/32000 (88%)] Loss: 1.33233 (QuantReg: 12.57703) QuantErr: 12.57703 batch_time=0.39733
Train Epoch: 6 [449/500 28736/32000 (90%)] Loss: 1.14661 (QuantReg: 12.96878) QuantErr: 12.96878 batch_time=0.39379
Train Epoch: 6 [457/500 29248/32000 (91%)] Loss: 1.71887 (QuantReg: 13.41202) QuantErr: 13.41202 batch_time=0.40201
Train Epoch: 6 [465/500 29760/32000 (93%)] Loss: 1.55513 (QuantReg: 12.43541) QuantErr: 12.43541 batch_time=0.39562
Train Epoch: 6 [473/500 30272/32000 (95%)] Loss: 1.35291 (QuantReg: 12.81878) QuantErr: 12.81878 batch_time=0.39511
Train Epoch: 6 [481/500 30784/32000 (96%)] Loss: 1.28457 (QuantReg: 12.51670) QuantErr: 12.51670 batch_time=0.39988
Train Epoch: 6 [489/500 31296/32000 (98%)] Loss: 0.89367 (QuantReg: 12.75255) QuantErr: 12.75255 batch_time=0.41184
Train Epoch: 6 [497/500 31808/32000 (99%)] Loss: 1.27891 (QuantReg: 12.90516) QuantErr: 12.90516 batch_time=0.39437
Train Epoch: 6 codebook_update_time=1.88140
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch6.pth ...
Done in 5.291s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch6.pth ...
Done in 9.908s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 1.4261233661174775
quant_reg : 12.712779527664184
quant_err : 12.712779527664184
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
MSRVTT_full_val/t2v_metrics/R5: 65.19114688128772
MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.203219315895373
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.03335310284704
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.150905432595573
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.95323853794192
MSRVTT_full_test/t2v_metrics/R1: 10.301003344481606
MSRVTT_full_test/t2v_metrics/R5: 29.297658862876254
MSRVTT_full_test/t2v_metrics/R10: 43.31103678929766
MSRVTT_full_test/t2v_metrics/R50: 77.22408026755853
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 51.96387959866221
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.55611551581911
MSRVTT_full_test/v2t_metrics/R1: 12.842809364548495
MSRVTT_full_test/v2t_metrics/R5: 35.18394648829432
MSRVTT_full_test/v2t_metrics/R10: 49.66555183946488
MSRVTT_full_test/v2t_metrics/R50: 82.50836120401338
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.04531772575251
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.206765259113084
mnt_best : 23.55611551581911
not_improved_count: 0
Train Epoch: 7 [1/500 64/32000 (0%)] Loss: 1.32218 (QuantReg: 11.57924) QuantErr: 11.57924 batch_time=28.55974
Train Epoch: 7 [9/500 576/32000 (2%)] Loss: 0.85175 (QuantReg: 12.45084) QuantErr: 12.45084 batch_time=0.43725
Train Epoch: 7 [17/500 1088/32000 (3%)] Loss: 1.65104 (QuantReg: 12.51750) QuantErr: 12.51750 batch_time=0.40875
Train Epoch: 7 [25/500 1600/32000 (5%)] Loss: 1.16210 (QuantReg: 12.23711) QuantErr: 12.23711 batch_time=0.43958
Train Epoch: 7 [33/500 2112/32000 (7%)] Loss: 1.64769 (QuantReg: 12.43781) QuantErr: 12.43781 batch_time=0.39793
Train Epoch: 7 [41/500 2624/32000 (8%)] Loss: 0.93835 (QuantReg: 12.79955) QuantErr: 12.79955 batch_time=0.39143
Train Epoch: 7 [49/500 3136/32000 (10%)] Loss: 1.84058 (QuantReg: 12.39545) QuantErr: 12.39545 batch_time=0.39000
Train Epoch: 7 [57/500 3648/32000 (11%)] Loss: 1.97366 (QuantReg: 12.03178) QuantErr: 12.03178 batch_time=0.39515
Train Epoch: 7 [65/500 4160/32000 (13%)] Loss: 1.58062 (QuantReg: 12.52463) QuantErr: 12.52463 batch_time=0.39815
Train Epoch: 7 [73/500 4672/32000 (15%)] Loss: 1.19947 (QuantReg: 12.90213) QuantErr: 12.90213 batch_time=0.41211
Train Epoch: 7 [81/500 5184/32000 (16%)] Loss: 1.31105 (QuantReg: 12.14385) QuantErr: 12.14385 batch_time=0.42213
Train Epoch: 7 [89/500 5696/32000 (18%)] Loss: 1.77674 (QuantReg: 12.23552) QuantErr: 12.23552 batch_time=0.39364
Train Epoch: 7 [97/500 6208/32000 (19%)] Loss: 1.58792 (QuantReg: 12.30590) QuantErr: 12.30590 batch_time=0.39805
Train Epoch: 7 [105/500 6720/32000 (21%)] Loss: 0.86738 (QuantReg: 12.65633) QuantErr: 12.65633 batch_time=0.39442
Train Epoch: 7 [113/500 7232/32000 (23%)] Loss: 1.74139 (QuantReg: 12.64655) QuantErr: 12.64655 batch_time=0.39876
Train Epoch: 7 [121/500 7744/32000 (24%)] Loss: 0.92286 (QuantReg: 12.50237) QuantErr: 12.50237 batch_time=0.40107
Train Epoch: 7 [129/500 8256/32000 (26%)] Loss: 1.05001 (QuantReg: 12.70234) QuantErr: 12.70234 batch_time=0.39614
Train Epoch: 7 [137/500 8768/32000 (27%)] Loss: 1.21653 (QuantReg: 12.50078) QuantErr: 12.50078 batch_time=0.38880
Train Epoch: 7 [145/500 9280/32000 (29%)] Loss: 1.32987 (QuantReg: 12.28858) QuantErr: 12.28858 batch_time=0.38866
Train Epoch: 7 [153/500 9792/32000 (31%)] Loss: 1.59632 (QuantReg: 12.41816) QuantErr: 12.41816 batch_time=0.40268
Train Epoch: 7 [161/500 10304/32000 (32%)] Loss: 1.82180 (QuantReg: 12.24576) QuantErr: 12.24576 batch_time=0.40016
Train Epoch: 7 [169/500 10816/32000 (34%)] Loss: 1.34921 (QuantReg: 12.72164) QuantErr: 12.72164 batch_time=0.38867
Train Epoch: 7 [177/500 11328/32000 (35%)] Loss: 1.11867 (QuantReg: 12.80189) QuantErr: 12.80189 batch_time=0.42935
Train Epoch: 7 [185/500 11840/32000 (37%)] Loss: 1.36131 (QuantReg: 12.87861) QuantErr: 12.87861 batch_time=0.38918
Train Epoch: 7 [193/500 12352/32000 (39%)] Loss: 1.38960 (QuantReg: 12.81411) QuantErr: 12.81411 batch_time=0.41190
Train Epoch: 7 [201/500 12864/32000 (40%)] Loss: 1.25236 (QuantReg: 12.93190) QuantErr: 12.93190 batch_time=0.43096
Train Epoch: 7 [209/500 13376/32000 (42%)] Loss: 1.50345 (QuantReg: 12.83652) QuantErr: 12.83652 batch_time=0.42609
Train Epoch: 7 [217/500 13888/32000 (43%)] Loss: 1.17874 (QuantReg: 12.70034) QuantErr: 12.70034 batch_time=0.40723
Train Epoch: 7 [225/500 14400/32000 (45%)] Loss: 1.01842 (QuantReg: 12.78678) QuantErr: 12.78678 batch_time=0.42804
Train Epoch: 7 [233/500 14912/32000 (47%)] Loss: 1.12577 (QuantReg: 12.90904) QuantErr: 12.90904 batch_time=0.44162
Train Epoch: 7 [241/500 15424/32000 (48%)] Loss: 1.01290 (QuantReg: 13.00919) QuantErr: 13.00919 batch_time=0.46316
Train Epoch: 7 [249/500 15936/32000 (50%)] Loss: 1.16460 (QuantReg: 13.09290) QuantErr: 13.09290 batch_time=0.40416
Train Epoch: 7 [257/500 16448/32000 (51%)] Loss: 1.07263 (QuantReg: 13.21177) QuantErr: 13.21177 batch_time=0.39225
Train Epoch: 7 [265/500 16960/32000 (53%)] Loss: 1.21364 (QuantReg: 12.69909) QuantErr: 12.69909 batch_time=0.39724
Train Epoch: 7 [273/500 17472/32000 (55%)] Loss: 1.69522 (QuantReg: 12.57870) QuantErr: 12.57870 batch_time=0.39758
Train Epoch: 7 [281/500 17984/32000 (56%)] Loss: 1.91684 (QuantReg: 13.18966) QuantErr: 13.18966 batch_time=0.40430
Train Epoch: 7 [289/500 18496/32000 (58%)] Loss: 1.13431 (QuantReg: 12.17914) QuantErr: 12.17914 batch_time=0.39274
Train Epoch: 7 [297/500 19008/32000 (59%)] Loss: 1.02320 (QuantReg: 12.71900) QuantErr: 12.71900 batch_time=0.38671
Train Epoch: 7 [305/500 19520/32000 (61%)] Loss: 1.39782 (QuantReg: 12.26230) QuantErr: 12.26230 batch_time=0.39128
Train Epoch: 7 [313/500 20032/32000 (63%)] Loss: 1.31017 (QuantReg: 12.85627) QuantErr: 12.85627 batch_time=0.39624
Train Epoch: 7 [321/500 20544/32000 (64%)] Loss: 1.26962 (QuantReg: 13.16357) QuantErr: 13.16357 batch_time=0.42325
Train Epoch: 7 [329/500 21056/32000 (66%)] Loss: 1.56009 (QuantReg: 12.89327) QuantErr: 12.89327 batch_time=0.58206
Train Epoch: 7 [337/500 21568/32000 (67%)] Loss: 1.25106 (QuantReg: 12.79889) QuantErr: 12.79889 batch_time=0.40533
Train Epoch: 7 [345/500 22080/32000 (69%)] Loss: 1.51934 (QuantReg: 13.12570) QuantErr: 13.12570 batch_time=0.38816
Train Epoch: 7 [353/500 22592/32000 (71%)] Loss: 1.95656 (QuantReg: 12.56301) QuantErr: 12.56301 batch_time=0.41448
Train Epoch: 7 [361/500 23104/32000 (72%)] Loss: 1.30864 (QuantReg: 12.20008) QuantErr: 12.20008 batch_time=0.39667
Train Epoch: 7 [369/500 23616/32000 (74%)] Loss: 1.43813 (QuantReg: 13.42429) QuantErr: 13.42429 batch_time=0.40320
Train Epoch: 7 [377/500 24128/32000 (75%)] Loss: 1.65338 (QuantReg: 13.02841) QuantErr: 13.02841 batch_time=0.41679
Train Epoch: 7 [385/500 24640/32000 (77%)] Loss: 1.58865 (QuantReg: 12.82321) QuantErr: 12.82321 batch_time=0.40685
Train Epoch: 7 [393/500 25152/32000 (79%)] Loss: 1.68278 (QuantReg: 12.45836) QuantErr: 12.45836 batch_time=0.39724
Train Epoch: 7 [401/500 25664/32000 (80%)] Loss: 1.30147 (QuantReg: 12.70099) QuantErr: 12.70099 batch_time=0.39541
Train Epoch: 7 [409/500 26176/32000 (82%)] Loss: 1.13165 (QuantReg: 12.86405) QuantErr: 12.86405 batch_time=0.40589
Train Epoch: 7 [417/500 26688/32000 (83%)] Loss: 1.03532 (QuantReg: 12.76072) QuantErr: 12.76072 batch_time=0.39405
Train Epoch: 7 [425/500 27200/32000 (85%)] Loss: 1.16949 (QuantReg: 12.89848) QuantErr: 12.89848 batch_time=0.39089
Train Epoch: 7 [433/500 27712/32000 (87%)] Loss: 1.32808 (QuantReg: 13.01906) QuantErr: 13.01906 batch_time=0.39200
Train Epoch: 7 [441/500 28224/32000 (88%)] Loss: 1.88165 (QuantReg: 13.01212) QuantErr: 13.01212 batch_time=0.39291
Train Epoch: 7 [449/500 28736/32000 (90%)] Loss: 0.94399 (QuantReg: 13.20956) QuantErr: 13.20956 batch_time=0.40240
Train Epoch: 7 [457/500 29248/32000 (91%)] Loss: 1.04493 (QuantReg: 13.15670) QuantErr: 13.15670 batch_time=0.41762
Train Epoch: 7 [465/500 29760/32000 (93%)] Loss: 1.50957 (QuantReg: 12.89462) QuantErr: 12.89462 batch_time=0.40047
Train Epoch: 7 [473/500 30272/32000 (95%)] Loss: 1.37466 (QuantReg: 13.00454) QuantErr: 13.00454 batch_time=0.39745
Train Epoch: 7 [481/500 30784/32000 (96%)] Loss: 1.34802 (QuantReg: 13.24660) QuantErr: 13.24660 batch_time=0.40798
Train Epoch: 7 [489/500 31296/32000 (98%)] Loss: 1.49278 (QuantReg: 13.03930) QuantErr: 13.03930 batch_time=0.43794
Train Epoch: 7 [497/500 31808/32000 (99%)] Loss: 1.68291 (QuantReg: 12.97049) QuantErr: 12.97049 batch_time=0.40276
Train Epoch: 7 codebook_update_time=2.53568
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch7.pth ...
Done in 5.651s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch7.pth ...
Done in 11.956s
removing stale ckpt [epoch 6] [took 0.66s]
epoch : 7
loss : 1.3080506743192672
quant_reg : 12.73624073600769
quant_err : 12.73624073600769
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.048289738430583
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.35640171430685
MSRVTT_full_val/v2t_metrics/R1: 33.199195171026155
MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.659959758551308
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.89453860490258
MSRVTT_full_test/t2v_metrics/R1: 10.568561872909699
MSRVTT_full_test/t2v_metrics/R5: 32.14046822742475
MSRVTT_full_test/t2v_metrics/R10: 46.32107023411371
MSRVTT_full_test/t2v_metrics/R50: 79.03010033444816
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.61204013377927
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.058143553672554
MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
MSRVTT_full_test/v2t_metrics/R5: 36.58862876254181
MSRVTT_full_test/v2t_metrics/R10: 51.00334448160535
MSRVTT_full_test/v2t_metrics/R50: 84.38127090301003
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.629096989966555
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.956166643796887
mnt_best : 25.058143553672554
not_improved_count: 0
Train Epoch: 8 [1/500 64/32000 (0%)] Loss: 1.55842 (QuantReg: 12.68894) QuantErr: 12.68894 batch_time=29.33590
Train Epoch: 8 [9/500 576/32000 (2%)] Loss: 1.34858 (QuantReg: 12.80345) QuantErr: 12.80345 batch_time=1.00101
Train Epoch: 8 [17/500 1088/32000 (3%)] Loss: 1.20706 (QuantReg: 12.65138) QuantErr: 12.65138 batch_time=0.39412
Train Epoch: 8 [25/500 1600/32000 (5%)] Loss: 1.13191 (QuantReg: 12.86974) QuantErr: 12.86974 batch_time=0.42482
Train Epoch: 8 [33/500 2112/32000 (7%)] Loss: 1.37290 (QuantReg: 12.29689) QuantErr: 12.29689 batch_time=0.41068
Train Epoch: 8 [41/500 2624/32000 (8%)] Loss: 1.35279 (QuantReg: 12.63088) QuantErr: 12.63088 batch_time=0.43044
Train Epoch: 8 [49/500 3136/32000 (10%)] Loss: 1.06518 (QuantReg: 12.48697) QuantErr: 12.48697 batch_time=0.39392
Train Epoch: 8 [57/500 3648/32000 (11%)] Loss: 1.21842 (QuantReg: 12.86059) QuantErr: 12.86059 batch_time=0.39280
Train Epoch: 8 [65/500 4160/32000 (13%)] Loss: 1.42282 (QuantReg: 12.52911) QuantErr: 12.52911 batch_time=0.57132
Train Epoch: 8 [73/500 4672/32000 (15%)] Loss: 1.34887 (QuantReg: 12.93160) QuantErr: 12.93160 batch_time=0.83991
Train Epoch: 8 [81/500 5184/32000 (16%)] Loss: 0.90887 (QuantReg: 12.80818) QuantErr: 12.80818 batch_time=0.39997
Train Epoch: 8 [89/500 5696/32000 (18%)] Loss: 1.67516 (QuantReg: 12.90873) QuantErr: 12.90873 batch_time=0.41426
Train Epoch: 8 [97/500 6208/32000 (19%)] Loss: 0.86731 (QuantReg: 12.91504) QuantErr: 12.91504 batch_time=0.39803
Train Epoch: 8 [105/500 6720/32000 (21%)] Loss: 1.10022 (QuantReg: 13.13116) QuantErr: 13.13116 batch_time=0.38661
Train Epoch: 8 [113/500 7232/32000 (23%)] Loss: 0.70313 (QuantReg: 12.86649) QuantErr: 12.86649 batch_time=0.45754
Train Epoch: 8 [121/500 7744/32000 (24%)] Loss: 1.37167 (QuantReg: 13.13373) QuantErr: 13.13373 batch_time=0.39805
Train Epoch: 8 [129/500 8256/32000 (26%)] Loss: 1.28220 (QuantReg: 12.43993) QuantErr: 12.43993 batch_time=0.57999
Train Epoch: 8 [137/500 8768/32000 (27%)] Loss: 0.43274 (QuantReg: 12.69907) QuantErr: 12.69907 batch_time=0.79236
Train Epoch: 8 [145/500 9280/32000 (29%)] Loss: 1.00589 (QuantReg: 13.26317) QuantErr: 13.26317 batch_time=0.40822
Train Epoch: 8 [153/500 9792/32000 (31%)] Loss: 1.26234 (QuantReg: 12.63164) QuantErr: 12.63164 batch_time=0.40864
Train Epoch: 8 [161/500 10304/32000 (32%)] Loss: 1.71040 (QuantReg: 13.32092) QuantErr: 13.32092 batch_time=0.40529
Train Epoch: 8 [169/500 10816/32000 (34%)] Loss: 1.54048 (QuantReg: 12.82443) QuantErr: 12.82443 batch_time=0.45898
Train Epoch: 8 [177/500 11328/32000 (35%)] Loss: 1.01596 (QuantReg: 12.96170) QuantErr: 12.96170 batch_time=1.64431
Train Epoch: 8 [185/500 11840/32000 (37%)] Loss: 0.74586 (QuantReg: 12.91616) QuantErr: 12.91616 batch_time=0.40086
Train Epoch: 8 [193/500 12352/32000 (39%)] Loss: 1.52682 (QuantReg: 12.42056) QuantErr: 12.42056 batch_time=0.39807
Train Epoch: 8 [201/500 12864/32000 (40%)] Loss: 1.55333 (QuantReg: 12.09552) QuantErr: 12.09552 batch_time=0.41097
Train Epoch: 8 [209/500 13376/32000 (42%)] Loss: 1.50344 (QuantReg: 12.76809) QuantErr: 12.76809 batch_time=0.38900
Train Epoch: 8 [217/500 13888/32000 (43%)] Loss: 1.45227 (QuantReg: 12.79904) QuantErr: 12.79904 batch_time=0.39408
Train Epoch: 8 [225/500 14400/32000 (45%)] Loss: 1.09667 (QuantReg: 12.63324) QuantErr: 12.63324 batch_time=0.39909
Train Epoch: 8 [233/500 14912/32000 (47%)] Loss: 1.30118 (QuantReg: 12.84026) QuantErr: 12.84026 batch_time=0.39940
Train Epoch: 8 [241/500 15424/32000 (48%)] Loss: 0.82106 (QuantReg: 12.69001) QuantErr: 12.69001 batch_time=1.66664
Train Epoch: 8 [249/500 15936/32000 (50%)] Loss: 1.84404 (QuantReg: 12.16315) QuantErr: 12.16315 batch_time=0.39369
Train Epoch: 8 [257/500 16448/32000 (51%)] Loss: 1.11804 (QuantReg: 12.36310) QuantErr: 12.36310 batch_time=0.42204
Train Epoch: 8 [265/500 16960/32000 (53%)] Loss: 0.90816 (QuantReg: 12.98158) QuantErr: 12.98158 batch_time=0.36726
Train Epoch: 8 [273/500 17472/32000 (55%)] Loss: 0.76186 (QuantReg: 12.46163) QuantErr: 12.46163 batch_time=0.43083
Train Epoch: 8 [281/500 17984/32000 (56%)] Loss: 1.20964 (QuantReg: 12.93160) QuantErr: 12.93160 batch_time=0.40366
Train Epoch: 8 [289/500 18496/32000 (58%)] Loss: 1.02551 (QuantReg: 13.15794) QuantErr: 13.15794 batch_time=0.39408
Train Epoch: 8 [297/500 19008/32000 (59%)] Loss: 1.04051 (QuantReg: 13.08180) QuantErr: 13.08180 batch_time=0.38340
Train Epoch: 8 [305/500 19520/32000 (61%)] Loss: 1.06953 (QuantReg: 12.88256) QuantErr: 12.88256 batch_time=0.42475
Train Epoch: 8 [313/500 20032/32000 (63%)] Loss: 1.51688 (QuantReg: 13.00453) QuantErr: 13.00453 batch_time=0.39177
Train Epoch: 8 [321/500 20544/32000 (64%)] Loss: 1.21984 (QuantReg: 12.52417) QuantErr: 12.52417 batch_time=0.38465
Train Epoch: 8 [329/500 21056/32000 (66%)] Loss: 1.34763 (QuantReg: 12.49238) QuantErr: 12.49238 batch_time=0.38998
Train Epoch: 8 [337/500 21568/32000 (67%)] Loss: 0.92828 (QuantReg: 13.10499) QuantErr: 13.10499 batch_time=0.40208
Train Epoch: 8 [345/500 22080/32000 (69%)] Loss: 1.25823 (QuantReg: 12.68890) QuantErr: 12.68890 batch_time=0.39353
Train Epoch: 8 [353/500 22592/32000 (71%)] Loss: 1.46366 (QuantReg: 12.76981) QuantErr: 12.76981 batch_time=0.39974
Train Epoch: 8 [361/500 23104/32000 (72%)] Loss: 1.27909 (QuantReg: 12.76572) QuantErr: 12.76572 batch_time=0.39207
Train Epoch: 8 [369/500 23616/32000 (74%)] Loss: 1.28306 (QuantReg: 12.83507) QuantErr: 12.83507 batch_time=0.39214
Train Epoch: 8 [377/500 24128/32000 (75%)] Loss: 1.28720 (QuantReg: 12.92552) QuantErr: 12.92552 batch_time=0.40043
Train Epoch: 8 [385/500 24640/32000 (77%)] Loss: 1.69558 (QuantReg: 12.56389) QuantErr: 12.56389 batch_time=0.40177
Train Epoch: 8 [393/500 25152/32000 (79%)] Loss: 1.09241 (QuantReg: 12.98970) QuantErr: 12.98970 batch_time=0.40594
Train Epoch: 8 [401/500 25664/32000 (80%)] Loss: 1.23037 (QuantReg: 12.27779) QuantErr: 12.27779 batch_time=0.38960
Train Epoch: 8 [409/500 26176/32000 (82%)] Loss: 1.51282 (QuantReg: 13.18117) QuantErr: 13.18117 batch_time=0.40228
Train Epoch: 8 [417/500 26688/32000 (83%)] Loss: 1.09120 (QuantReg: 12.95436) QuantErr: 12.95436 batch_time=0.40716
Train Epoch: 8 [425/500 27200/32000 (85%)] Loss: 1.25359 (QuantReg: 12.59213) QuantErr: 12.59213 batch_time=0.40077
Train Epoch: 8 [433/500 27712/32000 (87%)] Loss: 1.02360 (QuantReg: 13.30936) QuantErr: 13.30936 batch_time=0.38618
Train Epoch: 8 [441/500 28224/32000 (88%)] Loss: 0.68282 (QuantReg: 13.80806) QuantErr: 13.80806 batch_time=0.39277
Train Epoch: 8 [449/500 28736/32000 (90%)] Loss: 1.14396 (QuantReg: 13.21862) QuantErr: 13.21862 batch_time=0.40159
Train Epoch: 8 [457/500 29248/32000 (91%)] Loss: 1.53264 (QuantReg: 12.77875) QuantErr: 12.77875 batch_time=0.44635
Train Epoch: 8 [465/500 29760/32000 (93%)] Loss: 0.99200 (QuantReg: 13.28905) QuantErr: 13.28905 batch_time=0.38697
Train Epoch: 8 [473/500 30272/32000 (95%)] Loss: 1.25843 (QuantReg: 13.40023) QuantErr: 13.40023 batch_time=0.39268
Train Epoch: 8 [481/500 30784/32000 (96%)] Loss: 1.09755 (QuantReg: 12.88539) QuantErr: 12.88539 batch_time=0.38009
Train Epoch: 8 [489/500 31296/32000 (98%)] Loss: 1.01113 (QuantReg: 12.82397) QuantErr: 12.82397 batch_time=0.39490
Train Epoch: 8 [497/500 31808/32000 (99%)] Loss: 1.13089 (QuantReg: 12.48085) QuantErr: 12.48085 batch_time=0.39037
Train Epoch: 8 codebook_update_time=1.89601
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch8.pth ...
Done in 23.278s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch8.pth ...
Done in 29.089s
removing stale ckpt [epoch 7] [took 0.08s]
epoch : 8
loss : 1.2115695939064026
quant_reg : 12.862498723983764
quant_err : 12.862498723983764
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 4000
MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.104627766599597
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.51158823659063
MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
MSRVTT_full_val/v2t_metrics/R5: 71.22736418511066
MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.350100603621731
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.38116061166349
MSRVTT_full_test/t2v_metrics/R1: 11.57190635451505
MSRVTT_full_test/t2v_metrics/R5: 32.14046822742475
MSRVTT_full_test/t2v_metrics/R10: 45.752508361204015
MSRVTT_full_test/t2v_metrics/R50: 79.1304347826087
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.33311036789298
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.72116722505311
MSRVTT_full_test/v2t_metrics/R1: 12.909698996655518
MSRVTT_full_test/v2t_metrics/R5: 36.55518394648829
MSRVTT_full_test/v2t_metrics/R10: 52.14046822742475
MSRVTT_full_test/v2t_metrics/R50: 84.74916387959867
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.83595317725752
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.085732499640837
mnt_best : 25.72116722505311
not_improved_count: 0
Train Epoch: 9 [1/500 64/32000 (0%)] Loss: 0.99301 (QuantReg: 12.41275) QuantErr: 12.41275 batch_time=37.20070
Train Epoch: 9 [9/500 576/32000 (2%)] Loss: 1.56419 (QuantReg: 12.91445) QuantErr: 12.91445 batch_time=1.08790
Train Epoch: 9 [17/500 1088/32000 (3%)] Loss: 1.00348 (QuantReg: 12.37836) QuantErr: 12.37836 batch_time=0.38656
Train Epoch: 9 [25/500 1600/32000 (5%)] Loss: 1.40750 (QuantReg: 12.57606) QuantErr: 12.57606 batch_time=0.38764
Train Epoch: 9 [33/500 2112/32000 (7%)] Loss: 1.45614 (QuantReg: 12.85848) QuantErr: 12.85848 batch_time=0.39711
Train Epoch: 9 [41/500 2624/32000 (8%)] Loss: 1.33001 (QuantReg: 12.96681) QuantErr: 12.96681 batch_time=0.39312
Train Epoch: 9 [49/500 3136/32000 (10%)] Loss: 1.28344 (QuantReg: 13.11897) QuantErr: 13.11897 batch_time=0.40734
Train Epoch: 9 [57/500 3648/32000 (11%)] Loss: 1.30304 (QuantReg: 12.39273) QuantErr: 12.39273 batch_time=0.40085
Train Epoch: 9 [65/500 4160/32000 (13%)] Loss: 1.19913 (QuantReg: 12.94129) QuantErr: 12.94129 batch_time=3.37830
Train Epoch: 9 [73/500 4672/32000 (15%)] Loss: 0.97088 (QuantReg: 12.76013) QuantErr: 12.76013 batch_time=0.93141
Train Epoch: 9 [81/500 5184/32000 (16%)] Loss: 0.97116 (QuantReg: 13.14869) QuantErr: 13.14869 batch_time=0.43454
Train Epoch: 9 [89/500 5696/32000 (18%)] Loss: 0.87657 (QuantReg: 12.40771) QuantErr: 12.40771 batch_time=0.40593
Train Epoch: 9 [97/500 6208/32000 (19%)] Loss: 1.50191 (QuantReg: 12.92896) QuantErr: 12.92896 batch_time=0.45312
Train Epoch: 9 [105/500 6720/32000 (21%)] Loss: 0.76036 (QuantReg: 12.61481) QuantErr: 12.61481 batch_time=0.39033
Train Epoch: 9 [113/500 7232/32000 (23%)] Loss: 0.99558 (QuantReg: 12.89333) QuantErr: 12.89333 batch_time=0.39719
Train Epoch: 9 [121/500 7744/32000 (24%)] Loss: 0.94982 (QuantReg: 13.26719) QuantErr: 13.26719 batch_time=0.39742
Train Epoch: 9 [129/500 8256/32000 (26%)] Loss: 1.31320 (QuantReg: 13.23901) QuantErr: 13.23901 batch_time=3.27232
Train Epoch: 9 [137/500 8768/32000 (27%)] Loss: 1.22486 (QuantReg: 12.89528) QuantErr: 12.89528 batch_time=0.86163
Train Epoch: 9 [145/500 9280/32000 (29%)] Loss: 1.03611 (QuantReg: 12.97437) QuantErr: 12.97437 batch_time=0.39675
Train Epoch: 9 [153/500 9792/32000 (31%)] Loss: 1.65499 (QuantReg: 13.00704) QuantErr: 13.00704 batch_time=0.43313
Train Epoch: 9 [161/500 10304/32000 (32%)] Loss: 1.44207 (QuantReg: 12.89951) QuantErr: 12.89951 batch_time=0.43679
Train Epoch: 9 [169/500 10816/32000 (34%)] Loss: 1.16481 (QuantReg: 13.39339) QuantErr: 13.39339 batch_time=0.39322
Train Epoch: 9 [177/500 11328/32000 (35%)] Loss: 1.19234 (QuantReg: 12.54849) QuantErr: 12.54849 batch_time=0.40173
Train Epoch: 9 [185/500 11840/32000 (37%)] Loss: 0.88749 (QuantReg: 13.16688) QuantErr: 13.16688 batch_time=0.40905
Train Epoch: 9 [193/500 12352/32000 (39%)] Loss: 1.14075 (QuantReg: 12.94091) QuantErr: 12.94091 batch_time=3.16872
Train Epoch: 9 [201/500 12864/32000 (40%)] Loss: 1.19735 (QuantReg: 13.06623) QuantErr: 13.06623 batch_time=0.95419
Train Epoch: 9 [209/500 13376/32000 (42%)] Loss: 0.92110 (QuantReg: 12.64684) QuantErr: 12.64684 batch_time=0.39426
Train Epoch: 9 [217/500 13888/32000 (43%)] Loss: 0.70848 (QuantReg: 12.83695) QuantErr: 12.83695 batch_time=0.41585
Train Epoch: 9 [225/500 14400/32000 (45%)] Loss: 0.90654 (QuantReg: 13.13364) QuantErr: 13.13364 batch_time=0.40400
Train Epoch: 9 [233/500 14912/32000 (47%)] Loss: 1.19296 (QuantReg: 13.23134) QuantErr: 13.23134 batch_time=0.40369
Train Epoch: 9 [241/500 15424/32000 (48%)] Loss: 1.07528 (QuantReg: 12.74262) QuantErr: 12.74262 batch_time=0.40238
Train Epoch: 9 [249/500 15936/32000 (50%)] Loss: 1.07001 (QuantReg: 13.17001) QuantErr: 13.17001 batch_time=0.39806
Train Epoch: 9 [257/500 16448/32000 (51%)] Loss: 1.41199 (QuantReg: 12.61368) QuantErr: 12.61368 batch_time=3.51889
Train Epoch: 9 [265/500 16960/32000 (53%)] Loss: 1.20044 (QuantReg: 13.14703) QuantErr: 13.14703 batch_time=0.93228
Train Epoch: 9 [273/500 17472/32000 (55%)] Loss: 0.96192 (QuantReg: 12.68074) QuantErr: 12.68074 batch_time=0.38831
Train Epoch: 9 [281/500 17984/32000 (56%)] Loss: 1.15824 (QuantReg: 12.72560) QuantErr: 12.72560 batch_time=0.38991
Train Epoch: 9 [289/500 18496/32000 (58%)] Loss: 1.20939 (QuantReg: 12.76869) QuantErr: 12.76869 batch_time=0.38947
Train Epoch: 9 [297/500 19008/32000 (59%)] Loss: 1.12990 (QuantReg: 12.62537) QuantErr: 12.62537 batch_time=0.39538
Train Epoch: 9 [305/500 19520/32000 (61%)] Loss: 2.00929 (QuantReg: 12.47133) QuantErr: 12.47133 batch_time=0.41085
Train Epoch: 9 [313/500 20032/32000 (63%)] Loss: 1.44918 (QuantReg: 12.88950) QuantErr: 12.88950 batch_time=0.39336
Train Epoch: 9 [321/500 20544/32000 (64%)] Loss: 1.17321 (QuantReg: 12.72644) QuantErr: 12.72644 batch_time=3.23659
Train Epoch: 9 [329/500 21056/32000 (66%)] Loss: 0.97131 (QuantReg: 13.35077) QuantErr: 13.35077 batch_time=0.92712
Train Epoch: 9 [337/500 21568/32000 (67%)] Loss: 1.47357 (QuantReg: 13.27653) QuantErr: 13.27653 batch_time=0.39488
Train Epoch: 9 [345/500 22080/32000 (69%)] Loss: 0.67121 (QuantReg: 13.09030) QuantErr: 13.09030 batch_time=0.39994
Train Epoch: 9 [353/500 22592/32000 (71%)] Loss: 1.32708 (QuantReg: 12.87153) QuantErr: 12.87153 batch_time=0.40322
Train Epoch: 9 [361/500 23104/32000 (72%)] Loss: 0.99112 (QuantReg: 12.82763) QuantErr: 12.82763 batch_time=0.39581
Train Epoch: 9 [369/500 23616/32000 (74%)] Loss: 1.38739 (QuantReg: 13.50712) QuantErr: 13.50712 batch_time=0.40183
Train Epoch: 9 [377/500 24128/32000 (75%)] Loss: 0.94479 (QuantReg: 13.12572) QuantErr: 13.12572 batch_time=0.40268
Train Epoch: 9 [385/500 24640/32000 (77%)] Loss: 1.28195 (QuantReg: 13.18961) QuantErr: 13.18961 batch_time=2.48783
Train Epoch: 9 [393/500 25152/32000 (79%)] Loss: 1.08318 (QuantReg: 12.52872) QuantErr: 12.52872 batch_time=0.88514
Train Epoch: 9 [401/500 25664/32000 (80%)] Loss: 1.31621 (QuantReg: 12.94578) QuantErr: 12.94578 batch_time=0.38674
Train Epoch: 9 [409/500 26176/32000 (82%)] Loss: 1.10541 (QuantReg: 13.23524) QuantErr: 13.23524 batch_time=0.39530
Train Epoch: 9 [417/500 26688/32000 (83%)] Loss: 0.82774 (QuantReg: 13.16464) QuantErr: 13.16464 batch_time=0.38231
Train Epoch: 9 [425/500 27200/32000 (85%)] Loss: 1.21399 (QuantReg: 13.04692) QuantErr: 13.04692 batch_time=0.39340
Train Epoch: 9 [433/500 27712/32000 (87%)] Loss: 1.30288 (QuantReg: 12.89965) QuantErr: 12.89965 batch_time=0.42860
Train Epoch: 9 [441/500 28224/32000 (88%)] Loss: 1.10373 (QuantReg: 13.35847) QuantErr: 13.35847 batch_time=0.40495
Train Epoch: 9 [449/500 28736/32000 (90%)] Loss: 0.94747 (QuantReg: 13.32613) QuantErr: 13.32613 batch_time=2.92556
Train Epoch: 9 [457/500 29248/32000 (91%)] Loss: 0.97375 (QuantReg: 13.12235) QuantErr: 13.12235 batch_time=0.95021
Train Epoch: 9 [465/500 29760/32000 (93%)] Loss: 1.11510 (QuantReg: 13.00614) QuantErr: 13.00614 batch_time=0.40545
Train Epoch: 9 [473/500 30272/32000 (95%)] Loss: 1.51550 (QuantReg: 12.67106) QuantErr: 12.67106 batch_time=0.40581
Train Epoch: 9 [481/500 30784/32000 (96%)] Loss: 0.73549 (QuantReg: 13.38323) QuantErr: 13.38323 batch_time=0.43336
Train Epoch: 9 [489/500 31296/32000 (98%)] Loss: 0.84881 (QuantReg: 13.31977) QuantErr: 13.31977 batch_time=0.40123
Train Epoch: 9 [497/500 31808/32000 (99%)] Loss: 1.43438 (QuantReg: 13.37441) QuantErr: 13.37441 batch_time=0.39454
Train Epoch: 9 codebook_update_time=1.94753
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_bs64/checkpoint-epoch9.pth ...
Done in 6.260s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.0975127683877945
quant_reg : 12.951545705795288
quant_err : 12.951545705795288
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 4500
MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.808853118712275
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.078900202825814
MSRVTT_full_val/v2t_metrics/R1: 30.18108651911469
MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.5
MSRVTT_full_val/v2t_metrics/MeanR: 7.279678068410463
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.98379428060564
MSRVTT_full_test/t2v_metrics/R1: 11.33779264214047
MSRVTT_full_test/t2v_metrics/R5: 32.10702341137124
MSRVTT_full_test/t2v_metrics/R10: 46.08695652173913
MSRVTT_full_test/t2v_metrics/R50: 78.62876254180603
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.1886287625418
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.599738435635683
MSRVTT_full_test/v2t_metrics/R1: 13.076923076923077
MSRVTT_full_test/v2t_metrics/R5: 37.95986622073578
MSRVTT_full_test/v2t_metrics/R10: 51.67224080267559
MSRVTT_full_test/v2t_metrics/R50: 85.08361204013377
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.66020066889632
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.491430983504625
mnt_best : 25.72116722505311
not_improved_count: 1