-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_L1.txt
3303 lines (3303 loc) · 234 KB
/
HCQ_MSRVTT_full_L1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 718.8456420898438 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 31.74428963661194 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 195.07984566688538 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 199.8620789051056 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch0.pth ...
Done in 2.260s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch0.pth ...
Done in 4.208s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.2012072434607646
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 2.414486921529175
MSRVTT_full_val/t2v_metrics/R50: 9.6579476861167
MSRVTT_full_val/t2v_metrics/MedR: 258.0
MSRVTT_full_val/t2v_metrics/MeanR: 252.8893360160966
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.8370558644072049
MSRVTT_full_val/v2t_metrics/R1: 0.4024144869215292
MSRVTT_full_val/v2t_metrics/R5: 1.0060362173038229
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 10.261569416498993
MSRVTT_full_val/v2t_metrics/MedR: 252.0
MSRVTT_full_val/v2t_metrics/MeanR: 254.52515090543258
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.9339212944894927
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.23411371237458195
MSRVTT_full_test/t2v_metrics/R10: 0.3010033444816054
MSRVTT_full_test/t2v_metrics/R50: 1.806020066889632
MSRVTT_full_test/t2v_metrics/MedR: 1476.5
MSRVTT_full_test/t2v_metrics/MeanR: 1480.3498327759198
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.1330788363844947
MSRVTT_full_test/v2t_metrics/R1: 0.033444816053511704
MSRVTT_full_test/v2t_metrics/R5: 0.20066889632107024
MSRVTT_full_test/v2t_metrics/R10: 0.43478260869565216
MSRVTT_full_test/v2t_metrics/R50: 1.2374581939799332
MSRVTT_full_test/v2t_metrics/MedR: 1523.5
MSRVTT_full_test/v2t_metrics/MeanR: 1519.7267558528429
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14289828366882665
mnt_best : 0.1330788363844947
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 39.49795 (QuantReg: 22.52384) QuantErr: 22.52384 batch_time=31.60816
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 37.34332 (QuantReg: 22.38425) QuantErr: 22.38425 batch_time=0.39577
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 30.98862 (QuantReg: 22.66319) QuantErr: 22.66319 batch_time=0.41700
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 28.15693 (QuantReg: 22.66246) QuantErr: 22.66246 batch_time=0.40994
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 26.09302 (QuantReg: 22.66771) QuantErr: 22.66771 batch_time=0.40458
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 25.98850 (QuantReg: 22.56647) QuantErr: 22.56647 batch_time=0.38951
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 24.75508 (QuantReg: 22.54847) QuantErr: 22.54847 batch_time=0.40539
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 23.20779 (QuantReg: 22.65297) QuantErr: 22.65297 batch_time=0.43300
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 23.19925 (QuantReg: 22.64698) QuantErr: 22.64698 batch_time=0.39726
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 22.59953 (QuantReg: 22.55651) QuantErr: 22.55651 batch_time=0.40628
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 20.85506 (QuantReg: 22.59692) QuantErr: 22.59692 batch_time=0.43842
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 21.28959 (QuantReg: 22.62316) QuantErr: 22.62316 batch_time=0.43109
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 20.75331 (QuantReg: 22.66196) QuantErr: 22.66196 batch_time=0.40403
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 19.65504 (QuantReg: 22.62380) QuantErr: 22.62380 batch_time=0.41307
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 19.08725 (QuantReg: 22.66782) QuantErr: 22.66782 batch_time=0.42904
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 18.88899 (QuantReg: 22.67997) QuantErr: 22.67997 batch_time=0.41922
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 17.97786 (QuantReg: 22.68871) QuantErr: 22.68871 batch_time=0.39768
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 18.43876 (QuantReg: 22.74523) QuantErr: 22.74523 batch_time=0.42602
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 18.35728 (QuantReg: 22.70723) QuantErr: 22.70723 batch_time=2.15975
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 18.28521 (QuantReg: 22.74262) QuantErr: 22.74262 batch_time=0.42416
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 17.58568 (QuantReg: 22.69681) QuantErr: 22.69681 batch_time=0.41234
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 17.46659 (QuantReg: 22.72867) QuantErr: 22.72867 batch_time=0.41438
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 15.37688 (QuantReg: 22.65070) QuantErr: 22.65070 batch_time=0.41085
Train Epoch: 1 codebook_update_time=0.56089
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch1.pth ...
Done in 4.414s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch1.pth ...
Done in 8.720s
epoch : 1
loss : 22.399550048828125
quant_reg : 22.627704833984374
quant_err : 22.627704833984374
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 16.096579476861166
MSRVTT_full_val/t2v_metrics/R5: 46.27766599597585
MSRVTT_full_val/t2v_metrics/R10: 63.17907444668008
MSRVTT_full_val/t2v_metrics/R50: 93.96378269617706
MSRVTT_full_val/t2v_metrics/MedR: 6.0
MSRVTT_full_val/t2v_metrics/MeanR: 14.525150905432596
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 36.10434204616997
MSRVTT_full_val/v2t_metrics/R1: 19.718309859154928
MSRVTT_full_val/v2t_metrics/R5: 53.521126760563384
MSRVTT_full_val/v2t_metrics/R10: 67.6056338028169
MSRVTT_full_val/v2t_metrics/R50: 93.36016096579476
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 14.9476861167002
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 41.47559335748855
MSRVTT_full_test/t2v_metrics/R1: 4.548494983277592
MSRVTT_full_test/t2v_metrics/R5: 15.852842809364548
MSRVTT_full_test/t2v_metrics/R10: 25.986622073578594
MSRVTT_full_test/t2v_metrics/R50: 60.735785953177256
MSRVTT_full_test/t2v_metrics/MedR: 32.0
MSRVTT_full_test/t2v_metrics/MeanR: 87.53076923076924
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 12.328443086783441
MSRVTT_full_test/v2t_metrics/R1: 5.418060200668896
MSRVTT_full_test/v2t_metrics/R5: 20.066889632107024
MSRVTT_full_test/v2t_metrics/R10: 29.531772575250837
MSRVTT_full_test/v2t_metrics/R50: 63.979933110367895
MSRVTT_full_test/v2t_metrics/MedR: 28.0
MSRVTT_full_test/v2t_metrics/MeanR: 81.81086956521739
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 14.752687210486652
mnt_best : 12.328443086783441
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 18.10233 (QuantReg: 11.87307) QuantErr: 11.87307 batch_time=29.44275
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 17.77732 (QuantReg: 11.91683) QuantErr: 11.91683 batch_time=0.39909
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 17.72733 (QuantReg: 12.05382) QuantErr: 12.05382 batch_time=0.39026
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 14.69604 (QuantReg: 12.40336) QuantErr: 12.40336 batch_time=0.38426
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 15.10744 (QuantReg: 12.44991) QuantErr: 12.44991 batch_time=0.40691
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 15.95441 (QuantReg: 12.70280) QuantErr: 12.70280 batch_time=0.39451
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 15.05391 (QuantReg: 12.71427) QuantErr: 12.71427 batch_time=1.48034
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 15.24794 (QuantReg: 13.01373) QuantErr: 13.01373 batch_time=0.64609
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 12.43337 (QuantReg: 13.27625) QuantErr: 13.27625 batch_time=0.39033
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 14.45883 (QuantReg: 13.49717) QuantErr: 13.49717 batch_time=1.15059
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 17.37507 (QuantReg: 13.50080) QuantErr: 13.50080 batch_time=0.40580
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 13.97449 (QuantReg: 13.70474) QuantErr: 13.70474 batch_time=0.40856
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 15.98515 (QuantReg: 13.98277) QuantErr: 13.98277 batch_time=0.40062
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 14.62093 (QuantReg: 13.42213) QuantErr: 13.42213 batch_time=1.76242
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 15.66959 (QuantReg: 13.76247) QuantErr: 13.76247 batch_time=0.38856
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 17.46343 (QuantReg: 13.93058) QuantErr: 13.93058 batch_time=0.40141
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 13.65529 (QuantReg: 14.39009) QuantErr: 14.39009 batch_time=0.40352
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 14.13309 (QuantReg: 14.33200) QuantErr: 14.33200 batch_time=0.39500
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 14.36799 (QuantReg: 14.28442) QuantErr: 14.28442 batch_time=0.40388
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 16.09453 (QuantReg: 14.33061) QuantErr: 14.33061 batch_time=0.43805
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 14.20123 (QuantReg: 14.42404) QuantErr: 14.42404 batch_time=0.38907
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 13.84661 (QuantReg: 14.58818) QuantErr: 14.58818 batch_time=0.43294
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 13.82683 (QuantReg: 14.83412) QuantErr: 14.83412 batch_time=0.39782
Train Epoch: 2 codebook_update_time=0.46543
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch2.pth ...
Done in 4.130s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch2.pth ...
Done in 9.046s
removing stale ckpt [epoch 1] [took 0.33s]
removing stale ckpt [epoch 0] [took 0.03s]
epoch : 2
loss : 14.973683807373046
quant_reg : 13.490907512664794
quant_err : 13.490907512664794
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 21.327967806841045
MSRVTT_full_val/t2v_metrics/R5: 56.53923541247485
MSRVTT_full_val/t2v_metrics/R10: 71.22736418511066
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.776659959758552
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 44.121346755583296
MSRVTT_full_val/v2t_metrics/R1: 21.52917505030181
MSRVTT_full_val/v2t_metrics/R5: 61.16700201207244
MSRVTT_full_val/v2t_metrics/R10: 76.45875251509054
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.388329979879275
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 46.52188076891588
MSRVTT_full_test/t2v_metrics/R1: 7.558528428093646
MSRVTT_full_test/t2v_metrics/R5: 24.414715719063544
MSRVTT_full_test/t2v_metrics/R10: 36.05351170568562
MSRVTT_full_test/t2v_metrics/R50: 70.46822742474916
MSRVTT_full_test/t2v_metrics/MedR: 20.0
MSRVTT_full_test/t2v_metrics/MeanR: 65.54648829431439
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.80812482271822
MSRVTT_full_test/v2t_metrics/R1: 8.762541806020067
MSRVTT_full_test/v2t_metrics/R5: 27.49163879598662
MSRVTT_full_test/v2t_metrics/R10: 39.63210702341137
MSRVTT_full_test/v2t_metrics/R50: 73.9799331103679
MSRVTT_full_test/v2t_metrics/MedR: 17.0
MSRVTT_full_test/v2t_metrics/MeanR: 56.68494983277592
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.214166091750986
mnt_best : 18.80812482271822
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 14.02149 (QuantReg: 11.70376) QuantErr: 11.70376 batch_time=33.71718
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 12.53799 (QuantReg: 12.00362) QuantErr: 12.00362 batch_time=0.39284
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 13.39302 (QuantReg: 11.70116) QuantErr: 11.70116 batch_time=0.39433
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 14.00514 (QuantReg: 12.57281) QuantErr: 12.57281 batch_time=0.40830
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 13.11802 (QuantReg: 12.21014) QuantErr: 12.21014 batch_time=0.44480
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 12.48300 (QuantReg: 12.25318) QuantErr: 12.25318 batch_time=0.43726
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 11.61669 (QuantReg: 12.65630) QuantErr: 12.65630 batch_time=4.93298
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 12.18910 (QuantReg: 12.26964) QuantErr: 12.26964 batch_time=0.40203
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 13.26896 (QuantReg: 12.25048) QuantErr: 12.25048 batch_time=0.40465
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 12.41553 (QuantReg: 12.50833) QuantErr: 12.50833 batch_time=0.40584
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 12.11292 (QuantReg: 12.85411) QuantErr: 12.85411 batch_time=0.41929
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 13.46008 (QuantReg: 13.00527) QuantErr: 13.00527 batch_time=0.41125
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 11.41973 (QuantReg: 12.85218) QuantErr: 12.85218 batch_time=0.38730
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 12.29348 (QuantReg: 12.94612) QuantErr: 12.94612 batch_time=0.41524
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 12.10041 (QuantReg: 12.51146) QuantErr: 12.51146 batch_time=0.40188
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 12.12134 (QuantReg: 12.97768) QuantErr: 12.97768 batch_time=0.40585
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 10.38684 (QuantReg: 13.08421) QuantErr: 13.08421 batch_time=0.41124
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 11.61217 (QuantReg: 13.19003) QuantErr: 13.19003 batch_time=0.43571
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 13.26641 (QuantReg: 12.99355) QuantErr: 12.99355 batch_time=0.40142
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 11.26374 (QuantReg: 12.99477) QuantErr: 12.99477 batch_time=0.41204
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 12.41247 (QuantReg: 13.27071) QuantErr: 13.27071 batch_time=0.39980
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 12.65660 (QuantReg: 13.24327) QuantErr: 13.24327 batch_time=0.41098
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 11.64779 (QuantReg: 13.66502) QuantErr: 13.66502 batch_time=0.41934
Train Epoch: 3 codebook_update_time=0.42343
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch3.pth ...
Done in 4.326s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch3.pth ...
Done in 16.472s
removing stale ckpt [epoch 2] [took 0.03s]
epoch : 3
loss : 12.481977165222167
quant_reg : 12.754533981323242
quant_err : 12.754533981323242
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 24.748490945674043
MSRVTT_full_val/t2v_metrics/R5: 59.356136820925556
MSRVTT_full_val/t2v_metrics/R10: 75.85513078470825
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.064386317907445
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.12082936361093
MSRVTT_full_val/v2t_metrics/R1: 27.16297786720322
MSRVTT_full_val/v2t_metrics/R5: 65.3923541247485
MSRVTT_full_val/v2t_metrics/R10: 80.6841046277666
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.006036217303823
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.331611903894725
MSRVTT_full_test/t2v_metrics/R1: 8.62876254180602
MSRVTT_full_test/t2v_metrics/R5: 26.45484949832776
MSRVTT_full_test/t2v_metrics/R10: 37.9933110367893
MSRVTT_full_test/t2v_metrics/R50: 72.67558528428094
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 60.01438127090301
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.545670488621695
MSRVTT_full_test/v2t_metrics/R1: 9.498327759197325
MSRVTT_full_test/v2t_metrics/R5: 31.37123745819398
MSRVTT_full_test/v2t_metrics/R10: 43.91304347826087
MSRVTT_full_test/v2t_metrics/R50: 78.36120401337793
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.658026755852845
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.564457533292455
mnt_best : 20.545670488621695
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 10.66987 (QuantReg: 12.24523) QuantErr: 12.24523 batch_time=36.02622
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 12.69441 (QuantReg: 12.13173) QuantErr: 12.13173 batch_time=0.39948
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 9.37280 (QuantReg: 12.12717) QuantErr: 12.12717 batch_time=0.41037
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 10.51302 (QuantReg: 12.57289) QuantErr: 12.57289 batch_time=0.44039
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 9.94194 (QuantReg: 12.39471) QuantErr: 12.39471 batch_time=0.39088
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 10.39409 (QuantReg: 12.43028) QuantErr: 12.43028 batch_time=0.44260
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 11.09333 (QuantReg: 12.33693) QuantErr: 12.33693 batch_time=0.40141
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 10.36231 (QuantReg: 12.40902) QuantErr: 12.40902 batch_time=0.41327
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 10.21286 (QuantReg: 12.92824) QuantErr: 12.92824 batch_time=0.39176
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 9.93175 (QuantReg: 12.57999) QuantErr: 12.57999 batch_time=0.40715
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 8.59459 (QuantReg: 12.44403) QuantErr: 12.44403 batch_time=0.41872
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 11.02537 (QuantReg: 12.60856) QuantErr: 12.60856 batch_time=0.40086
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 11.20419 (QuantReg: 12.90561) QuantErr: 12.90561 batch_time=0.40171
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 10.36976 (QuantReg: 12.48410) QuantErr: 12.48410 batch_time=0.40793
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 8.96074 (QuantReg: 12.73000) QuantErr: 12.73000 batch_time=0.38836
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 11.61548 (QuantReg: 12.58820) QuantErr: 12.58820 batch_time=0.40519
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 9.93322 (QuantReg: 12.28435) QuantErr: 12.28435 batch_time=0.42595
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 12.31914 (QuantReg: 12.65851) QuantErr: 12.65851 batch_time=0.42629
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 11.26132 (QuantReg: 12.76093) QuantErr: 12.76093 batch_time=0.45770
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 10.31074 (QuantReg: 12.90252) QuantErr: 12.90252 batch_time=0.40263
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 9.55751 (QuantReg: 13.11491) QuantErr: 13.11491 batch_time=0.39040
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 11.19553 (QuantReg: 12.71603) QuantErr: 12.71603 batch_time=0.42243
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 10.53226 (QuantReg: 13.01500) QuantErr: 13.01500 batch_time=0.43562
Train Epoch: 4 codebook_update_time=0.67750
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch4.pth ...
Done in 23.598s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch4.pth ...
Done in 28.921s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 11.038489490509033
quant_reg : 12.632359203338623
quant_err : 12.632359203338623
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 24.748490945674043
MSRVTT_full_val/t2v_metrics/R5: 60.36217303822938
MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.503018108651911
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.56171977598913
MSRVTT_full_val/v2t_metrics/R1: 26.156941649899398
MSRVTT_full_val/v2t_metrics/R5: 67.00201207243461
MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.183098591549296
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.44216056318534
MSRVTT_full_test/t2v_metrics/R1: 8.695652173913043
MSRVTT_full_test/t2v_metrics/R5: 26.82274247491639
MSRVTT_full_test/t2v_metrics/R10: 39.063545150501675
MSRVTT_full_test/t2v_metrics/R50: 73.87959866220736
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.14866220735786
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.886179111503036
MSRVTT_full_test/v2t_metrics/R1: 10.602006688963211
MSRVTT_full_test/v2t_metrics/R5: 31.872909698996654
MSRVTT_full_test/v2t_metrics/R10: 46.32107023411371
MSRVTT_full_test/v2t_metrics/R50: 80.06688963210702
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 42.93946488294314
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.014747517729017
mnt_best : 20.886179111503036
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 10.71105 (QuantReg: 12.13640) QuantErr: 12.13640 batch_time=35.62675
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 10.40845 (QuantReg: 12.43810) QuantErr: 12.43810 batch_time=0.40240
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 9.77889 (QuantReg: 12.42992) QuantErr: 12.42992 batch_time=0.40218
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 10.50717 (QuantReg: 12.35401) QuantErr: 12.35401 batch_time=0.38828
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 10.74602 (QuantReg: 12.29907) QuantErr: 12.29907 batch_time=0.39416
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 9.25035 (QuantReg: 12.47705) QuantErr: 12.47705 batch_time=0.55626
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 11.23244 (QuantReg: 12.45434) QuantErr: 12.45434 batch_time=0.40601
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 10.37747 (QuantReg: 12.48346) QuantErr: 12.48346 batch_time=0.39045
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 11.63840 (QuantReg: 12.54638) QuantErr: 12.54638 batch_time=0.44736
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 12.21579 (QuantReg: 12.54743) QuantErr: 12.54743 batch_time=0.38249
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 11.17505 (QuantReg: 12.41008) QuantErr: 12.41008 batch_time=0.39083
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 10.37071 (QuantReg: 12.74586) QuantErr: 12.74586 batch_time=0.37538
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 11.19382 (QuantReg: 12.49717) QuantErr: 12.49717 batch_time=0.43799
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 8.75254 (QuantReg: 12.99968) QuantErr: 12.99968 batch_time=0.43104
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 11.54028 (QuantReg: 12.63180) QuantErr: 12.63180 batch_time=0.40877
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 8.76136 (QuantReg: 12.86229) QuantErr: 12.86229 batch_time=0.42179
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 11.74095 (QuantReg: 12.62530) QuantErr: 12.62530 batch_time=0.40299
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 9.93086 (QuantReg: 12.90833) QuantErr: 12.90833 batch_time=0.43350
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 9.95438 (QuantReg: 12.80993) QuantErr: 12.80993 batch_time=0.40543
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 8.28265 (QuantReg: 12.64763) QuantErr: 12.64763 batch_time=0.41700
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 9.66545 (QuantReg: 12.87011) QuantErr: 12.87011 batch_time=0.38501
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 9.20818 (QuantReg: 12.59995) QuantErr: 12.59995 batch_time=0.46726
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 7.14364 (QuantReg: 13.04236) QuantErr: 13.04236 batch_time=0.53458
Train Epoch: 5 codebook_update_time=0.46558
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch5.pth ...
Done in 5.161s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch5.pth ...
Done in 10.161s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 10.045966205596924
quant_reg : 12.650605129241944
quant_err : 12.650605129241944
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 64.1851106639839
MSRVTT_full_val/t2v_metrics/R10: 77.2635814889336
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.434607645875252
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.61810448491835
MSRVTT_full_val/v2t_metrics/R1: 30.58350100603622
MSRVTT_full_val/v2t_metrics/R5: 69.61770623742454
MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.21327967806841
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.22975687881379
MSRVTT_full_test/t2v_metrics/R1: 9.19732441471572
MSRVTT_full_test/t2v_metrics/R5: 29.297658862876254
MSRVTT_full_test/t2v_metrics/R10: 43.41137123745819
MSRVTT_full_test/t2v_metrics/R50: 76.8561872909699
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.3505016722408
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.70035466163077
MSRVTT_full_test/v2t_metrics/R1: 12.341137123745819
MSRVTT_full_test/v2t_metrics/R5: 34.74916387959866
MSRVTT_full_test/v2t_metrics/R10: 48.862876254180605
MSRVTT_full_test/v2t_metrics/R50: 81.93979933110369
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.083946488294316
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.5693285003312
mnt_best : 22.70035466163077
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 9.14635 (QuantReg: 12.61827) QuantErr: 12.61827 batch_time=32.59657
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 8.57504 (QuantReg: 12.51983) QuantErr: 12.51983 batch_time=0.40194
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 9.18007 (QuantReg: 12.53815) QuantErr: 12.53815 batch_time=0.42465
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 9.52044 (QuantReg: 12.14651) QuantErr: 12.14651 batch_time=0.43719
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 7.71904 (QuantReg: 12.36567) QuantErr: 12.36567 batch_time=0.39861
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 10.40046 (QuantReg: 12.38438) QuantErr: 12.38438 batch_time=0.38821
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 8.87288 (QuantReg: 12.64741) QuantErr: 12.64741 batch_time=0.40608
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 9.26620 (QuantReg: 12.72392) QuantErr: 12.72392 batch_time=0.44266
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 10.56551 (QuantReg: 12.49788) QuantErr: 12.49788 batch_time=0.41119
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 9.20042 (QuantReg: 12.84590) QuantErr: 12.84590 batch_time=0.41517
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 7.56449 (QuantReg: 12.56962) QuantErr: 12.56962 batch_time=0.40852
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 9.73160 (QuantReg: 12.55550) QuantErr: 12.55550 batch_time=0.40613
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 9.89241 (QuantReg: 12.81250) QuantErr: 12.81250 batch_time=0.41645
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 9.52533 (QuantReg: 12.80872) QuantErr: 12.80872 batch_time=2.00944
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 10.41447 (QuantReg: 12.86673) QuantErr: 12.86673 batch_time=1.27097
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 9.54800 (QuantReg: 12.74190) QuantErr: 12.74190 batch_time=0.40726
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 9.55612 (QuantReg: 12.79424) QuantErr: 12.79424 batch_time=0.39770
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 9.16665 (QuantReg: 13.09422) QuantErr: 13.09422 batch_time=0.65067
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 8.49284 (QuantReg: 12.82249) QuantErr: 12.82249 batch_time=0.42040
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 8.34840 (QuantReg: 12.89504) QuantErr: 12.89504 batch_time=0.45271
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 8.74470 (QuantReg: 12.80721) QuantErr: 12.80721 batch_time=0.39536
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 8.68264 (QuantReg: 12.97554) QuantErr: 12.97554 batch_time=1.60442
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 9.75568 (QuantReg: 12.89647) QuantErr: 12.89647 batch_time=0.40840
Train Epoch: 6 codebook_update_time=0.63051
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch6.pth ...
Done in 4.852s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch6.pth ...
Done in 9.180s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 9.124582710266113
quant_reg : 12.705857822418213
quant_err : 12.705857822418213
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 27.766599597585515
MSRVTT_full_val/t2v_metrics/R5: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.482897384305835
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.51515257034553
MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
MSRVTT_full_val/v2t_metrics/R5: 70.62374245472837
MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.227364185110664
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.996708662803314
MSRVTT_full_test/t2v_metrics/R1: 9.765886287625419
MSRVTT_full_test/t2v_metrics/R5: 30.735785953177256
MSRVTT_full_test/t2v_metrics/R10: 44.48160535117057
MSRVTT_full_test/t2v_metrics/R50: 77.82608695652173
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.63795986622073
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.723502771683712
MSRVTT_full_test/v2t_metrics/R1: 11.237458193979933
MSRVTT_full_test/v2t_metrics/R5: 35.719063545150505
MSRVTT_full_test/v2t_metrics/R10: 50.869565217391305
MSRVTT_full_test/v2t_metrics/R50: 83.07692307692308
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.4876254180602
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.332250542633265
mnt_best : 23.723502771683712
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 8.66740 (QuantReg: 12.26917) QuantErr: 12.26917 batch_time=38.03647
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 9.23838 (QuantReg: 12.58732) QuantErr: 12.58732 batch_time=0.39034
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 8.80175 (QuantReg: 12.72493) QuantErr: 12.72493 batch_time=0.39019
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 7.14077 (QuantReg: 12.77327) QuantErr: 12.77327 batch_time=0.41771
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 8.18925 (QuantReg: 12.61812) QuantErr: 12.61812 batch_time=0.40032
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 8.00810 (QuantReg: 12.52519) QuantErr: 12.52519 batch_time=0.38909
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 10.02768 (QuantReg: 12.32197) QuantErr: 12.32197 batch_time=0.87404
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 9.60296 (QuantReg: 12.38981) QuantErr: 12.38981 batch_time=0.41573
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 8.98322 (QuantReg: 12.68363) QuantErr: 12.68363 batch_time=0.39168
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 9.22732 (QuantReg: 12.62011) QuantErr: 12.62011 batch_time=0.40233
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 8.25288 (QuantReg: 12.70265) QuantErr: 12.70265 batch_time=0.39643
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 9.21223 (QuantReg: 13.32178) QuantErr: 13.32178 batch_time=0.38735
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 9.01300 (QuantReg: 12.54767) QuantErr: 12.54767 batch_time=0.86999
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 10.20812 (QuantReg: 12.53410) QuantErr: 12.53410 batch_time=3.93220
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 8.78574 (QuantReg: 12.67858) QuantErr: 12.67858 batch_time=0.40592
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 10.58719 (QuantReg: 12.87107) QuantErr: 12.87107 batch_time=0.41028
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 7.98877 (QuantReg: 12.99805) QuantErr: 12.99805 batch_time=1.28491
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 9.75777 (QuantReg: 13.01333) QuantErr: 13.01333 batch_time=0.41172
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 9.14436 (QuantReg: 12.91026) QuantErr: 12.91026 batch_time=0.39753
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 9.47043 (QuantReg: 12.86142) QuantErr: 12.86142 batch_time=0.40048
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 6.46658 (QuantReg: 12.80296) QuantErr: 12.80296 batch_time=0.38925
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 7.67534 (QuantReg: 12.66438) QuantErr: 12.66438 batch_time=0.39002
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 7.83017 (QuantReg: 13.34122) QuantErr: 13.34122 batch_time=0.41926
Train Epoch: 7 codebook_update_time=0.44326
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch7.pth ...
Done in 23.292s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch7.pth ...
Done in 28.746s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 8.453279584884644
quant_reg : 12.749325252532959
quant_err : 12.749325252532959
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 25.955734406438633
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 77.06237424547284
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.653923541247485
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.76147421038054
MSRVTT_full_val/v2t_metrics/R1: 30.985915492957748
MSRVTT_full_val/v2t_metrics/R5: 71.22736418511066
MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.386317907444668
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.08950532139273
MSRVTT_full_test/t2v_metrics/R1: 10.0
MSRVTT_full_test/t2v_metrics/R5: 30.100334448160535
MSRVTT_full_test/t2v_metrics/R10: 44.414715719063544
MSRVTT_full_test/t2v_metrics/R50: 77.85953177257525
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.25484949832776
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.733733926204522
MSRVTT_full_test/v2t_metrics/R1: 11.77257525083612
MSRVTT_full_test/v2t_metrics/R5: 34.71571906354515
MSRVTT_full_test/v2t_metrics/R10: 50.43478260869565
MSRVTT_full_test/v2t_metrics/R50: 82.6086956521739
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.76672240802676
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.418430928992212
mnt_best : 23.733733926204522
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 7.77085 (QuantReg: 12.62397) QuantErr: 12.62397 batch_time=35.98919
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 7.84460 (QuantReg: 12.77481) QuantErr: 12.77481 batch_time=3.88137
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 7.95860 (QuantReg: 12.72100) QuantErr: 12.72100 batch_time=0.43437
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 7.96348 (QuantReg: 12.93909) QuantErr: 12.93909 batch_time=0.40958
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 8.34936 (QuantReg: 12.31043) QuantErr: 12.31043 batch_time=0.40208
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 10.07245 (QuantReg: 12.70868) QuantErr: 12.70868 batch_time=0.39119
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 7.29999 (QuantReg: 12.97077) QuantErr: 12.97077 batch_time=0.88919
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 7.17147 (QuantReg: 13.05650) QuantErr: 13.05650 batch_time=0.41031
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 8.02314 (QuantReg: 12.79714) QuantErr: 12.79714 batch_time=0.38622
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 9.47398 (QuantReg: 12.66828) QuantErr: 12.66828 batch_time=0.41679
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 7.32817 (QuantReg: 12.63902) QuantErr: 12.63902 batch_time=0.43747
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 8.14139 (QuantReg: 12.63560) QuantErr: 12.63560 batch_time=0.39229
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 7.50104 (QuantReg: 12.88100) QuantErr: 12.88100 batch_time=0.38892
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 8.35529 (QuantReg: 12.86603) QuantErr: 12.86603 batch_time=0.38656
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 8.25439 (QuantReg: 12.91891) QuantErr: 12.91891 batch_time=0.38637
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 7.80174 (QuantReg: 12.88395) QuantErr: 12.88395 batch_time=0.37852
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 7.39327 (QuantReg: 12.79852) QuantErr: 12.79852 batch_time=0.38701
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 7.97499 (QuantReg: 13.04577) QuantErr: 13.04577 batch_time=0.49721
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 8.02343 (QuantReg: 12.77414) QuantErr: 12.77414 batch_time=0.65214
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 8.49168 (QuantReg: 13.03554) QuantErr: 13.03554 batch_time=0.38956
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 6.56017 (QuantReg: 13.02319) QuantErr: 13.02319 batch_time=0.39408
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 6.27245 (QuantReg: 12.88964) QuantErr: 12.88964 batch_time=0.51133
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 7.89669 (QuantReg: 12.83150) QuantErr: 12.83150 batch_time=0.37989
Train Epoch: 8 codebook_update_time=0.43742
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch8.pth ...
Done in 4.973s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch8.pth ...
Done in 9.499s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 7.951428991317749
quant_reg : 12.857873432159424
quant_err : 12.857873432159424
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 70.22132796780684
MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.3762575452716295
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.0416536275967
MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R10: 87.92756539235413
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.400402414486922
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.344615953491285
MSRVTT_full_test/t2v_metrics/R1: 11.73913043478261
MSRVTT_full_test/t2v_metrics/R5: 33.07692307692308
MSRVTT_full_test/t2v_metrics/R10: 46.65551839464883
MSRVTT_full_test/t2v_metrics/R50: 79.53177257525084
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.226254180602005
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.263625894363518
MSRVTT_full_test/v2t_metrics/R1: 12.54180602006689
MSRVTT_full_test/v2t_metrics/R5: 37.22408026755853
MSRVTT_full_test/v2t_metrics/R10: 52.240802675585286
MSRVTT_full_test/v2t_metrics/R50: 84.08026755852843
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.966722408026754
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.999997839463767
mnt_best : 26.263625894363518
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 7.90188 (QuantReg: 12.21461) QuantErr: 12.21461 batch_time=33.94872
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 7.45036 (QuantReg: 12.82357) QuantErr: 12.82357 batch_time=0.40623
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 9.07741 (QuantReg: 12.65688) QuantErr: 12.65688 batch_time=0.86161
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 9.19606 (QuantReg: 12.48086) QuantErr: 12.48086 batch_time=0.40207
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 8.13097 (QuantReg: 12.83492) QuantErr: 12.83492 batch_time=0.42307
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 7.64007 (QuantReg: 12.90901) QuantErr: 12.90901 batch_time=0.40732
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 8.79595 (QuantReg: 12.63868) QuantErr: 12.63868 batch_time=0.38274
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 8.65936 (QuantReg: 12.91586) QuantErr: 12.91586 batch_time=0.42374
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 5.73016 (QuantReg: 13.05623) QuantErr: 13.05623 batch_time=0.38571
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 5.45056 (QuantReg: 12.90577) QuantErr: 12.90577 batch_time=0.38909
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 7.83929 (QuantReg: 12.90291) QuantErr: 12.90291 batch_time=0.40576
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 7.33324 (QuantReg: 13.18076) QuantErr: 13.18076 batch_time=0.45138
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 7.00854 (QuantReg: 13.00811) QuantErr: 13.00811 batch_time=0.39830
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 8.07313 (QuantReg: 12.74189) QuantErr: 12.74189 batch_time=0.39638
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 8.69144 (QuantReg: 12.70936) QuantErr: 12.70936 batch_time=0.40620
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 7.46786 (QuantReg: 13.04775) QuantErr: 13.04775 batch_time=0.43484
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 7.65220 (QuantReg: 12.79992) QuantErr: 12.79992 batch_time=0.40724
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 6.56224 (QuantReg: 13.38867) QuantErr: 13.38867 batch_time=0.41008
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 8.11316 (QuantReg: 13.01414) QuantErr: 13.01414 batch_time=0.39649
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 7.43387 (QuantReg: 13.35576) QuantErr: 13.35576 batch_time=0.43687
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.39862 (QuantReg: 12.99770) QuantErr: 12.99770 batch_time=0.42895
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 6.57992 (QuantReg: 13.19323) QuantErr: 13.19323 batch_time=0.82066
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 7.72266 (QuantReg: 12.84375) QuantErr: 12.84375 batch_time=0.40006
Train Epoch: 9 codebook_update_time=0.47202
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch9.pth ...
Done in 6.201s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 7.485374895095825
quant_reg : 12.91818058013916
quant_err : 12.91818058013916
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.476861167002012
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.230990787251386
MSRVTT_full_val/v2t_metrics/R1: 36.41851106639839
MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
MSRVTT_full_val/v2t_metrics/R10: 85.71428571428571
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.696177062374246
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.146761957741035
MSRVTT_full_test/t2v_metrics/R1: 10.635451505016722
MSRVTT_full_test/t2v_metrics/R5: 30.93645484949833
MSRVTT_full_test/t2v_metrics/R10: 44.34782608695652
MSRVTT_full_test/t2v_metrics/R50: 77.123745819398
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.718060200668894
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.436158716293466
MSRVTT_full_test/v2t_metrics/R1: 12.274247491638796
MSRVTT_full_test/v2t_metrics/R5: 37.09030100334448
MSRVTT_full_test/v2t_metrics/R10: 52.10702341137124
MSRVTT_full_test/v2t_metrics/R50: 83.04347826086956
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.61705685618729
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.733189223012765
mnt_best : 26.263625894363518
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 7.98403 (QuantReg: 12.60446) QuantErr: 12.60446 batch_time=40.89091
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 5.59983 (QuantReg: 12.88146) QuantErr: 12.88146 batch_time=0.40159
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 6.01090 (QuantReg: 12.57000) QuantErr: 12.57000 batch_time=0.41146
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 7.03343 (QuantReg: 12.43331) QuantErr: 12.43331 batch_time=0.40208
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 6.54912 (QuantReg: 13.18118) QuantErr: 13.18118 batch_time=0.39277
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 7.24209 (QuantReg: 12.55242) QuantErr: 12.55242 batch_time=0.39532
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 7.26451 (QuantReg: 12.76284) QuantErr: 12.76284 batch_time=3.97019
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 9.00258 (QuantReg: 13.09462) QuantErr: 13.09462 batch_time=0.39924
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 6.45661 (QuantReg: 12.84941) QuantErr: 12.84941 batch_time=0.39955
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 7.56940 (QuantReg: 12.94000) QuantErr: 12.94000 batch_time=0.41848
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 7.05272 (QuantReg: 12.77451) QuantErr: 12.77451 batch_time=0.42658
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 7.97122 (QuantReg: 13.13860) QuantErr: 13.13860 batch_time=0.40495
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 7.65662 (QuantReg: 12.48341) QuantErr: 12.48341 batch_time=0.38709
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 8.24879 (QuantReg: 12.94026) QuantErr: 12.94026 batch_time=0.41240
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 5.50606 (QuantReg: 13.36690) QuantErr: 13.36690 batch_time=0.45775
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 8.81540 (QuantReg: 13.01651) QuantErr: 13.01651 batch_time=0.42619
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 7.37475 (QuantReg: 13.05726) QuantErr: 13.05726 batch_time=0.39893
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 6.77784 (QuantReg: 13.40046) QuantErr: 13.40046 batch_time=0.41343
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 5.71620 (QuantReg: 12.92500) QuantErr: 12.92500 batch_time=0.39068
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 6.64957 (QuantReg: 12.80534) QuantErr: 12.80534 batch_time=0.41392
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.41110 (QuantReg: 12.99695) QuantErr: 12.99695 batch_time=0.41056
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 5.97659 (QuantReg: 13.09880) QuantErr: 13.09880 batch_time=0.40393
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 6.27055 (QuantReg: 13.14175) QuantErr: 13.14175 batch_time=0.40262
Train Epoch: 10 codebook_update_time=0.48303
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch10.pth ...
Done in 4.147s
removing stale ckpt [epoch 9] [took 0.08s]
epoch : 10
loss : 7.067066215515137
quant_reg : 12.944038570404052
quant_err : 12.944038570404052
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 30.382293762575454
MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.13682092555332
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.97404629737407
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 74.24547283702213
MSRVTT_full_val/v2t_metrics/R10: 86.72032193158954
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.575452716297787
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.85943359132971
MSRVTT_full_test/t2v_metrics/R1: 11.270903010033445
MSRVTT_full_test/t2v_metrics/R5: 33.11036789297659
MSRVTT_full_test/t2v_metrics/R10: 46.65551839464883
MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.46622073578595
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.918423352448595
MSRVTT_full_test/v2t_metrics/R1: 13.177257525083611
MSRVTT_full_test/v2t_metrics/R5: 38.46153846153846
MSRVTT_full_test/v2t_metrics/R10: 53.37792642140468
MSRVTT_full_test/v2t_metrics/R50: 83.7123745819398
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.52625418060201
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.0195696049355
mnt_best : 26.263625894363518
not_improved_count: 2
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 5.97322 (QuantReg: 12.89840) QuantErr: 12.89840 batch_time=33.75825
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 7.75796 (QuantReg: 12.64964) QuantErr: 12.64964 batch_time=0.40631
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 6.99095 (QuantReg: 12.75397) QuantErr: 12.75397 batch_time=0.70024
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 7.59066 (QuantReg: 12.31626) QuantErr: 12.31626 batch_time=0.39682
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 8.31740 (QuantReg: 12.84676) QuantErr: 12.84676 batch_time=0.47748
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 5.35719 (QuantReg: 12.78331) QuantErr: 12.78331 batch_time=0.39768
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 6.87802 (QuantReg: 13.08679) QuantErr: 13.08679 batch_time=0.97696
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 6.91077 (QuantReg: 12.85339) QuantErr: 12.85339 batch_time=0.39743
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 6.14752 (QuantReg: 13.03632) QuantErr: 13.03632 batch_time=0.40795
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 6.53312 (QuantReg: 12.93784) QuantErr: 12.93784 batch_time=0.39319
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 7.94156 (QuantReg: 13.05040) QuantErr: 13.05040 batch_time=0.40978
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 7.82302 (QuantReg: 12.93157) QuantErr: 12.93157 batch_time=0.54079
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 7.24476 (QuantReg: 12.85083) QuantErr: 12.85083 batch_time=0.40476
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 7.55065 (QuantReg: 13.01783) QuantErr: 13.01783 batch_time=0.39842
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 7.68376 (QuantReg: 12.90526) QuantErr: 12.90526 batch_time=0.41358
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 6.43544 (QuantReg: 13.05480) QuantErr: 13.05480 batch_time=0.44016
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 7.44890 (QuantReg: 12.92042) QuantErr: 12.92042 batch_time=0.41245
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 8.33270 (QuantReg: 12.99202) QuantErr: 12.99202 batch_time=0.45523
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 6.93947 (QuantReg: 12.88988) QuantErr: 12.88988 batch_time=0.42757
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 7.53977 (QuantReg: 12.87202) QuantErr: 12.87202 batch_time=1.94035
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 5.61969 (QuantReg: 13.30184) QuantErr: 13.30184 batch_time=0.40815
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 4.46524 (QuantReg: 13.02108) QuantErr: 13.02108 batch_time=0.41581
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 6.45186 (QuantReg: 13.33990) QuantErr: 13.33990 batch_time=0.53739
Train Epoch: 11 codebook_update_time=0.41068
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch11.pth ...
Done in 4.708s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 6.784471035003662
quant_reg : 12.957321010589599
quant_err : 12.957321010589599
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 27.364185110663986
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 78.06841046277665
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.661971830985916
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.20472597988572
MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.26056338028169
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.80320100609537
MSRVTT_full_test/t2v_metrics/R1: 10.602006688963211
MSRVTT_full_test/t2v_metrics/R5: 31.705685618729095
MSRVTT_full_test/t2v_metrics/R10: 44.64882943143812
MSRVTT_full_test/t2v_metrics/R50: 77.45819397993311
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.303177257525086
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.666740602385552
MSRVTT_full_test/v2t_metrics/R1: 11.906354515050166
MSRVTT_full_test/v2t_metrics/R5: 36.88963210702341
MSRVTT_full_test/v2t_metrics/R10: 51.438127090301
MSRVTT_full_test/v2t_metrics/R50: 83.24414715719064
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.42040133779264
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.26980158171471
mnt_best : 26.263625894363518
not_improved_count: 3
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 6.88349 (QuantReg: 12.87817) QuantErr: 12.87817 batch_time=33.93579
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 9.04668 (QuantReg: 12.60923) QuantErr: 12.60923 batch_time=0.73745
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 7.07218 (QuantReg: 12.85329) QuantErr: 12.85329 batch_time=0.39877
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 6.52361 (QuantReg: 12.99038) QuantErr: 12.99038 batch_time=0.40247
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 6.41294 (QuantReg: 12.98804) QuantErr: 12.98804 batch_time=0.39893
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 6.66519 (QuantReg: 12.72920) QuantErr: 12.72920 batch_time=0.40051
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 7.31886 (QuantReg: 12.73151) QuantErr: 12.73151 batch_time=0.41091
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 6.49320 (QuantReg: 13.24494) QuantErr: 13.24494 batch_time=0.41158
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 7.25924 (QuantReg: 12.81208) QuantErr: 12.81208 batch_time=0.40026
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 6.64161 (QuantReg: 12.53743) QuantErr: 12.53743 batch_time=0.40431
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 5.63496 (QuantReg: 12.72746) QuantErr: 12.72746 batch_time=0.40267
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 6.44168 (QuantReg: 12.67121) QuantErr: 12.67121 batch_time=0.39304
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 6.54852 (QuantReg: 12.81329) QuantErr: 12.81329 batch_time=0.40304
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 7.20043 (QuantReg: 13.04055) QuantErr: 13.04055 batch_time=1.05292
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 5.04929 (QuantReg: 13.12038) QuantErr: 13.12038 batch_time=0.42337
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 6.78187 (QuantReg: 13.00866) QuantErr: 13.00866 batch_time=0.42333
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 6.46314 (QuantReg: 12.86598) QuantErr: 12.86598 batch_time=0.42986
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 6.35378 (QuantReg: 13.01974) QuantErr: 13.01974 batch_time=0.65107
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 6.32099 (QuantReg: 13.14989) QuantErr: 13.14989 batch_time=0.40321
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 6.65861 (QuantReg: 13.05977) QuantErr: 13.05977 batch_time=0.40797
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 7.01458 (QuantReg: 12.88423) QuantErr: 12.88423 batch_time=0.39717
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 5.80835 (QuantReg: 13.37851) QuantErr: 13.37851 batch_time=0.39912
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 6.52368 (QuantReg: 13.18763) QuantErr: 13.18763 batch_time=0.40213
Train Epoch: 12 codebook_update_time=0.45854
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch12.pth ...
Done in 4.348s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch12.pth ...
Done in 8.341s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 6.466595821380615
quant_reg : 12.961618785858155
quant_err : 12.961618785858155
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 32.99798792756539
MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.29979879275654
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.04898566387895
MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
MSRVTT_full_val/v2t_metrics/R10: 87.12273641851107
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.479879275653923
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.13555987643377
MSRVTT_full_test/t2v_metrics/R1: 11.906354515050166
MSRVTT_full_test/t2v_metrics/R5: 35.01672240802676
MSRVTT_full_test/t2v_metrics/R10: 48.49498327759197
MSRVTT_full_test/t2v_metrics/R50: 80.33444816053512
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.12441471571906
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.24271407549861
MSRVTT_full_test/v2t_metrics/R1: 13.74581939799331
MSRVTT_full_test/v2t_metrics/R5: 40.802675585284284
MSRVTT_full_test/v2t_metrics/R10: 55.88628762541806
MSRVTT_full_test/v2t_metrics/R50: 84.61538461538461
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.10351170568562
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.52982152975473
mnt_best : 27.24271407549861
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 4.96877 (QuantReg: 13.09289) QuantErr: 13.09289 batch_time=36.51606
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 7.11337 (QuantReg: 13.26767) QuantErr: 13.26767 batch_time=0.40671
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 6.92568 (QuantReg: 13.14144) QuantErr: 13.14144 batch_time=0.40647
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.87298 (QuantReg: 12.55634) QuantErr: 12.55634 batch_time=0.39917
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 5.81412 (QuantReg: 13.21790) QuantErr: 13.21790 batch_time=0.86490
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 5.69225 (QuantReg: 13.18446) QuantErr: 13.18446 batch_time=0.42245
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 6.26741 (QuantReg: 12.78940) QuantErr: 12.78940 batch_time=3.23524
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 6.85882 (QuantReg: 12.91873) QuantErr: 12.91873 batch_time=0.39819
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 6.87634 (QuantReg: 13.11733) QuantErr: 13.11733 batch_time=0.38600
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 5.00557 (QuantReg: 12.83521) QuantErr: 12.83521 batch_time=0.40059
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 5.20763 (QuantReg: 12.96992) QuantErr: 12.96992 batch_time=0.38828
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 7.29963 (QuantReg: 12.75923) QuantErr: 12.75923 batch_time=0.40826
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 5.64597 (QuantReg: 13.21129) QuantErr: 13.21129 batch_time=0.38984
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 7.71781 (QuantReg: 12.51776) QuantErr: 12.51776 batch_time=1.39111
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 6.18323 (QuantReg: 13.01869) QuantErr: 13.01869 batch_time=0.39268
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 4.67523 (QuantReg: 13.10538) QuantErr: 13.10538 batch_time=0.40580
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 6.49612 (QuantReg: 13.28518) QuantErr: 13.28518 batch_time=0.39010
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 6.95741 (QuantReg: 13.41383) QuantErr: 13.41383 batch_time=0.40046
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 6.00491 (QuantReg: 13.13482) QuantErr: 13.13482 batch_time=0.40064
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 5.56747 (QuantReg: 13.14117) QuantErr: 13.14117 batch_time=1.50270
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 5.51672 (QuantReg: 13.29762) QuantErr: 13.29762 batch_time=0.38699
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 6.74762 (QuantReg: 13.10607) QuantErr: 13.10607 batch_time=0.39875
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 5.89857 (QuantReg: 13.17221) QuantErr: 13.17221 batch_time=0.41561
Train Epoch: 13 codebook_update_time=0.50943
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch13.pth ...
Done in 5.124s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 6.254003549575805
quant_reg : 13.042649738311768
quant_err : 13.042649738311768
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.895372233400402
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.303216881138894
MSRVTT_full_val/v2t_metrics/R1: 34.80885311871227
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.0321931589537225
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.65605254975875
MSRVTT_full_test/t2v_metrics/R1: 10.76923076923077
MSRVTT_full_test/t2v_metrics/R5: 32.97658862876254
MSRVTT_full_test/t2v_metrics/R10: 46.58862876254181
MSRVTT_full_test/t2v_metrics/R50: 79.96655518394648
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.11939799331104
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.481409428021863
MSRVTT_full_test/v2t_metrics/R1: 14.214046822742475
MSRVTT_full_test/v2t_metrics/R5: 39.531772575250834
MSRVTT_full_test/v2t_metrics/R10: 53.812709030100336
MSRVTT_full_test/v2t_metrics/R50: 84.64882943143813
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.058528428093645
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.154177879252117
mnt_best : 27.24271407549861
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 7.21242 (QuantReg: 13.00114) QuantErr: 13.00114 batch_time=36.55609
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 7.30973 (QuantReg: 13.01889) QuantErr: 13.01889 batch_time=0.40517
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 6.95847 (QuantReg: 12.90530) QuantErr: 12.90530 batch_time=0.42026
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 5.99214 (QuantReg: 12.73302) QuantErr: 12.73302 batch_time=0.42588
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 6.25416 (QuantReg: 12.86175) QuantErr: 12.86175 batch_time=0.42033
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 6.42371 (QuantReg: 12.87835) QuantErr: 12.87835 batch_time=0.42375
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 6.25424 (QuantReg: 13.07035) QuantErr: 13.07035 batch_time=0.45410
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 7.29057 (QuantReg: 12.67962) QuantErr: 12.67962 batch_time=0.41853
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.71089 (QuantReg: 13.09335) QuantErr: 13.09335 batch_time=0.41357
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 5.09835 (QuantReg: 12.77695) QuantErr: 12.77695 batch_time=0.40002
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 6.55493 (QuantReg: 12.96231) QuantErr: 12.96231 batch_time=0.40689
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 6.12961 (QuantReg: 13.06355) QuantErr: 13.06355 batch_time=0.39342
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 6.43954 (QuantReg: 13.52666) QuantErr: 13.52666 batch_time=0.41184
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 5.92646 (QuantReg: 13.31176) QuantErr: 13.31176 batch_time=0.41994
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.10043 (QuantReg: 13.27909) QuantErr: 13.27909 batch_time=0.39932
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 6.27578 (QuantReg: 12.95312) QuantErr: 12.95312 batch_time=1.06853
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 4.46029 (QuantReg: 13.17663) QuantErr: 13.17663 batch_time=0.39617
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 6.59735 (QuantReg: 12.86549) QuantErr: 12.86549 batch_time=0.38746
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 6.76525 (QuantReg: 13.07986) QuantErr: 13.07986 batch_time=0.40344
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 5.25296 (QuantReg: 13.02791) QuantErr: 13.02791 batch_time=1.44166
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 6.01355 (QuantReg: 13.13218) QuantErr: 13.13218 batch_time=0.39939
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 5.42980 (QuantReg: 12.76365) QuantErr: 12.76365 batch_time=0.95919
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 4.83278 (QuantReg: 13.19440) QuantErr: 13.19440 batch_time=0.44129
Train Epoch: 14 codebook_update_time=0.44685
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch14.pth ...
Done in 3.752s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 6.036417518615723
quant_reg : 13.083238750457763
quant_err : 13.083238750457763
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 68.61167002012073
MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.569416498993964
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.46162928638256
MSRVTT_full_val/v2t_metrics/R1: 36.82092555331992
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 87.32394366197182
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.832997987927565
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.25776057338753
MSRVTT_full_test/t2v_metrics/R1: 11.97324414715719
MSRVTT_full_test/t2v_metrics/R5: 34.147157190635454
MSRVTT_full_test/t2v_metrics/R10: 48.06020066889632
MSRVTT_full_test/t2v_metrics/R50: 79.89966555183946
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.15652173913043
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.984683209137128
MSRVTT_full_test/v2t_metrics/R1: 13.812709030100335
MSRVTT_full_test/v2t_metrics/R5: 41.80602006688963
MSRVTT_full_test/v2t_metrics/R10: 55.585284280936456
MSRVTT_full_test/v2t_metrics/R50: 85.55183946488295
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.833444816053515
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.780386385253493
mnt_best : 27.24271407549861
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 7.74913 (QuantReg: 12.57231) QuantErr: 12.57231 batch_time=30.06087
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 4.73178 (QuantReg: 13.23613) QuantErr: 13.23613 batch_time=0.44289
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 5.50698 (QuantReg: 12.95464) QuantErr: 12.95464 batch_time=0.42027
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 6.57769 (QuantReg: 12.80397) QuantErr: 12.80397 batch_time=0.53460
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 5.30353 (QuantReg: 13.23506) QuantErr: 13.23506 batch_time=0.40757
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 5.75007 (QuantReg: 12.90080) QuantErr: 12.90080 batch_time=0.40750
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 5.90563 (QuantReg: 13.07048) QuantErr: 13.07048 batch_time=0.40314
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 5.10818 (QuantReg: 12.98156) QuantErr: 12.98156 batch_time=0.40153
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 6.61573 (QuantReg: 13.04811) QuantErr: 13.04811 batch_time=0.41615
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 6.84640 (QuantReg: 13.04403) QuantErr: 13.04403 batch_time=0.41024
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 5.61115 (QuantReg: 12.94785) QuantErr: 12.94785 batch_time=0.41051
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 6.36905 (QuantReg: 13.30319) QuantErr: 13.30319 batch_time=0.41894
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.88896 (QuantReg: 12.83305) QuantErr: 12.83305 batch_time=0.39701
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 6.11091 (QuantReg: 13.24864) QuantErr: 13.24864 batch_time=0.40611
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 4.90399 (QuantReg: 13.36924) QuantErr: 13.36924 batch_time=0.47130
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 6.10574 (QuantReg: 13.15313) QuantErr: 13.15313 batch_time=0.40254
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 6.42347 (QuantReg: 13.12519) QuantErr: 13.12519 batch_time=0.41211
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 5.81926 (QuantReg: 13.31871) QuantErr: 13.31871 batch_time=0.41219
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 7.08675 (QuantReg: 12.72823) QuantErr: 12.72823 batch_time=0.43583
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 5.23602 (QuantReg: 13.53762) QuantErr: 13.53762 batch_time=0.39242
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 5.04547 (QuantReg: 13.25009) QuantErr: 13.25009 batch_time=0.41985
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 6.36019 (QuantReg: 12.92800) QuantErr: 12.92800 batch_time=0.40489
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 5.27257 (QuantReg: 12.97940) QuantErr: 12.97940 batch_time=0.41006
Train Epoch: 15 codebook_update_time=0.41129
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L1/checkpoint-epoch15.pth ...
Done in 16.249s
removing stale ckpt [epoch 14] [took 0.08s]
epoch : 15
loss : 5.770687482833862
quant_reg : 13.070415390014649
quant_err : 13.070415390014649
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 30.58350100603622
MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614