-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_M16.txt
2603 lines (2603 loc) · 190 KB
/
HCQ_MSRVTT_1kB_M16.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 961.0538115501404 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 115.5458436012268 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 94.90481948852539 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch0.pth ...
Done in 1.554s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch0.pth ...
Done in 3.038s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.2
MSRVTT_miech_test/t2v_metrics/R5: 0.8
MSRVTT_miech_test/t2v_metrics/R10: 1.5
MSRVTT_miech_test/t2v_metrics/R50: 5.4
MSRVTT_miech_test/t2v_metrics/MedR: 504.0
MSRVTT_miech_test/t2v_metrics/MeanR: 499.463
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.6214465011907718
MSRVTT_miech_test/v2t_metrics/R1: 0.0
MSRVTT_miech_test/v2t_metrics/R5: 0.4
MSRVTT_miech_test/v2t_metrics/R10: 1.1
MSRVTT_miech_test/v2t_metrics/R50: 5.3
MSRVTT_miech_test/v2t_metrics/MedR: 491.5
MSRVTT_miech_test/v2t_metrics/MeanR: 499.9535
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.6214465011907718
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.81094 (QuantReg: 16.73027) QuantErr: 16.73027 batch_time=27.21953
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.55246 (QuantReg: 16.73875) QuantErr: 16.73875 batch_time=0.46448
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.31548 (QuantReg: 16.65480) QuantErr: 16.65480 batch_time=0.47181
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.45460 (QuantReg: 16.64963) QuantErr: 16.64963 batch_time=0.46142
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.40411 (QuantReg: 16.66629) QuantErr: 16.66629 batch_time=0.46841
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.14331 (QuantReg: 16.69083) QuantErr: 16.69083 batch_time=0.45960
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.96338 (QuantReg: 16.72161) QuantErr: 16.72161 batch_time=2.04785
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.50070 (QuantReg: 16.70802) QuantErr: 16.70802 batch_time=0.45666
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.15449 (QuantReg: 16.68964) QuantErr: 16.68964 batch_time=0.47168
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.16041 (QuantReg: 16.69517) QuantErr: 16.69517 batch_time=0.49965
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.07101 (QuantReg: 16.68890) QuantErr: 16.68890 batch_time=0.45732
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.19285 (QuantReg: 16.69063) QuantErr: 16.69063 batch_time=0.46493
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.60392 (QuantReg: 16.68882) QuantErr: 16.68882 batch_time=5.55154
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.71939 (QuantReg: 16.69358) QuantErr: 16.69358 batch_time=0.46174
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.60073 (QuantReg: 16.71125) QuantErr: 16.71125 batch_time=0.48286
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.31244 (QuantReg: 16.71024) QuantErr: 16.71024 batch_time=0.48256
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.22291 (QuantReg: 16.69511) QuantErr: 16.69511 batch_time=0.46878
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.59846 (QuantReg: 16.67181) QuantErr: 16.67181 batch_time=0.46504
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.45110 (QuantReg: 16.70773) QuantErr: 16.70773 batch_time=0.46321
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.66964 (QuantReg: 16.69552) QuantErr: 16.69552 batch_time=0.46421
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.44630 (QuantReg: 16.68963) QuantErr: 16.68963 batch_time=0.46246
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.69038 (QuantReg: 16.68659) QuantErr: 16.68659 batch_time=1.09417
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.03021 (QuantReg: 16.70175) QuantErr: 16.70175 batch_time=0.53304
Train Epoch: 1 codebook_update_time=1.02907
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch1.pth ...
Done in 4.049s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch1.pth ...
Done in 8.149s
epoch : 1
loss : 5.388509889602661
quant_reg : 16.69260549926758
quant_err : 16.69260549926758
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 7.8
MSRVTT_miech_test/t2v_metrics/R5: 28.5
MSRVTT_miech_test/t2v_metrics/R10: 41.4
MSRVTT_miech_test/t2v_metrics/R50: 75.3
MSRVTT_miech_test/t2v_metrics/MedR: 15.0
MSRVTT_miech_test/t2v_metrics/MeanR: 50.54
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.95623538724141
MSRVTT_miech_test/v2t_metrics/R1: 9.3
MSRVTT_miech_test/v2t_metrics/R5: 28.5
MSRVTT_miech_test/v2t_metrics/R10: 41.8
MSRVTT_miech_test/v2t_metrics/R50: 75.4
MSRVTT_miech_test/v2t_metrics/MedR: 15.0
MSRVTT_miech_test/v2t_metrics/MeanR: 51.9345
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.292975057373503
mnt_best : 20.95623538724141
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.97513 (QuantReg: 6.97306) QuantErr: 6.97306 batch_time=33.18690
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.81661 (QuantReg: 7.14266) QuantErr: 7.14266 batch_time=0.48875
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.04463 (QuantReg: 7.33501) QuantErr: 7.33501 batch_time=0.49213
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.96944 (QuantReg: 7.37550) QuantErr: 7.37550 batch_time=0.46999
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.57529 (QuantReg: 7.39087) QuantErr: 7.39087 batch_time=0.46518
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.43218 (QuantReg: 7.54533) QuantErr: 7.54533 batch_time=0.44779
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.87065 (QuantReg: 7.72130) QuantErr: 7.72130 batch_time=0.98465
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.47680 (QuantReg: 7.57126) QuantErr: 7.57126 batch_time=0.45929
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.50946 (QuantReg: 7.82489) QuantErr: 7.82489 batch_time=0.46506
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.73751 (QuantReg: 8.22030) QuantErr: 8.22030 batch_time=0.45197
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.75302 (QuantReg: 8.13891) QuantErr: 8.13891 batch_time=0.46597
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.19769 (QuantReg: 8.19813) QuantErr: 8.19813 batch_time=0.46955
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.60841 (QuantReg: 8.44461) QuantErr: 8.44461 batch_time=0.44433
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.78505 (QuantReg: 8.52889) QuantErr: 8.52889 batch_time=2.46642
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.33649 (QuantReg: 8.31491) QuantErr: 8.31491 batch_time=0.50015
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.03036 (QuantReg: 8.38628) QuantErr: 8.38628 batch_time=0.46730
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.60747 (QuantReg: 8.32956) QuantErr: 8.32956 batch_time=0.46180
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 2.99575 (QuantReg: 8.66938) QuantErr: 8.66938 batch_time=0.45397
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.00943 (QuantReg: 8.59014) QuantErr: 8.59014 batch_time=0.45387
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.71626 (QuantReg: 8.79442) QuantErr: 8.79442 batch_time=0.45022
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.49012 (QuantReg: 9.04052) QuantErr: 9.04052 batch_time=0.46688
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.31390 (QuantReg: 9.20425) QuantErr: 9.20425 batch_time=0.44879
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.01687 (QuantReg: 9.18277) QuantErr: 9.18277 batch_time=0.45389
Train Epoch: 2 codebook_update_time=1.20928
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch2.pth ...
Done in 4.153s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch2.pth ...
Done in 10.449s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.6617276973724366
quant_reg : 8.188733043670654
quant_err : 8.188733043670654
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 11.1
MSRVTT_miech_test/t2v_metrics/R5: 34.5
MSRVTT_miech_test/t2v_metrics/R10: 47.9
MSRVTT_miech_test/t2v_metrics/R50: 82.5
MSRVTT_miech_test/t2v_metrics/MedR: 12.0
MSRVTT_miech_test/t2v_metrics/MeanR: 42.764
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.37297941290156
MSRVTT_miech_test/v2t_metrics/R1: 10.6
MSRVTT_miech_test/v2t_metrics/R5: 34.8
MSRVTT_miech_test/v2t_metrics/R10: 49.5
MSRVTT_miech_test/v2t_metrics/R50: 81.4
MSRVTT_miech_test/v2t_metrics/MedR: 11.0
MSRVTT_miech_test/v2t_metrics/MeanR: 42.277
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.332783552409456
mnt_best : 26.37297941290156
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.69465 (QuantReg: 7.46277) QuantErr: 7.46277 batch_time=30.07526
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.20184 (QuantReg: 7.26815) QuantErr: 7.26815 batch_time=0.48003
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.23164 (QuantReg: 7.36514) QuantErr: 7.36514 batch_time=0.47900
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.85190 (QuantReg: 7.31973) QuantErr: 7.31973 batch_time=0.45107
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.15243 (QuantReg: 7.84884) QuantErr: 7.84884 batch_time=0.49050
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.20523 (QuantReg: 7.58844) QuantErr: 7.58844 batch_time=0.47321
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.92630 (QuantReg: 7.62808) QuantErr: 7.62808 batch_time=5.83548
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 2.79126 (QuantReg: 7.46265) QuantErr: 7.46265 batch_time=0.51633
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.06112 (QuantReg: 7.79267) QuantErr: 7.79267 batch_time=0.45033
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.08014 (QuantReg: 7.83695) QuantErr: 7.83695 batch_time=0.46495
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.96282 (QuantReg: 7.62125) QuantErr: 7.62125 batch_time=0.47659
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.10464 (QuantReg: 7.78194) QuantErr: 7.78194 batch_time=0.44710
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.32921 (QuantReg: 8.02129) QuantErr: 8.02129 batch_time=0.45151
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.20104 (QuantReg: 8.01787) QuantErr: 8.01787 batch_time=0.44607
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 2.98199 (QuantReg: 7.77126) QuantErr: 7.77126 batch_time=0.45901
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.84744 (QuantReg: 7.99340) QuantErr: 7.99340 batch_time=0.44330
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.71671 (QuantReg: 8.04041) QuantErr: 8.04041 batch_time=0.44948
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.18927 (QuantReg: 8.07335) QuantErr: 8.07335 batch_time=0.51515
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.25185 (QuantReg: 8.31111) QuantErr: 8.31111 batch_time=0.45958
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.82539 (QuantReg: 8.06248) QuantErr: 8.06248 batch_time=0.44873
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.33631 (QuantReg: 8.21335) QuantErr: 8.21335 batch_time=0.49659
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.53051 (QuantReg: 8.28526) QuantErr: 8.28526 batch_time=0.48388
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.58430 (QuantReg: 8.26304) QuantErr: 8.26304 batch_time=0.45264
Train Epoch: 3 codebook_update_time=0.95570
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch3.pth ...
Done in 17.161s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch3.pth ...
Done in 21.111s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.1073292427062986
quant_reg : 7.826627775192261
quant_err : 7.826627775192261
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 12.6
MSRVTT_miech_test/t2v_metrics/R5: 37.4
MSRVTT_miech_test/t2v_metrics/R10: 51.5
MSRVTT_miech_test/t2v_metrics/R50: 83.8
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.3195
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.952303681661544
MSRVTT_miech_test/v2t_metrics/R1: 12.6
MSRVTT_miech_test/v2t_metrics/R5: 38.5
MSRVTT_miech_test/v2t_metrics/R10: 52.4
MSRVTT_miech_test/v2t_metrics/R50: 82.6
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.092
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.40272083662653
mnt_best : 28.952303681661544
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.76218 (QuantReg: 7.44658) QuantErr: 7.44658 batch_time=36.77431
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.76388 (QuantReg: 7.62311) QuantErr: 7.62311 batch_time=0.47133
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.67330 (QuantReg: 7.33612) QuantErr: 7.33612 batch_time=0.44800
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.03787 (QuantReg: 7.58789) QuantErr: 7.58789 batch_time=0.51913
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.81262 (QuantReg: 7.64140) QuantErr: 7.64140 batch_time=0.45216
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.42719 (QuantReg: 7.70101) QuantErr: 7.70101 batch_time=0.46130
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.00715 (QuantReg: 7.63710) QuantErr: 7.63710 batch_time=0.47726
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.38182 (QuantReg: 7.63715) QuantErr: 7.63715 batch_time=0.45379
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.70352 (QuantReg: 7.78614) QuantErr: 7.78614 batch_time=0.46179
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.62997 (QuantReg: 7.96894) QuantErr: 7.96894 batch_time=0.45196
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.16911 (QuantReg: 7.85369) QuantErr: 7.85369 batch_time=0.47381
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.06692 (QuantReg: 7.69980) QuantErr: 7.69980 batch_time=0.45123
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.43142 (QuantReg: 8.05340) QuantErr: 8.05340 batch_time=0.45828
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.66306 (QuantReg: 7.55979) QuantErr: 7.55979 batch_time=0.48568
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.72450 (QuantReg: 7.94057) QuantErr: 7.94057 batch_time=0.44518
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.89923 (QuantReg: 7.87005) QuantErr: 7.87005 batch_time=0.46171
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.58551 (QuantReg: 8.15011) QuantErr: 8.15011 batch_time=0.45986
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.79632 (QuantReg: 7.72604) QuantErr: 7.72604 batch_time=0.47663
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.84292 (QuantReg: 8.27799) QuantErr: 8.27799 batch_time=0.50404
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.48618 (QuantReg: 7.75107) QuantErr: 7.75107 batch_time=0.45748
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.43944 (QuantReg: 8.18284) QuantErr: 8.18284 batch_time=0.43914
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.38963 (QuantReg: 8.15597) QuantErr: 8.15597 batch_time=0.57292
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.36268 (QuantReg: 7.97602) QuantErr: 7.97602 batch_time=0.44177
Train Epoch: 4 codebook_update_time=2.02074
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch4.pth ...
Done in 3.924s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch4.pth ...
Done in 7.959s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 2.7172999153137205
quant_reg : 7.847575422286988
quant_err : 7.847575422286988
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 13.2
MSRVTT_miech_test/t2v_metrics/R5: 39.0
MSRVTT_miech_test/t2v_metrics/R10: 53.6
MSRVTT_miech_test/t2v_metrics/R50: 85.1
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.9555
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.218143272396905
MSRVTT_miech_test/v2t_metrics/R1: 14.1
MSRVTT_miech_test/v2t_metrics/R5: 40.1
MSRVTT_miech_test/v2t_metrics/R10: 55.5
MSRVTT_miech_test/v2t_metrics/R50: 83.7
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.336
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.541728521833676
mnt_best : 30.218143272396905
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.22866 (QuantReg: 7.48903) QuantErr: 7.48903 batch_time=32.23884
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.28877 (QuantReg: 7.82849) QuantErr: 7.82849 batch_time=0.43670
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.41003 (QuantReg: 7.67052) QuantErr: 7.67052 batch_time=0.52623
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.52311 (QuantReg: 7.76333) QuantErr: 7.76333 batch_time=0.46464
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.86972 (QuantReg: 7.77072) QuantErr: 7.77072 batch_time=0.47044
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.11127 (QuantReg: 7.78059) QuantErr: 7.78059 batch_time=0.45174
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.47397 (QuantReg: 7.86183) QuantErr: 7.86183 batch_time=0.53941
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.51202 (QuantReg: 7.84828) QuantErr: 7.84828 batch_time=0.47450
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.26452 (QuantReg: 7.61976) QuantErr: 7.61976 batch_time=0.45786
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.33605 (QuantReg: 8.07251) QuantErr: 8.07251 batch_time=0.46165
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.31534 (QuantReg: 7.93633) QuantErr: 7.93633 batch_time=0.47432
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.68695 (QuantReg: 7.92083) QuantErr: 7.92083 batch_time=0.44365
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.63357 (QuantReg: 8.09080) QuantErr: 8.09080 batch_time=0.48739
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.43916 (QuantReg: 7.77411) QuantErr: 7.77411 batch_time=0.45645
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.45303 (QuantReg: 7.80212) QuantErr: 7.80212 batch_time=0.43493
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 1.98143 (QuantReg: 8.16578) QuantErr: 8.16578 batch_time=0.45375
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.47706 (QuantReg: 8.10193) QuantErr: 8.10193 batch_time=0.45946
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.41957 (QuantReg: 7.97042) QuantErr: 7.97042 batch_time=0.45898
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.30556 (QuantReg: 8.07107) QuantErr: 8.07107 batch_time=0.48107
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.73752 (QuantReg: 7.94074) QuantErr: 7.94074 batch_time=0.50417
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.43888 (QuantReg: 7.90234) QuantErr: 7.90234 batch_time=0.46154
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.67713 (QuantReg: 8.04866) QuantErr: 8.04866 batch_time=0.45859
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.13836 (QuantReg: 8.32006) QuantErr: 8.32006 batch_time=0.47404
Train Epoch: 5 codebook_update_time=1.00013
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch5.pth ...
Done in 3.957s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch5.pth ...
Done in 8.242s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.485234401702881
quant_reg : 7.887817771911621
quant_err : 7.887817771911621
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 13.7
MSRVTT_miech_test/t2v_metrics/R5: 41.8
MSRVTT_miech_test/t2v_metrics/R10: 56.1
MSRVTT_miech_test/t2v_metrics/R50: 85.2
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.127
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.78971028978127
MSRVTT_miech_test/v2t_metrics/R1: 14.3
MSRVTT_miech_test/v2t_metrics/R5: 40.8
MSRVTT_miech_test/v2t_metrics/R10: 56.0
MSRVTT_miech_test/v2t_metrics/R50: 85.1
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.57
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.968928172521522
mnt_best : 31.78971028978127
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.23731 (QuantReg: 7.81805) QuantErr: 7.81805 batch_time=39.00692
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.13594 (QuantReg: 8.09431) QuantErr: 8.09431 batch_time=0.44293
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.50379 (QuantReg: 7.59292) QuantErr: 7.59292 batch_time=0.44852
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.51215 (QuantReg: 7.84921) QuantErr: 7.84921 batch_time=0.49365
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.17663 (QuantReg: 7.62049) QuantErr: 7.62049 batch_time=0.50734
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.30273 (QuantReg: 8.03846) QuantErr: 8.03846 batch_time=0.45618
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.50089 (QuantReg: 7.98113) QuantErr: 7.98113 batch_time=0.44657
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.37124 (QuantReg: 7.74817) QuantErr: 7.74817 batch_time=0.45223
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.31247 (QuantReg: 7.70070) QuantErr: 7.70070 batch_time=0.96558
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.29675 (QuantReg: 7.90966) QuantErr: 7.90966 batch_time=0.46314
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 1.95310 (QuantReg: 8.06638) QuantErr: 8.06638 batch_time=0.47455
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.30809 (QuantReg: 8.21255) QuantErr: 8.21255 batch_time=0.43804
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.32669 (QuantReg: 7.86008) QuantErr: 7.86008 batch_time=0.47418
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.53797 (QuantReg: 8.18979) QuantErr: 8.18979 batch_time=0.46812
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.44771 (QuantReg: 7.84630) QuantErr: 7.84630 batch_time=0.44447
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.32230 (QuantReg: 8.08929) QuantErr: 8.08929 batch_time=0.44154
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.51129 (QuantReg: 7.97120) QuantErr: 7.97120 batch_time=0.45168
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.60133 (QuantReg: 8.14701) QuantErr: 8.14701 batch_time=0.55154
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.80548 (QuantReg: 8.11519) QuantErr: 8.11519 batch_time=0.45106
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.41404 (QuantReg: 8.03190) QuantErr: 8.03190 batch_time=0.44458
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.35000 (QuantReg: 8.09598) QuantErr: 8.09598 batch_time=0.46181
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.12662 (QuantReg: 8.19868) QuantErr: 8.19868 batch_time=0.44340
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.17336 (QuantReg: 7.91471) QuantErr: 7.91471 batch_time=0.44796
Train Epoch: 6 codebook_update_time=0.92266
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch6.pth ...
Done in 3.990s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch6.pth ...
Done in 8.060s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.2692238759994505
quant_reg : 8.001437873840333
quant_err : 8.001437873840333
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 14.2
MSRVTT_miech_test/t2v_metrics/R5: 42.5
MSRVTT_miech_test/t2v_metrics/R10: 56.4
MSRVTT_miech_test/t2v_metrics/R50: 85.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.0315
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.40799223656117
MSRVTT_miech_test/v2t_metrics/R1: 15.3
MSRVTT_miech_test/v2t_metrics/R5: 43.1
MSRVTT_miech_test/v2t_metrics/R10: 56.9
MSRVTT_miech_test/v2t_metrics/R50: 85.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.5
MSRVTT_miech_test/v2t_metrics/MeanR: 35.105
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.478063012340776
mnt_best : 32.40799223656117
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.31434 (QuantReg: 7.96619) QuantErr: 7.96619 batch_time=35.83298
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.21266 (QuantReg: 7.70550) QuantErr: 7.70550 batch_time=0.44839
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.39437 (QuantReg: 8.26280) QuantErr: 8.26280 batch_time=0.67188
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.25663 (QuantReg: 7.69974) QuantErr: 7.69974 batch_time=0.46132
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.02522 (QuantReg: 7.97798) QuantErr: 7.97798 batch_time=0.50015
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.13667 (QuantReg: 7.92535) QuantErr: 7.92535 batch_time=0.45615
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.23413 (QuantReg: 8.18636) QuantErr: 8.18636 batch_time=0.47616
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.18754 (QuantReg: 7.85653) QuantErr: 7.85653 batch_time=0.49285
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.01721 (QuantReg: 8.19160) QuantErr: 8.19160 batch_time=0.44406
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.06549 (QuantReg: 8.21418) QuantErr: 8.21418 batch_time=0.46223
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.25962 (QuantReg: 7.89589) QuantErr: 7.89589 batch_time=0.48662
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.27214 (QuantReg: 8.03473) QuantErr: 8.03473 batch_time=0.44930
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.81199 (QuantReg: 7.96491) QuantErr: 7.96491 batch_time=3.04308
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.00742 (QuantReg: 8.15600) QuantErr: 8.15600 batch_time=0.66658
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.03339 (QuantReg: 8.28105) QuantErr: 8.28105 batch_time=0.44958
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.22726 (QuantReg: 8.35177) QuantErr: 8.35177 batch_time=0.45299
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.75204 (QuantReg: 8.26684) QuantErr: 8.26684 batch_time=0.47716
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.09640 (QuantReg: 8.00682) QuantErr: 8.00682 batch_time=0.45642
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.00880 (QuantReg: 8.12076) QuantErr: 8.12076 batch_time=0.46315
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.92186 (QuantReg: 8.26965) QuantErr: 8.26965 batch_time=0.46199
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.16013 (QuantReg: 8.32448) QuantErr: 8.32448 batch_time=0.45582
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.03898 (QuantReg: 8.19399) QuantErr: 8.19399 batch_time=0.50633
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.94786 (QuantReg: 8.32555) QuantErr: 8.32555 batch_time=0.45750
Train Epoch: 7 codebook_update_time=0.95807
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch7.pth ...
Done in 3.873s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch7.pth ...
Done in 7.727s
removing stale ckpt [epoch 6] [took 0.02s]
epoch : 7
loss : 2.1061579451560974
quant_reg : 8.066210456848145
quant_err : 8.066210456848145
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 15.8
MSRVTT_miech_test/t2v_metrics/R5: 43.9
MSRVTT_miech_test/t2v_metrics/R10: 57.0
MSRVTT_miech_test/t2v_metrics/R50: 86.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.8225
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.0668638069958
MSRVTT_miech_test/v2t_metrics/R1: 15.9
MSRVTT_miech_test/v2t_metrics/R5: 43.8
MSRVTT_miech_test/v2t_metrics/R10: 57.4
MSRVTT_miech_test/v2t_metrics/R50: 86.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.697500000000005
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.19225227182864
mnt_best : 34.0668638069958
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.08106 (QuantReg: 8.07400) QuantErr: 8.07400 batch_time=29.30857
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.05865 (QuantReg: 8.20905) QuantErr: 8.20905 batch_time=0.45869
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.22613 (QuantReg: 8.08582) QuantErr: 8.08582 batch_time=0.44303
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.00763 (QuantReg: 8.03549) QuantErr: 8.03549 batch_time=0.46615
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.12072 (QuantReg: 8.34309) QuantErr: 8.34309 batch_time=0.46421
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.17863 (QuantReg: 8.11041) QuantErr: 8.11041 batch_time=0.45345
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.84381 (QuantReg: 8.06801) QuantErr: 8.06801 batch_time=1.89482
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.19601 (QuantReg: 7.93996) QuantErr: 7.93996 batch_time=0.81200
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.80363 (QuantReg: 8.01551) QuantErr: 8.01551 batch_time=0.45119
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.61897 (QuantReg: 8.44633) QuantErr: 8.44633 batch_time=0.47854
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.25479 (QuantReg: 8.50804) QuantErr: 8.50804 batch_time=0.52519
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.84562 (QuantReg: 8.10803) QuantErr: 8.10803 batch_time=0.45460
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.85937 (QuantReg: 8.26161) QuantErr: 8.26161 batch_time=0.45155
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.02406 (QuantReg: 8.27931) QuantErr: 8.27931 batch_time=0.46319
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.61357 (QuantReg: 7.92823) QuantErr: 7.92823 batch_time=0.45196
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.93217 (QuantReg: 8.07445) QuantErr: 8.07445 batch_time=0.45908
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.23276 (QuantReg: 8.22869) QuantErr: 8.22869 batch_time=0.45546
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.87528 (QuantReg: 8.40858) QuantErr: 8.40858 batch_time=0.56704
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.75553 (QuantReg: 7.97957) QuantErr: 7.97957 batch_time=0.44549
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.73425 (QuantReg: 8.32686) QuantErr: 8.32686 batch_time=0.46215
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.80021 (QuantReg: 8.06524) QuantErr: 8.06524 batch_time=0.46114
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.92219 (QuantReg: 8.25013) QuantErr: 8.25013 batch_time=0.48008
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.27836 (QuantReg: 8.34792) QuantErr: 8.34792 batch_time=0.86471
Train Epoch: 8 codebook_update_time=1.04304
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch8.pth ...
Done in 4.358s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 1.9740468816757202
quant_reg : 8.151617033004761
quant_err : 8.151617033004761
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 13.8
MSRVTT_miech_test/t2v_metrics/R5: 42.2
MSRVTT_miech_test/t2v_metrics/R10: 58.3
MSRVTT_miech_test/t2v_metrics/R50: 86.7
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.7485
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.380734622169356
MSRVTT_miech_test/v2t_metrics/R1: 15.9
MSRVTT_miech_test/v2t_metrics/R5: 43.9
MSRVTT_miech_test/v2t_metrics/R10: 57.9
MSRVTT_miech_test/v2t_metrics/R50: 87.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 31.2515
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.31732303031623
mnt_best : 34.0668638069958
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.24216 (QuantReg: 8.06878) QuantErr: 8.06878 batch_time=40.88326
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.92767 (QuantReg: 7.79159) QuantErr: 7.79159 batch_time=0.48662
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.77262 (QuantReg: 7.87657) QuantErr: 7.87657 batch_time=0.44611
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.89537 (QuantReg: 8.32361) QuantErr: 8.32361 batch_time=0.45637
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.86561 (QuantReg: 8.02739) QuantErr: 8.02739 batch_time=0.46458
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.82842 (QuantReg: 8.14631) QuantErr: 8.14631 batch_time=0.44498
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.93447 (QuantReg: 8.21515) QuantErr: 8.21515 batch_time=0.47449
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.68635 (QuantReg: 8.04349) QuantErr: 8.04349 batch_time=0.51296
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.75890 (QuantReg: 7.89641) QuantErr: 7.89641 batch_time=0.44428
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.81297 (QuantReg: 8.00936) QuantErr: 8.00936 batch_time=0.46709
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.86151 (QuantReg: 8.15792) QuantErr: 8.15792 batch_time=0.44826
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.73584 (QuantReg: 8.59550) QuantErr: 8.59550 batch_time=0.44253
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.56672 (QuantReg: 8.16649) QuantErr: 8.16649 batch_time=0.65040
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.73728 (QuantReg: 8.45763) QuantErr: 8.45763 batch_time=0.66491
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.50678 (QuantReg: 8.21140) QuantErr: 8.21140 batch_time=0.45845
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.97087 (QuantReg: 8.33431) QuantErr: 8.33431 batch_time=0.45935
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.69697 (QuantReg: 8.31262) QuantErr: 8.31262 batch_time=0.68018
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.26714 (QuantReg: 8.29247) QuantErr: 8.29247 batch_time=0.47883
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.53073 (QuantReg: 8.14638) QuantErr: 8.14638 batch_time=0.44803
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.07401 (QuantReg: 8.23827) QuantErr: 8.23827 batch_time=0.46419
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.08815 (QuantReg: 8.21031) QuantErr: 8.21031 batch_time=0.44020
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.69942 (QuantReg: 8.31525) QuantErr: 8.31525 batch_time=0.44225
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.63503 (QuantReg: 8.36032) QuantErr: 8.36032 batch_time=0.44163
Train Epoch: 9 codebook_update_time=0.90227
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch9.pth ...
Done in 4.336s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.865707030773163
quant_reg : 8.228561639785767
quant_err : 8.228561639785767
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 15.2
MSRVTT_miech_test/t2v_metrics/R5: 42.3
MSRVTT_miech_test/t2v_metrics/R10: 56.9
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.591499999999996
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.196992518008884
MSRVTT_miech_test/v2t_metrics/R1: 15.7
MSRVTT_miech_test/v2t_metrics/R5: 44.1
MSRVTT_miech_test/v2t_metrics/R10: 58.2
MSRVTT_miech_test/v2t_metrics/R50: 87.6
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 31.4535
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.283651795574265
mnt_best : 34.0668638069958
not_improved_count: 2
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.91077 (QuantReg: 8.07942) QuantErr: 8.07942 batch_time=32.97625
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.72493 (QuantReg: 8.55114) QuantErr: 8.55114 batch_time=0.48921
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.21311 (QuantReg: 7.99022) QuantErr: 7.99022 batch_time=0.50037
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.84409 (QuantReg: 8.24886) QuantErr: 8.24886 batch_time=0.43595
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.49115 (QuantReg: 8.35929) QuantErr: 8.35929 batch_time=0.45119
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.76186 (QuantReg: 8.35053) QuantErr: 8.35053 batch_time=0.45311
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.65081 (QuantReg: 8.41336) QuantErr: 8.41336 batch_time=0.44454
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.71132 (QuantReg: 8.15969) QuantErr: 8.15969 batch_time=0.46241
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.41903 (QuantReg: 8.31458) QuantErr: 8.31458 batch_time=0.47068
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.82873 (QuantReg: 8.56642) QuantErr: 8.56642 batch_time=0.45762
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.05049 (QuantReg: 8.41653) QuantErr: 8.41653 batch_time=0.44010
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.15413 (QuantReg: 8.17058) QuantErr: 8.17058 batch_time=0.43976
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.49586 (QuantReg: 8.33796) QuantErr: 8.33796 batch_time=0.43783
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.15268 (QuantReg: 8.27500) QuantErr: 8.27500 batch_time=0.44000
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.86774 (QuantReg: 8.38865) QuantErr: 8.38865 batch_time=0.45469
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.92058 (QuantReg: 8.27889) QuantErr: 8.27889 batch_time=0.64062
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.93789 (QuantReg: 8.08569) QuantErr: 8.08569 batch_time=0.45687
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.50709 (QuantReg: 8.54406) QuantErr: 8.54406 batch_time=0.43759
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.03055 (QuantReg: 8.36063) QuantErr: 8.36063 batch_time=0.54991
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.83249 (QuantReg: 8.57354) QuantErr: 8.57354 batch_time=0.44768
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.82569 (QuantReg: 8.52214) QuantErr: 8.52214 batch_time=0.43818
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.63949 (QuantReg: 8.18101) QuantErr: 8.18101 batch_time=0.87239
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.84246 (QuantReg: 8.11548) QuantErr: 8.11548 batch_time=0.45156
Train Epoch: 10 codebook_update_time=0.90487
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch10.pth ...
Done in 4.716s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch10.pth ...
Done in 9.691s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 1.7701992344856263
quant_reg : 8.275035278320313
quant_err : 8.275035278320313
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 16.2
MSRVTT_miech_test/t2v_metrics/R5: 44.0
MSRVTT_miech_test/t2v_metrics/R10: 58.1
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.076
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.597756768151406
MSRVTT_miech_test/v2t_metrics/R1: 17.3
MSRVTT_miech_test/v2t_metrics/R5: 47.0
MSRVTT_miech_test/v2t_metrics/R10: 60.2
MSRVTT_miech_test/v2t_metrics/R50: 86.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.2505
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.58026248244376
mnt_best : 34.597756768151406
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.81212 (QuantReg: 8.23765) QuantErr: 8.23765 batch_time=30.73233
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.54099 (QuantReg: 8.48286) QuantErr: 8.48286 batch_time=0.46801
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.95155 (QuantReg: 8.37155) QuantErr: 8.37155 batch_time=0.46503
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.11028 (QuantReg: 7.94072) QuantErr: 7.94072 batch_time=0.46422
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.69013 (QuantReg: 8.44175) QuantErr: 8.44175 batch_time=0.52810
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.76163 (QuantReg: 8.03815) QuantErr: 8.03815 batch_time=0.47208
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.69912 (QuantReg: 8.32416) QuantErr: 8.32416 batch_time=0.47515
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.82941 (QuantReg: 8.23922) QuantErr: 8.23922 batch_time=0.49826
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.84313 (QuantReg: 8.40905) QuantErr: 8.40905 batch_time=0.47370
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.57045 (QuantReg: 8.13689) QuantErr: 8.13689 batch_time=0.47255
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.89806 (QuantReg: 8.31311) QuantErr: 8.31311 batch_time=0.49628
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.63334 (QuantReg: 8.36789) QuantErr: 8.36789 batch_time=0.46073
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.94087 (QuantReg: 8.30246) QuantErr: 8.30246 batch_time=0.45893
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.86465 (QuantReg: 8.28023) QuantErr: 8.28023 batch_time=0.45396
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.77379 (QuantReg: 8.13114) QuantErr: 8.13114 batch_time=0.47271
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.73362 (QuantReg: 8.20012) QuantErr: 8.20012 batch_time=0.46645
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.60775 (QuantReg: 8.26979) QuantErr: 8.26979 batch_time=0.46564
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.50869 (QuantReg: 8.19429) QuantErr: 8.19429 batch_time=0.46441
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.87307 (QuantReg: 8.18469) QuantErr: 8.18469 batch_time=4.54086
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.35191 (QuantReg: 8.45116) QuantErr: 8.45116 batch_time=0.46106
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.65231 (QuantReg: 8.22374) QuantErr: 8.22374 batch_time=1.61170
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.88144 (QuantReg: 8.43589) QuantErr: 8.43589 batch_time=0.45329
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.68080 (QuantReg: 8.37250) QuantErr: 8.37250 batch_time=0.46488
Train Epoch: 11 codebook_update_time=0.98033
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch11.pth ...
Done in 4.665s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch11.pth ...
Done in 9.513s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 1.7090234241485596
quant_reg : 8.322199701309204
quant_err : 8.322199701309204
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 16.6
MSRVTT_miech_test/t2v_metrics/R5: 44.3
MSRVTT_miech_test/t2v_metrics/R10: 57.8
MSRVTT_miech_test/t2v_metrics/R50: 86.4
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.799499999999995
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.89901886388832
MSRVTT_miech_test/v2t_metrics/R1: 16.3
MSRVTT_miech_test/v2t_metrics/R5: 45.4
MSRVTT_miech_test/v2t_metrics/R10: 58.2
MSRVTT_miech_test/v2t_metrics/R50: 87.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.5
MSRVTT_miech_test/v2t_metrics/MeanR: 30.9085
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.05275418713946
mnt_best : 34.89901886388832
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.79438 (QuantReg: 8.14120) QuantErr: 8.14120 batch_time=29.74678
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.48905 (QuantReg: 8.30429) QuantErr: 8.30429 batch_time=0.46161
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.66927 (QuantReg: 8.34702) QuantErr: 8.34702 batch_time=0.56841
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.58074 (QuantReg: 8.32265) QuantErr: 8.32265 batch_time=0.46145
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.47919 (QuantReg: 8.25530) QuantErr: 8.25530 batch_time=0.50403
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.75373 (QuantReg: 8.41229) QuantErr: 8.41229 batch_time=0.46107
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.79901 (QuantReg: 8.12842) QuantErr: 8.12842 batch_time=0.69835
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.37908 (QuantReg: 8.41643) QuantErr: 8.41643 batch_time=0.87375
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.60382 (QuantReg: 7.97724) QuantErr: 7.97724 batch_time=0.42871
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.49267 (QuantReg: 8.40093) QuantErr: 8.40093 batch_time=0.47317
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.59124 (QuantReg: 8.50984) QuantErr: 8.50984 batch_time=0.48224
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.76351 (QuantReg: 8.29811) QuantErr: 8.29811 batch_time=0.49645
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.53360 (QuantReg: 8.18884) QuantErr: 8.18884 batch_time=0.68653
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.62567 (QuantReg: 8.37005) QuantErr: 8.37005 batch_time=1.97676
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.61329 (QuantReg: 8.40324) QuantErr: 8.40324 batch_time=0.49879
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.62813 (QuantReg: 8.15754) QuantErr: 8.15754 batch_time=0.45756
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.83481 (QuantReg: 8.40501) QuantErr: 8.40501 batch_time=0.45947
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.79884 (QuantReg: 8.42262) QuantErr: 8.42262 batch_time=0.43916
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.48531 (QuantReg: 8.60685) QuantErr: 8.60685 batch_time=0.43667
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.80507 (QuantReg: 8.44205) QuantErr: 8.44205 batch_time=0.43331
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.72942 (QuantReg: 8.14377) QuantErr: 8.14377 batch_time=0.44367
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.84010 (QuantReg: 8.45414) QuantErr: 8.45414 batch_time=0.48758
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.63774 (QuantReg: 8.50500) QuantErr: 8.50500 batch_time=0.67718
Train Epoch: 12 codebook_update_time=0.92207
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch12.pth ...
Done in 4.459s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 1.6331921138763428
quant_reg : 8.360306173324584
quant_err : 8.360306173324584
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 16.2
MSRVTT_miech_test/t2v_metrics/R5: 43.7
MSRVTT_miech_test/t2v_metrics/R10: 59.7
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.8895
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.8329503606824
MSRVTT_miech_test/v2t_metrics/R1: 16.3
MSRVTT_miech_test/v2t_metrics/R5: 46.3
MSRVTT_miech_test/v2t_metrics/R10: 60.5
MSRVTT_miech_test/v2t_metrics/R50: 87.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.1075
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.74165485687373
mnt_best : 34.89901886388832
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.53480 (QuantReg: 8.36827) QuantErr: 8.36827 batch_time=29.18323
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.65823 (QuantReg: 8.28508) QuantErr: 8.28508 batch_time=0.45312
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.48214 (QuantReg: 8.45139) QuantErr: 8.45139 batch_time=0.46166
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.94886 (QuantReg: 8.51437) QuantErr: 8.51437 batch_time=0.44528
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.69527 (QuantReg: 8.55994) QuantErr: 8.55994 batch_time=0.47837
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.80490 (QuantReg: 8.26164) QuantErr: 8.26164 batch_time=0.48560
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.47248 (QuantReg: 8.41196) QuantErr: 8.41196 batch_time=0.45687
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.61432 (QuantReg: 8.48981) QuantErr: 8.48981 batch_time=0.43652
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.27921 (QuantReg: 8.40589) QuantErr: 8.40589 batch_time=0.44263
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.73749 (QuantReg: 8.13724) QuantErr: 8.13724 batch_time=0.44975
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.62521 (QuantReg: 8.31678) QuantErr: 8.31678 batch_time=0.44924
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.51818 (QuantReg: 8.58510) QuantErr: 8.58510 batch_time=0.43823
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.82053 (QuantReg: 8.08733) QuantErr: 8.08733 batch_time=0.44223
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.88195 (QuantReg: 8.49431) QuantErr: 8.49431 batch_time=2.16521
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.54498 (QuantReg: 8.61453) QuantErr: 8.61453 batch_time=0.47681
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.66531 (QuantReg: 8.27076) QuantErr: 8.27076 batch_time=0.46544
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.50674 (QuantReg: 8.70856) QuantErr: 8.70856 batch_time=0.51504
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.64428 (QuantReg: 8.34895) QuantErr: 8.34895 batch_time=0.46236
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.20079 (QuantReg: 8.45135) QuantErr: 8.45135 batch_time=0.44836
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.41382 (QuantReg: 8.74492) QuantErr: 8.74492 batch_time=0.47342
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.27038 (QuantReg: 8.49209) QuantErr: 8.49209 batch_time=0.46548
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.30575 (QuantReg: 8.23767) QuantErr: 8.23767 batch_time=0.75182
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.98289 (QuantReg: 8.34495) QuantErr: 8.34495 batch_time=0.45205
Train Epoch: 13 codebook_update_time=0.89178
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch13.pth ...
Done in 5.549s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch13.pth ...
Done in 10.442s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.575174551486969
quant_reg : 8.412688789367676
quant_err : 8.412688789367676
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 16.8
MSRVTT_miech_test/t2v_metrics/R5: 45.2
MSRVTT_miech_test/t2v_metrics/R10: 58.2
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.272
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.35549323887923
MSRVTT_miech_test/v2t_metrics/R1: 16.5
MSRVTT_miech_test/v2t_metrics/R5: 47.0
MSRVTT_miech_test/v2t_metrics/R10: 61.1
MSRVTT_miech_test/v2t_metrics/R50: 87.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.8955
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.18603543459612
mnt_best : 35.35549323887923
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.88561 (QuantReg: 8.19696) QuantErr: 8.19696 batch_time=33.86220
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.54453 (QuantReg: 8.38608) QuantErr: 8.38608 batch_time=0.54696
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.34702 (QuantReg: 8.17761) QuantErr: 8.17761 batch_time=0.47006
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.47657 (QuantReg: 8.62699) QuantErr: 8.62699 batch_time=0.45878
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.62419 (QuantReg: 8.31022) QuantErr: 8.31022 batch_time=0.44129
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.37029 (QuantReg: 8.57721) QuantErr: 8.57721 batch_time=0.45419
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.52915 (QuantReg: 8.36988) QuantErr: 8.36988 batch_time=0.45009
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.00713 (QuantReg: 8.25360) QuantErr: 8.25360 batch_time=0.46856
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.65538 (QuantReg: 8.61986) QuantErr: 8.61986 batch_time=0.44191
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.53611 (QuantReg: 8.51544) QuantErr: 8.51544 batch_time=0.49923
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.55235 (QuantReg: 8.44535) QuantErr: 8.44535 batch_time=0.56191
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.46528 (QuantReg: 8.34027) QuantErr: 8.34027 batch_time=0.44937
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.57832 (QuantReg: 8.39624) QuantErr: 8.39624 batch_time=0.44863
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.47400 (QuantReg: 8.31409) QuantErr: 8.31409 batch_time=0.46517
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.37931 (QuantReg: 8.35325) QuantErr: 8.35325 batch_time=0.43907
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.30437 (QuantReg: 8.56397) QuantErr: 8.56397 batch_time=0.44879
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.61682 (QuantReg: 8.33031) QuantErr: 8.33031 batch_time=0.44856
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.51506 (QuantReg: 8.58838) QuantErr: 8.58838 batch_time=0.44919
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.12262 (QuantReg: 8.56287) QuantErr: 8.56287 batch_time=0.46494
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.87848 (QuantReg: 8.68010) QuantErr: 8.68010 batch_time=0.44773
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.20135 (QuantReg: 8.69871) QuantErr: 8.69871 batch_time=0.44089
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.49455 (QuantReg: 8.32490) QuantErr: 8.32490 batch_time=0.45841
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.40532 (QuantReg: 8.36509) QuantErr: 8.36509 batch_time=0.82570
Train Epoch: 14 codebook_update_time=0.92257
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch14.pth ...
Done in 5.754s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch14.pth ...
Done in 10.846s
removing stale ckpt [epoch 13] [took 0.05s]
epoch : 14
loss : 1.5191276977062225
quant_reg : 8.467192108154297
quant_err : 8.467192108154297
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 16.7
MSRVTT_miech_test/t2v_metrics/R5: 45.0
MSRVTT_miech_test/t2v_metrics/R10: 59.6
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.7845
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.513358723033086
MSRVTT_miech_test/v2t_metrics/R1: 17.0
MSRVTT_miech_test/v2t_metrics/R5: 46.6
MSRVTT_miech_test/v2t_metrics/R10: 61.5
MSRVTT_miech_test/v2t_metrics/R50: 87.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.75
MSRVTT_miech_test/v2t_metrics/MeanR: 28.6385
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.52329781748556
mnt_best : 35.513358723033086
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.61722 (QuantReg: 8.50315) QuantErr: 8.50315 batch_time=30.88110
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.64532 (QuantReg: 8.28748) QuantErr: 8.28748 batch_time=0.43489
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.64919 (QuantReg: 8.34612) QuantErr: 8.34612 batch_time=0.48582
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.97272 (QuantReg: 8.27425) QuantErr: 8.27425 batch_time=0.44958
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.65105 (QuantReg: 8.46880) QuantErr: 8.46880 batch_time=0.45994
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.12567 (QuantReg: 8.48753) QuantErr: 8.48753 batch_time=0.43956
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.60001 (QuantReg: 8.23183) QuantErr: 8.23183 batch_time=0.44551
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.16596 (QuantReg: 8.58945) QuantErr: 8.58945 batch_time=0.43346
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.87397 (QuantReg: 8.52634) QuantErr: 8.52634 batch_time=0.44385
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.41460 (QuantReg: 8.41420) QuantErr: 8.41420 batch_time=0.53758
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.30149 (QuantReg: 8.60937) QuantErr: 8.60937 batch_time=0.47588
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.41119 (QuantReg: 8.25063) QuantErr: 8.25063 batch_time=0.46073
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.82584 (QuantReg: 8.62771) QuantErr: 8.62771 batch_time=0.43746
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.45211 (QuantReg: 8.50200) QuantErr: 8.50200 batch_time=0.44201
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.51568 (QuantReg: 8.44222) QuantErr: 8.44222 batch_time=0.46007
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.39779 (QuantReg: 8.40180) QuantErr: 8.40180 batch_time=0.52742
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.31891 (QuantReg: 8.51406) QuantErr: 8.51406 batch_time=0.48962
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.76020 (QuantReg: 8.62988) QuantErr: 8.62988 batch_time=0.73626
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.17557 (QuantReg: 8.43836) QuantErr: 8.43836 batch_time=0.43663
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.22872 (QuantReg: 8.69663) QuantErr: 8.69663 batch_time=0.43581
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.36375 (QuantReg: 8.77693) QuantErr: 8.77693 batch_time=0.43858
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.58837 (QuantReg: 8.67899) QuantErr: 8.67899 batch_time=0.65720
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.51780 (QuantReg: 8.84644) QuantErr: 8.84644 batch_time=0.46551
Train Epoch: 15 codebook_update_time=0.92766
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch15.pth ...
Done in 12.642s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch15.pth ...
Done in 29.022s
removing stale ckpt [epoch 14] [took 0.03s]
epoch : 15
loss : 1.4927544975280762
quant_reg : 8.49676725769043
quant_err : 8.49676725769043
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 17.1
MSRVTT_miech_test/t2v_metrics/R5: 46.0
MSRVTT_miech_test/t2v_metrics/R10: 59.7
MSRVTT_miech_test/t2v_metrics/R50: 87.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.5
MSRVTT_miech_test/t2v_metrics/MeanR: 32.381
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.07802521299651
MSRVTT_miech_test/v2t_metrics/R1: 16.4
MSRVTT_miech_test/v2t_metrics/R5: 46.6
MSRVTT_miech_test/v2t_metrics/R10: 60.8
MSRVTT_miech_test/v2t_metrics/R50: 88.2
MSRVTT_miech_test/v2t_metrics/MedR: 6.5
MSRVTT_miech_test/v2t_metrics/MeanR: 29.171
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.95101155654987
mnt_best : 36.07802521299651
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.43100 (QuantReg: 8.23243) QuantErr: 8.23243 batch_time=28.12268
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.66578 (QuantReg: 8.21801) QuantErr: 8.21801 batch_time=0.45588
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.11811 (QuantReg: 8.31074) QuantErr: 8.31074 batch_time=0.45816
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.77570 (QuantReg: 8.26270) QuantErr: 8.26270 batch_time=0.46968
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.45710 (QuantReg: 8.46986) QuantErr: 8.46986 batch_time=0.46664
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.70741 (QuantReg: 8.60902) QuantErr: 8.60902 batch_time=0.44007
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.69902 (QuantReg: 8.48994) QuantErr: 8.48994 batch_time=0.44538
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.38886 (QuantReg: 8.83915) QuantErr: 8.83915 batch_time=0.50030
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.53232 (QuantReg: 8.25447) QuantErr: 8.25447 batch_time=0.44207
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.15620 (QuantReg: 8.52502) QuantErr: 8.52502 batch_time=0.47378
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.30699 (QuantReg: 8.65395) QuantErr: 8.65395 batch_time=0.45634
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.39196 (QuantReg: 8.49999) QuantErr: 8.49999 batch_time=0.45132
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.48489 (QuantReg: 8.43086) QuantErr: 8.43086 batch_time=0.43890
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.57074 (QuantReg: 8.48651) QuantErr: 8.48651 batch_time=0.45248
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.59348 (QuantReg: 8.64312) QuantErr: 8.64312 batch_time=0.46513
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.45245 (QuantReg: 8.57404) QuantErr: 8.57404 batch_time=0.44324
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.46248 (QuantReg: 8.78883) QuantErr: 8.78883 batch_time=0.45027
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.54541 (QuantReg: 8.32564) QuantErr: 8.32564 batch_time=0.76030
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.42320 (QuantReg: 8.47791) QuantErr: 8.47791 batch_time=0.46696
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.39621 (QuantReg: 8.61979) QuantErr: 8.61979 batch_time=0.50672
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.45475 (QuantReg: 8.54365) QuantErr: 8.54365 batch_time=0.50707
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.29720 (QuantReg: 8.65997) QuantErr: 8.65997 batch_time=0.47604
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.55329 (QuantReg: 8.43362) QuantErr: 8.43362 batch_time=0.46504
Train Epoch: 16 codebook_update_time=0.88559
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch16.pth ...
Done in 4.974s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch16.pth ...
Done in 9.875s
removing stale ckpt [epoch 15] [took 0.10s]
epoch : 16
loss : 1.4294831473827363
quant_reg : 8.552006393432617
quant_err : 8.552006393432617
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 18.4
MSRVTT_miech_test/t2v_metrics/R5: 47.3
MSRVTT_miech_test/t2v_metrics/R10: 60.1
MSRVTT_miech_test/t2v_metrics/R50: 88.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.75
MSRVTT_miech_test/t2v_metrics/MeanR: 31.0465
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.39823835678397
MSRVTT_miech_test/v2t_metrics/R1: 17.5
MSRVTT_miech_test/v2t_metrics/R5: 45.9
MSRVTT_miech_test/v2t_metrics/R10: 61.8
MSRVTT_miech_test/v2t_metrics/R50: 88.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.5
MSRVTT_miech_test/v2t_metrics/MeanR: 28.8325
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.7518949460207
mnt_best : 37.39823835678397
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.34003 (QuantReg: 8.33321) QuantErr: 8.33321 batch_time=29.73514
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.25091 (QuantReg: 8.57446) QuantErr: 8.57446 batch_time=0.63523
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.43010 (QuantReg: 8.59176) QuantErr: 8.59176 batch_time=0.45329
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.56343 (QuantReg: 8.35932) QuantErr: 8.35932 batch_time=0.51052
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.52709 (QuantReg: 8.67025) QuantErr: 8.67025 batch_time=0.44414
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.42532 (QuantReg: 8.65399) QuantErr: 8.65399 batch_time=0.45094
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.77570 (QuantReg: 8.29630) QuantErr: 8.29630 batch_time=0.57860
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.39422 (QuantReg: 8.70441) QuantErr: 8.70441 batch_time=0.45779
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.43334 (QuantReg: 8.57443) QuantErr: 8.57443 batch_time=0.45309
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.08982 (QuantReg: 8.75864) QuantErr: 8.75864 batch_time=0.52937
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.19991 (QuantReg: 8.78904) QuantErr: 8.78904 batch_time=0.45101
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.24473 (QuantReg: 8.56537) QuantErr: 8.56537 batch_time=0.45860
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.67276 (QuantReg: 8.60498) QuantErr: 8.60498 batch_time=1.01814
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.49534 (QuantReg: 8.66392) QuantErr: 8.66392 batch_time=2.56043
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.66333 (QuantReg: 8.44491) QuantErr: 8.44491 batch_time=0.45043
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.55261 (QuantReg: 8.43073) QuantErr: 8.43073 batch_time=0.46710
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.16521 (QuantReg: 8.46203) QuantErr: 8.46203 batch_time=0.56606
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.30691 (QuantReg: 8.44205) QuantErr: 8.44205 batch_time=0.44709
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.59507 (QuantReg: 8.69249) QuantErr: 8.69249 batch_time=0.46290
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.52065 (QuantReg: 8.60003) QuantErr: 8.60003 batch_time=5.35205
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.12761 (QuantReg: 8.61184) QuantErr: 8.61184 batch_time=0.48645
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.37779 (QuantReg: 8.63683) QuantErr: 8.63683 batch_time=0.46071
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.41419 (QuantReg: 8.58260) QuantErr: 8.58260 batch_time=0.45469
Train Epoch: 17 codebook_update_time=0.92615
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch17.pth ...
Done in 5.253s
removing stale ckpt [epoch 16] [took 0.02s]
epoch : 17
loss : 1.4025504398345947
quant_reg : 8.563298263549804
quant_err : 8.563298263549804
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 17.8
MSRVTT_miech_test/t2v_metrics/R5: 46.5
MSRVTT_miech_test/t2v_metrics/R10: 61.2
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.623000000000005
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.000545402235545
MSRVTT_miech_test/v2t_metrics/R1: 17.7
MSRVTT_miech_test/v2t_metrics/R5: 47.7
MSRVTT_miech_test/v2t_metrics/R10: 62.7
MSRVTT_miech_test/v2t_metrics/R50: 87.3
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.661
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.54796422539303
mnt_best : 37.39823835678397
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.39166 (QuantReg: 8.40263) QuantErr: 8.40263 batch_time=31.61158
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.51166 (QuantReg: 8.44311) QuantErr: 8.44311 batch_time=0.47130
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.59611 (QuantReg: 8.42038) QuantErr: 8.42038 batch_time=1.77290
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.88758 (QuantReg: 8.54896) QuantErr: 8.54896 batch_time=0.47176
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.35111 (QuantReg: 8.74917) QuantErr: 8.74917 batch_time=0.67979
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.38198 (QuantReg: 8.57288) QuantErr: 8.57288 batch_time=0.45568
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.15688 (QuantReg: 8.63284) QuantErr: 8.63284 batch_time=0.47361
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.45436 (QuantReg: 8.78420) QuantErr: 8.78420 batch_time=0.50017
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.42309 (QuantReg: 8.81921) QuantErr: 8.81921 batch_time=0.46882
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.54405 (QuantReg: 8.55704) QuantErr: 8.55704 batch_time=0.51374
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.09408 (QuantReg: 8.67506) QuantErr: 8.67506 batch_time=0.51226
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.28107 (QuantReg: 8.92895) QuantErr: 8.92895 batch_time=0.48055
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.57226 (QuantReg: 8.63212) QuantErr: 8.63212 batch_time=0.46139
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.30081 (QuantReg: 8.49165) QuantErr: 8.49165 batch_time=0.44031
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.28798 (QuantReg: 8.55189) QuantErr: 8.55189 batch_time=0.43947
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.43095 (QuantReg: 8.65345) QuantErr: 8.65345 batch_time=0.46976
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.73368 (QuantReg: 8.81808) QuantErr: 8.81808 batch_time=0.46159
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.33839 (QuantReg: 8.56719) QuantErr: 8.56719 batch_time=0.45032
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.47202 (QuantReg: 8.60623) QuantErr: 8.60623 batch_time=0.47050
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.35491 (QuantReg: 8.70683) QuantErr: 8.70683 batch_time=0.45483
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.41106 (QuantReg: 8.42507) QuantErr: 8.42507 batch_time=0.46643
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.56519 (QuantReg: 8.69354) QuantErr: 8.69354 batch_time=0.46361
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.36705 (QuantReg: 8.60838) QuantErr: 8.60838 batch_time=0.46488
Train Epoch: 18 codebook_update_time=1.12885
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch18.pth ...
Done in 4.848s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 1.3756271464824676
quant_reg : 8.632653289794922
quant_err : 8.632653289794922
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 18.2
MSRVTT_miech_test/t2v_metrics/R5: 46.0
MSRVTT_miech_test/t2v_metrics/R10: 59.8
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.284
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.85616423841203
MSRVTT_miech_test/v2t_metrics/R1: 18.3
MSRVTT_miech_test/v2t_metrics/R5: 47.1
MSRVTT_miech_test/v2t_metrics/R10: 61.0
MSRVTT_miech_test/v2t_metrics/R50: 87.5
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.8585
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.462832507570624
mnt_best : 37.39823835678397
not_improved_count: 2
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.52283 (QuantReg: 8.46316) QuantErr: 8.46316 batch_time=29.29488
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.16988 (QuantReg: 8.71683) QuantErr: 8.71683 batch_time=0.45329
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.39008 (QuantReg: 8.22570) QuantErr: 8.22570 batch_time=0.48685
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.37794 (QuantReg: 8.54398) QuantErr: 8.54398 batch_time=0.45558
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.43870 (QuantReg: 8.60566) QuantErr: 8.60566 batch_time=0.46580
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.56153 (QuantReg: 8.71427) QuantErr: 8.71427 batch_time=0.49147
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.47984 (QuantReg: 8.50263) QuantErr: 8.50263 batch_time=7.17346
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.20107 (QuantReg: 8.76186) QuantErr: 8.76186 batch_time=0.46387
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.59701 (QuantReg: 8.67882) QuantErr: 8.67882 batch_time=0.45599
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.14240 (QuantReg: 8.31208) QuantErr: 8.31208 batch_time=0.50432
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.23608 (QuantReg: 8.68823) QuantErr: 8.68823 batch_time=0.46448
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.32405 (QuantReg: 8.56341) QuantErr: 8.56341 batch_time=0.44576
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.28708 (QuantReg: 8.61269) QuantErr: 8.61269 batch_time=0.45904
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.28174 (QuantReg: 8.62437) QuantErr: 8.62437 batch_time=0.47059
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.60302 (QuantReg: 8.62339) QuantErr: 8.62339 batch_time=0.46536
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.07626 (QuantReg: 8.86333) QuantErr: 8.86333 batch_time=0.45541
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.20403 (QuantReg: 8.69498) QuantErr: 8.69498 batch_time=0.46147
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.26938 (QuantReg: 8.79412) QuantErr: 8.79412 batch_time=0.47033
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.55986 (QuantReg: 8.67303) QuantErr: 8.67303 batch_time=0.48900
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.44145 (QuantReg: 8.48702) QuantErr: 8.48702 batch_time=0.47735
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.37416 (QuantReg: 8.61062) QuantErr: 8.61062 batch_time=0.45177
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.18676 (QuantReg: 8.55971) QuantErr: 8.55971 batch_time=0.47099
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.50634 (QuantReg: 8.34750) QuantErr: 8.34750 batch_time=0.46954
Train Epoch: 19 codebook_update_time=1.12256
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M16/checkpoint-epoch19.pth ...
Done in 6.939s
removing stale ckpt [epoch 18] [took 0.02s]
epoch : 19
loss : 1.3440263841152191
quant_reg : 8.668018970489502
quant_err : 8.668018970489502
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 17.5
MSRVTT_miech_test/t2v_metrics/R5: 46.9
MSRVTT_miech_test/t2v_metrics/R10: 61.4
MSRVTT_miech_test/t2v_metrics/R50: 87.5
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.094