-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_L1.txt
2595 lines (2595 loc) · 192 KB
/
HCQ_MSRVTT_1kB_L1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 773.848790884018 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 114.63792705535889 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 82.19015836715698 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch0.pth ...
Done in 1.677s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch0.pth ...
Done in 3.415s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.1
MSRVTT_miech_test/t2v_metrics/R5: 0.5
MSRVTT_miech_test/t2v_metrics/R10: 0.6
MSRVTT_miech_test/t2v_metrics/R50: 5.0
MSRVTT_miech_test/t2v_metrics/MedR: 502.5
MSRVTT_miech_test/t2v_metrics/MeanR: 500.393
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.31072325059538586
MSRVTT_miech_test/v2t_metrics/R1: 0.0
MSRVTT_miech_test/v2t_metrics/R5: 0.5
MSRVTT_miech_test/v2t_metrics/R10: 0.8
MSRVTT_miech_test/v2t_metrics/R50: 4.5
MSRVTT_miech_test/v2t_metrics/MedR: 515.0
MSRVTT_miech_test/v2t_metrics/MeanR: 506.1835
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.31072325059538586
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 39.47011 (QuantReg: 22.51172) QuantErr: 22.51172 batch_time=33.75569
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 38.14620 (QuantReg: 22.37913) QuantErr: 22.37913 batch_time=2.45325
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 34.59476 (QuantReg: 22.44696) QuantErr: 22.44696 batch_time=0.39944
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 29.42259 (QuantReg: 22.44535) QuantErr: 22.44535 batch_time=0.75516
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 26.42833 (QuantReg: 22.54477) QuantErr: 22.54477 batch_time=0.40218
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 24.30130 (QuantReg: 22.43163) QuantErr: 22.43163 batch_time=0.49966
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 24.22937 (QuantReg: 22.62312) QuantErr: 22.62312 batch_time=0.39624
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 22.77871 (QuantReg: 22.60013) QuantErr: 22.60013 batch_time=0.41617
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 23.24564 (QuantReg: 22.50975) QuantErr: 22.50975 batch_time=0.40516
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 21.39157 (QuantReg: 22.53990) QuantErr: 22.53990 batch_time=0.39726
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 21.54585 (QuantReg: 22.55175) QuantErr: 22.55175 batch_time=0.42267
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 21.27766 (QuantReg: 22.54229) QuantErr: 22.54229 batch_time=0.40827
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 20.01805 (QuantReg: 22.60019) QuantErr: 22.60019 batch_time=0.54228
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 19.33574 (QuantReg: 22.60562) QuantErr: 22.60562 batch_time=0.39694
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 19.23982 (QuantReg: 22.56432) QuantErr: 22.56432 batch_time=0.42223
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 18.73969 (QuantReg: 22.66190) QuantErr: 22.66190 batch_time=0.41926
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 19.65483 (QuantReg: 22.56136) QuantErr: 22.56136 batch_time=0.45943
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 18.72408 (QuantReg: 22.58389) QuantErr: 22.58389 batch_time=0.41212
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 19.90007 (QuantReg: 22.62226) QuantErr: 22.62226 batch_time=1.58255
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 21.31228 (QuantReg: 22.54951) QuantErr: 22.54951 batch_time=0.39818
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 19.83918 (QuantReg: 22.60647) QuantErr: 22.60647 batch_time=0.41766
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 18.42447 (QuantReg: 22.68834) QuantErr: 22.68834 batch_time=0.39881
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 15.64518 (QuantReg: 22.58954) QuantErr: 22.58954 batch_time=0.40306
Train Epoch: 1 codebook_update_time=0.44923
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch1.pth ...
Done in 4.563s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch1.pth ...
Done in 9.035s
epoch : 1
loss : 23.004264335632325
quant_reg : 22.558331146240235
quant_err : 22.558331146240235
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 9.4
MSRVTT_miech_test/t2v_metrics/R5: 29.9
MSRVTT_miech_test/t2v_metrics/R10: 44.5
MSRVTT_miech_test/t2v_metrics/R50: 78.6
MSRVTT_miech_test/t2v_metrics/MedR: 13.0
MSRVTT_miech_test/t2v_metrics/MeanR: 46.087
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.212380678836055
MSRVTT_miech_test/v2t_metrics/R1: 10.0
MSRVTT_miech_test/v2t_metrics/R5: 32.5
MSRVTT_miech_test/v2t_metrics/R10: 45.3
MSRVTT_miech_test/v2t_metrics/R50: 78.3
MSRVTT_miech_test/v2t_metrics/MedR: 13.0
MSRVTT_miech_test/v2t_metrics/MeanR: 47.47
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.509090060254348
mnt_best : 23.212380678836055
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 17.43936 (QuantReg: 11.54594) QuantErr: 11.54594 batch_time=30.62970
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 16.43305 (QuantReg: 11.35877) QuantErr: 11.35877 batch_time=0.40364
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 16.30775 (QuantReg: 12.06793) QuantErr: 12.06793 batch_time=0.39760
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 17.64536 (QuantReg: 11.87438) QuantErr: 11.87438 batch_time=0.40064
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 16.72039 (QuantReg: 11.90272) QuantErr: 11.90272 batch_time=1.03396
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 15.50074 (QuantReg: 12.96538) QuantErr: 12.96538 batch_time=0.54217
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 17.78900 (QuantReg: 12.53465) QuantErr: 12.53465 batch_time=0.40040
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 16.14030 (QuantReg: 12.85550) QuantErr: 12.85550 batch_time=0.40716
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 14.51978 (QuantReg: 12.84403) QuantErr: 12.84403 batch_time=0.39572
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 13.50896 (QuantReg: 13.40492) QuantErr: 13.40492 batch_time=0.39626
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 16.10179 (QuantReg: 13.28533) QuantErr: 13.28533 batch_time=0.52644
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 17.06365 (QuantReg: 13.02889) QuantErr: 13.02889 batch_time=0.43013
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 15.71891 (QuantReg: 13.53376) QuantErr: 13.53376 batch_time=0.40672
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 15.60490 (QuantReg: 13.59201) QuantErr: 13.59201 batch_time=0.40328
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 14.47570 (QuantReg: 13.62023) QuantErr: 13.62023 batch_time=0.98496
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 14.13923 (QuantReg: 14.02548) QuantErr: 14.02548 batch_time=0.40706
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 13.28438 (QuantReg: 14.53993) QuantErr: 14.53993 batch_time=0.75688
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 14.10795 (QuantReg: 14.12801) QuantErr: 14.12801 batch_time=0.43504
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 14.33472 (QuantReg: 14.56124) QuantErr: 14.56124 batch_time=0.40865
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 13.06679 (QuantReg: 14.18870) QuantErr: 14.18870 batch_time=0.41185
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 13.48801 (QuantReg: 14.06668) QuantErr: 14.06668 batch_time=0.45397
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 14.61469 (QuantReg: 14.73817) QuantErr: 14.73817 batch_time=0.41880
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 14.10312 (QuantReg: 14.34653) QuantErr: 14.34653 batch_time=0.40240
Train Epoch: 2 codebook_update_time=0.41810
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch2.pth ...
Done in 4.204s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch2.pth ...
Done in 8.384s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 15.42320152282715
quant_reg : 13.34117865371704
quant_err : 13.34117865371704
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 11.4
MSRVTT_miech_test/t2v_metrics/R5: 37.9
MSRVTT_miech_test/t2v_metrics/R10: 50.2
MSRVTT_miech_test/t2v_metrics/R50: 83.2
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 40.157
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.88790729908038
MSRVTT_miech_test/v2t_metrics/R1: 11.9
MSRVTT_miech_test/v2t_metrics/R5: 37.3
MSRVTT_miech_test/v2t_metrics/R10: 49.5
MSRVTT_miech_test/v2t_metrics/R50: 82.5
MSRVTT_miech_test/v2t_metrics/MedR: 11.0
MSRVTT_miech_test/v2t_metrics/MeanR: 39.3855
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.00831598229496
mnt_best : 27.88790729908038
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 14.19038 (QuantReg: 12.20504) QuantErr: 12.20504 batch_time=33.33547
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 13.89458 (QuantReg: 11.98402) QuantErr: 11.98402 batch_time=0.42421
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 13.31095 (QuantReg: 11.77733) QuantErr: 11.77733 batch_time=0.41113
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 13.86524 (QuantReg: 12.00462) QuantErr: 12.00462 batch_time=0.43417
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 13.65553 (QuantReg: 12.11991) QuantErr: 12.11991 batch_time=0.39956
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 12.86022 (QuantReg: 12.21860) QuantErr: 12.21860 batch_time=0.42752
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 14.13010 (QuantReg: 12.31317) QuantErr: 12.31317 batch_time=0.39527
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 13.50948 (QuantReg: 12.55603) QuantErr: 12.55603 batch_time=0.40895
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 12.92792 (QuantReg: 12.73589) QuantErr: 12.73589 batch_time=0.40783
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 13.68627 (QuantReg: 12.38800) QuantErr: 12.38800 batch_time=0.46064
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 13.03564 (QuantReg: 12.78097) QuantErr: 12.78097 batch_time=0.43825
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 14.47116 (QuantReg: 12.52165) QuantErr: 12.52165 batch_time=0.42575
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 14.17427 (QuantReg: 13.00101) QuantErr: 13.00101 batch_time=4.01962
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 12.69377 (QuantReg: 12.82129) QuantErr: 12.82129 batch_time=0.40521
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 13.53201 (QuantReg: 12.59543) QuantErr: 12.59543 batch_time=0.44115
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 13.00321 (QuantReg: 13.12404) QuantErr: 13.12404 batch_time=0.40509
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 13.24077 (QuantReg: 12.91362) QuantErr: 12.91362 batch_time=0.79096
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 11.79090 (QuantReg: 12.74204) QuantErr: 12.74204 batch_time=0.74746
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 13.59607 (QuantReg: 12.99763) QuantErr: 12.99763 batch_time=0.40308
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 11.82134 (QuantReg: 13.23424) QuantErr: 13.23424 batch_time=0.43902
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 13.31285 (QuantReg: 13.47512) QuantErr: 13.47512 batch_time=0.42236
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 13.10858 (QuantReg: 13.13122) QuantErr: 13.13122 batch_time=0.39590
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 11.46240 (QuantReg: 13.45502) QuantErr: 13.45502 batch_time=0.40849
Train Epoch: 3 codebook_update_time=0.49717
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch3.pth ...
Done in 4.122s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch3.pth ...
Done in 8.302s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 13.254443386077881
quant_reg : 12.573423419952393
quant_err : 12.573423419952393
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 14.0
MSRVTT_miech_test/t2v_metrics/R5: 39.2
MSRVTT_miech_test/t2v_metrics/R10: 53.8
MSRVTT_miech_test/t2v_metrics/R50: 84.5
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.86
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.9076125538077
MSRVTT_miech_test/v2t_metrics/R1: 14.2
MSRVTT_miech_test/v2t_metrics/R5: 38.2
MSRVTT_miech_test/v2t_metrics/R10: 53.4
MSRVTT_miech_test/v2t_metrics/R50: 84.6
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.218
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.7112614169186
mnt_best : 30.9076125538077
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 12.24158 (QuantReg: 12.06451) QuantErr: 12.06451 batch_time=28.02142
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 12.11939 (QuantReg: 11.87605) QuantErr: 11.87605 batch_time=0.39851
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 12.99228 (QuantReg: 12.05025) QuantErr: 12.05025 batch_time=1.62917
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 11.41338 (QuantReg: 12.25241) QuantErr: 12.25241 batch_time=0.41044
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 13.25156 (QuantReg: 12.03387) QuantErr: 12.03387 batch_time=0.40462
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 11.48609 (QuantReg: 12.30199) QuantErr: 12.30199 batch_time=0.43774
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 11.78933 (QuantReg: 12.44425) QuantErr: 12.44425 batch_time=0.40516
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 11.42651 (QuantReg: 12.18880) QuantErr: 12.18880 batch_time=0.42958
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 13.68217 (QuantReg: 12.09868) QuantErr: 12.09868 batch_time=0.40778
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 11.01597 (QuantReg: 12.31607) QuantErr: 12.31607 batch_time=0.38822
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 13.63887 (QuantReg: 12.66977) QuantErr: 12.66977 batch_time=0.43736
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 12.96862 (QuantReg: 12.44632) QuantErr: 12.44632 batch_time=0.41690
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 10.93685 (QuantReg: 12.31167) QuantErr: 12.31167 batch_time=0.40162
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 11.13231 (QuantReg: 12.69315) QuantErr: 12.69315 batch_time=0.39165
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 11.16014 (QuantReg: 12.62325) QuantErr: 12.62325 batch_time=0.40241
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 11.02400 (QuantReg: 12.90488) QuantErr: 12.90488 batch_time=0.43105
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 11.46787 (QuantReg: 12.95362) QuantErr: 12.95362 batch_time=0.43219
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 11.73361 (QuantReg: 12.77979) QuantErr: 12.77979 batch_time=0.39963
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 11.29597 (QuantReg: 12.96758) QuantErr: 12.96758 batch_time=0.41225
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 11.74684 (QuantReg: 12.74053) QuantErr: 12.74053 batch_time=0.42025
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 11.85578 (QuantReg: 12.58318) QuantErr: 12.58318 batch_time=0.46168
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 11.12870 (QuantReg: 12.95496) QuantErr: 12.95496 batch_time=0.39746
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 10.39583 (QuantReg: 13.29091) QuantErr: 13.29091 batch_time=0.45543
Train Epoch: 4 codebook_update_time=0.46026
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch4.pth ...
Done in 4.417s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch4.pth ...
Done in 8.662s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 11.711074295043945
quant_reg : 12.44312604522705
quant_err : 12.44312604522705
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 14.9
MSRVTT_miech_test/t2v_metrics/R5: 41.9
MSRVTT_miech_test/t2v_metrics/R10: 58.1
MSRVTT_miech_test/t2v_metrics/R50: 86.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.29
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.102348601009346
MSRVTT_miech_test/v2t_metrics/R1: 17.3
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 56.4
MSRVTT_miech_test/v2t_metrics/R50: 86.5
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.2755
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.61290462764593
mnt_best : 33.102348601009346
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 11.31428 (QuantReg: 12.22632) QuantErr: 12.22632 batch_time=33.65436
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 11.30921 (QuantReg: 11.99150) QuantErr: 11.99150 batch_time=2.47494
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 11.47607 (QuantReg: 11.90784) QuantErr: 11.90784 batch_time=0.42878
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 9.96153 (QuantReg: 12.21704) QuantErr: 12.21704 batch_time=0.41027
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 11.76542 (QuantReg: 12.45435) QuantErr: 12.45435 batch_time=0.49899
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 8.50068 (QuantReg: 12.44417) QuantErr: 12.44417 batch_time=0.41130
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 11.32301 (QuantReg: 12.55168) QuantErr: 12.55168 batch_time=0.41268
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 11.34299 (QuantReg: 12.08590) QuantErr: 12.08590 batch_time=0.40442
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 12.94715 (QuantReg: 12.44090) QuantErr: 12.44090 batch_time=1.49855
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 9.43069 (QuantReg: 12.22727) QuantErr: 12.22727 batch_time=0.42119
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 10.88756 (QuantReg: 12.69641) QuantErr: 12.69641 batch_time=0.40259
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 9.96939 (QuantReg: 12.58472) QuantErr: 12.58472 batch_time=0.40651
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 8.92341 (QuantReg: 12.43947) QuantErr: 12.43947 batch_time=0.42962
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 11.28197 (QuantReg: 12.55110) QuantErr: 12.55110 batch_time=2.72656
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 11.74527 (QuantReg: 12.73380) QuantErr: 12.73380 batch_time=0.42500
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 9.38312 (QuantReg: 12.88463) QuantErr: 12.88463 batch_time=0.41169
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 8.84635 (QuantReg: 12.65850) QuantErr: 12.65850 batch_time=0.38964
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 10.42243 (QuantReg: 12.66226) QuantErr: 12.66226 batch_time=0.40487
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 9.70435 (QuantReg: 12.35521) QuantErr: 12.35521 batch_time=0.39453
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 9.48479 (QuantReg: 12.92469) QuantErr: 12.92469 batch_time=0.40654
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 10.01056 (QuantReg: 12.72803) QuantErr: 12.72803 batch_time=0.49091
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 8.07802 (QuantReg: 13.20098) QuantErr: 13.20098 batch_time=0.39853
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 10.78590 (QuantReg: 12.57439) QuantErr: 12.57439 batch_time=0.41838
Train Epoch: 5 codebook_update_time=0.42538
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch5.pth ...
Done in 4.293s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch5.pth ...
Done in 8.407s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 10.607116243362427
quant_reg : 12.531304187774658
quant_err : 12.531304187774658
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 17.2
MSRVTT_miech_test/t2v_metrics/R5: 43.9
MSRVTT_miech_test/t2v_metrics/R10: 58.5
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.64
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.34947306634721
MSRVTT_miech_test/v2t_metrics/R1: 16.2
MSRVTT_miech_test/v2t_metrics/R5: 44.0
MSRVTT_miech_test/v2t_metrics/R10: 58.3
MSRVTT_miech_test/v2t_metrics/R50: 87.5
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.0145
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.63741039227711
mnt_best : 35.34947306634721
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 10.59931 (QuantReg: 12.23463) QuantErr: 12.23463 batch_time=35.33791
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 8.81248 (QuantReg: 12.13199) QuantErr: 12.13199 batch_time=0.46157
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 11.39352 (QuantReg: 12.05853) QuantErr: 12.05853 batch_time=0.50318
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 9.84938 (QuantReg: 12.43899) QuantErr: 12.43899 batch_time=0.40720
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 9.98526 (QuantReg: 12.54375) QuantErr: 12.54375 batch_time=0.40278
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 8.05497 (QuantReg: 12.25947) QuantErr: 12.25947 batch_time=0.67809
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 10.21521 (QuantReg: 12.50941) QuantErr: 12.50941 batch_time=0.43362
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 9.66104 (QuantReg: 12.68368) QuantErr: 12.68368 batch_time=0.42791
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 7.98029 (QuantReg: 12.66123) QuantErr: 12.66123 batch_time=0.43657
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 9.08545 (QuantReg: 12.43545) QuantErr: 12.43545 batch_time=0.38932
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 11.23204 (QuantReg: 12.74666) QuantErr: 12.74666 batch_time=0.41783
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 11.40745 (QuantReg: 12.90985) QuantErr: 12.90985 batch_time=0.43450
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 8.76280 (QuantReg: 13.00489) QuantErr: 13.00489 batch_time=0.42954
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 8.93499 (QuantReg: 12.61261) QuantErr: 12.61261 batch_time=0.40729
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 10.11624 (QuantReg: 12.67547) QuantErr: 12.67547 batch_time=0.42780
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 8.66091 (QuantReg: 12.60831) QuantErr: 12.60831 batch_time=0.38816
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 10.16540 (QuantReg: 12.23903) QuantErr: 12.23903 batch_time=0.41187
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 9.05478 (QuantReg: 12.57277) QuantErr: 12.57277 batch_time=0.42552
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 8.25678 (QuantReg: 12.98023) QuantErr: 12.98023 batch_time=0.40891
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 10.55573 (QuantReg: 12.80965) QuantErr: 12.80965 batch_time=0.43346
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 8.45090 (QuantReg: 12.76354) QuantErr: 12.76354 batch_time=0.41143
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 8.26454 (QuantReg: 12.75662) QuantErr: 12.75662 batch_time=0.40572
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 9.49467 (QuantReg: 12.73362) QuantErr: 12.73362 batch_time=0.85014
Train Epoch: 6 codebook_update_time=0.43068
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch6.pth ...
Done in 4.264s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch6.pth ...
Done in 8.354s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 9.623465221405029
quant_reg : 12.569846393585205
quant_err : 12.569846393585205
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 45.5
MSRVTT_miech_test/t2v_metrics/R10: 59.0
MSRVTT_miech_test/t2v_metrics/R50: 87.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.7675
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.15141983229556
MSRVTT_miech_test/v2t_metrics/R1: 16.7
MSRVTT_miech_test/v2t_metrics/R5: 45.4
MSRVTT_miech_test/v2t_metrics/R10: 59.7
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.6475
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.63818284660821
mnt_best : 36.15141983229556
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 9.83223 (QuantReg: 12.24847) QuantErr: 12.24847 batch_time=33.52329
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 9.99234 (QuantReg: 12.59956) QuantErr: 12.59956 batch_time=0.70546
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 8.45759 (QuantReg: 12.57084) QuantErr: 12.57084 batch_time=0.40831
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 8.75420 (QuantReg: 12.58071) QuantErr: 12.58071 batch_time=0.40399
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 9.33173 (QuantReg: 12.51890) QuantErr: 12.51890 batch_time=0.40307
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 8.59698 (QuantReg: 12.57460) QuantErr: 12.57460 batch_time=0.41331
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 10.82995 (QuantReg: 12.47791) QuantErr: 12.47791 batch_time=0.99852
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 8.20480 (QuantReg: 12.61105) QuantErr: 12.61105 batch_time=0.78007
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 9.97898 (QuantReg: 12.44987) QuantErr: 12.44987 batch_time=0.45077
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 7.87685 (QuantReg: 12.55578) QuantErr: 12.55578 batch_time=0.43478
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 7.89972 (QuantReg: 12.54563) QuantErr: 12.54563 batch_time=0.43596
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 10.01037 (QuantReg: 12.62530) QuantErr: 12.62530 batch_time=0.39852
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 10.66777 (QuantReg: 12.92181) QuantErr: 12.92181 batch_time=0.40923
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 7.72540 (QuantReg: 12.63610) QuantErr: 12.63610 batch_time=0.40170
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 10.28205 (QuantReg: 12.85514) QuantErr: 12.85514 batch_time=0.45522
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 7.70463 (QuantReg: 12.80066) QuantErr: 12.80066 batch_time=0.42655
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 9.54371 (QuantReg: 12.73184) QuantErr: 12.73184 batch_time=0.69486
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 8.19999 (QuantReg: 12.79913) QuantErr: 12.79913 batch_time=0.40853
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 9.93723 (QuantReg: 12.62382) QuantErr: 12.62382 batch_time=0.40824
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 10.96578 (QuantReg: 12.53464) QuantErr: 12.53464 batch_time=0.60116
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 9.12285 (QuantReg: 12.89525) QuantErr: 12.89525 batch_time=0.41689
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 7.94098 (QuantReg: 12.66081) QuantErr: 12.66081 batch_time=0.39972
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 8.80719 (QuantReg: 12.62621) QuantErr: 12.62621 batch_time=0.40710
Train Epoch: 7 codebook_update_time=0.91262
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch7.pth ...
Done in 4.234s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 8.956707876205444
quant_reg : 12.660375679016113
quant_err : 12.660375679016113
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 16.3
MSRVTT_miech_test/t2v_metrics/R5: 45.8
MSRVTT_miech_test/t2v_metrics/R10: 60.2
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.048
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.55356800666467
MSRVTT_miech_test/v2t_metrics/R1: 16.6
MSRVTT_miech_test/v2t_metrics/R5: 44.0
MSRVTT_miech_test/v2t_metrics/R10: 57.7
MSRVTT_miech_test/v2t_metrics/R50: 87.3
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 31.319
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.79996917247456
mnt_best : 36.15141983229556
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 6.65203 (QuantReg: 12.46207) QuantErr: 12.46207 batch_time=28.12031
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 9.81330 (QuantReg: 12.20310) QuantErr: 12.20310 batch_time=0.41679
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 10.12443 (QuantReg: 12.55367) QuantErr: 12.55367 batch_time=0.41540
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 9.18053 (QuantReg: 12.40817) QuantErr: 12.40817 batch_time=0.40388
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 6.70315 (QuantReg: 12.31123) QuantErr: 12.31123 batch_time=0.41786
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 7.93377 (QuantReg: 12.27259) QuantErr: 12.27259 batch_time=0.39521
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 8.00126 (QuantReg: 12.86382) QuantErr: 12.86382 batch_time=0.83393
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 6.71762 (QuantReg: 12.74016) QuantErr: 12.74016 batch_time=0.51903
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 10.68524 (QuantReg: 12.78587) QuantErr: 12.78587 batch_time=0.41855
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 7.53831 (QuantReg: 13.17018) QuantErr: 13.17018 batch_time=0.41026
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 6.95960 (QuantReg: 12.98180) QuantErr: 12.98180 batch_time=0.40590
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 9.16403 (QuantReg: 12.69377) QuantErr: 12.69377 batch_time=0.48114
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 7.24525 (QuantReg: 12.62866) QuantErr: 12.62866 batch_time=0.40809
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 8.76202 (QuantReg: 12.85789) QuantErr: 12.85789 batch_time=0.41639
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 7.98667 (QuantReg: 12.89361) QuantErr: 12.89361 batch_time=0.75396
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 8.70905 (QuantReg: 12.65417) QuantErr: 12.65417 batch_time=0.41457
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 8.78420 (QuantReg: 12.78840) QuantErr: 12.78840 batch_time=0.41090
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 8.89784 (QuantReg: 12.54161) QuantErr: 12.54161 batch_time=0.40563
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 8.17531 (QuantReg: 12.65753) QuantErr: 12.65753 batch_time=0.39730
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 7.53056 (QuantReg: 13.06651) QuantErr: 13.06651 batch_time=0.38769
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 8.42067 (QuantReg: 12.56428) QuantErr: 12.56428 batch_time=0.38251
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 7.33879 (QuantReg: 12.76834) QuantErr: 12.76834 batch_time=0.42769
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 8.43994 (QuantReg: 12.85346) QuantErr: 12.85346 batch_time=0.43601
Train Epoch: 8 codebook_update_time=0.45313
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch8.pth ...
Done in 4.255s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch8.pth ...
Done in 8.424s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 8.422050859451295
quant_reg : 12.713641845703124
quant_err : 12.713641845703124
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 18.5
MSRVTT_miech_test/t2v_metrics/R5: 45.5
MSRVTT_miech_test/t2v_metrics/R10: 60.7
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.225
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.10712199470364
MSRVTT_miech_test/v2t_metrics/R1: 17.4
MSRVTT_miech_test/v2t_metrics/R5: 44.9
MSRVTT_miech_test/v2t_metrics/R10: 59.9
MSRVTT_miech_test/v2t_metrics/R50: 87.3
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.2915
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.03635062865264
mnt_best : 37.10712199470364
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 8.56584 (QuantReg: 12.48657) QuantErr: 12.48657 batch_time=38.46723
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 9.30808 (QuantReg: 12.53826) QuantErr: 12.53826 batch_time=0.40223
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 7.39184 (QuantReg: 12.51006) QuantErr: 12.51006 batch_time=0.41824
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 7.19528 (QuantReg: 12.45648) QuantErr: 12.45648 batch_time=0.41869
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 9.24959 (QuantReg: 12.48414) QuantErr: 12.48414 batch_time=0.40002
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 8.08729 (QuantReg: 12.59617) QuantErr: 12.59617 batch_time=0.39750
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 7.20712 (QuantReg: 12.52551) QuantErr: 12.52551 batch_time=0.41238
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 8.66221 (QuantReg: 12.37391) QuantErr: 12.37391 batch_time=0.41642
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 7.12511 (QuantReg: 12.70565) QuantErr: 12.70565 batch_time=0.41190
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 8.45540 (QuantReg: 12.59986) QuantErr: 12.59986 batch_time=0.40235
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 8.19538 (QuantReg: 12.56106) QuantErr: 12.56106 batch_time=0.41796
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 8.23368 (QuantReg: 12.99952) QuantErr: 12.99952 batch_time=0.44500
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 7.26841 (QuantReg: 13.02078) QuantErr: 13.02078 batch_time=0.39759
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 8.05673 (QuantReg: 12.84521) QuantErr: 12.84521 batch_time=0.40992
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 8.96795 (QuantReg: 12.74822) QuantErr: 12.74822 batch_time=0.40429
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 8.63573 (QuantReg: 12.81245) QuantErr: 12.81245 batch_time=0.39419
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 8.53051 (QuantReg: 12.67827) QuantErr: 12.67827 batch_time=0.39760
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 8.82734 (QuantReg: 12.81610) QuantErr: 12.81610 batch_time=0.40215
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 8.11588 (QuantReg: 12.66601) QuantErr: 12.66601 batch_time=2.23313
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 7.61860 (QuantReg: 12.91309) QuantErr: 12.91309 batch_time=0.39401
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.25941 (QuantReg: 12.79525) QuantErr: 12.79525 batch_time=0.40640
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 8.40029 (QuantReg: 12.81400) QuantErr: 12.81400 batch_time=0.42754
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 8.16442 (QuantReg: 13.00103) QuantErr: 13.00103 batch_time=0.40893
Train Epoch: 9 codebook_update_time=0.55804
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch9.pth ...
Done in 10.272s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch9.pth ...
Done in 14.642s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 8.124150358200072
quant_reg : 12.717600269317627
quant_err : 12.717600269317627
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 18.3
MSRVTT_miech_test/t2v_metrics/R5: 47.3
MSRVTT_miech_test/t2v_metrics/R10: 61.0
MSRVTT_miech_test/t2v_metrics/R50: 87.5
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.185
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.51578357805454
MSRVTT_miech_test/v2t_metrics/R1: 17.6
MSRVTT_miech_test/v2t_metrics/R5: 47.8
MSRVTT_miech_test/v2t_metrics/R10: 61.6
MSRVTT_miech_test/v2t_metrics/R50: 88.2
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.9405
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.282677329868164
mnt_best : 37.51578357805454
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 9.77574 (QuantReg: 12.59635) QuantErr: 12.59635 batch_time=29.31958
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 7.05274 (QuantReg: 12.72404) QuantErr: 12.72404 batch_time=1.09064
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 8.81480 (QuantReg: 12.89752) QuantErr: 12.89752 batch_time=0.43278
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 7.53244 (QuantReg: 12.69039) QuantErr: 12.69039 batch_time=0.40120
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 7.83185 (QuantReg: 12.43798) QuantErr: 12.43798 batch_time=0.43124
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 8.32508 (QuantReg: 12.10097) QuantErr: 12.10097 batch_time=0.39249
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 6.99112 (QuantReg: 12.71524) QuantErr: 12.71524 batch_time=0.41188
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 6.55364 (QuantReg: 12.86478) QuantErr: 12.86478 batch_time=0.40948
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 7.96481 (QuantReg: 12.89153) QuantErr: 12.89153 batch_time=0.41279
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 9.24498 (QuantReg: 12.61623) QuantErr: 12.61623 batch_time=0.43308
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 8.04574 (QuantReg: 13.02652) QuantErr: 13.02652 batch_time=0.45071
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 7.07183 (QuantReg: 12.92027) QuantErr: 12.92027 batch_time=0.39429
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 7.45740 (QuantReg: 13.16500) QuantErr: 13.16500 batch_time=0.38863
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 7.41579 (QuantReg: 13.01050) QuantErr: 13.01050 batch_time=0.39714
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 6.37210 (QuantReg: 12.89501) QuantErr: 12.89501 batch_time=0.45551
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 8.73458 (QuantReg: 13.04216) QuantErr: 13.04216 batch_time=0.41050
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 6.70556 (QuantReg: 13.08480) QuantErr: 13.08480 batch_time=0.39811
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 7.50230 (QuantReg: 12.69728) QuantErr: 12.69728 batch_time=0.38897
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 6.94935 (QuantReg: 12.89904) QuantErr: 12.89904 batch_time=0.41180
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 7.92981 (QuantReg: 12.72688) QuantErr: 12.72688 batch_time=0.42661
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.55096 (QuantReg: 12.83976) QuantErr: 12.83976 batch_time=0.41069
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 7.07967 (QuantReg: 12.68853) QuantErr: 12.68853 batch_time=0.45431
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 8.55502 (QuantReg: 12.69946) QuantErr: 12.69946 batch_time=0.39042
Train Epoch: 10 codebook_update_time=0.41223
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch10.pth ...
Done in 4.279s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch10.pth ...
Done in 8.346s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 7.658029893875122
quant_reg : 12.785059501647948
quant_err : 12.785059501647948
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 19.1
MSRVTT_miech_test/t2v_metrics/R5: 48.2
MSRVTT_miech_test/t2v_metrics/R10: 61.7
MSRVTT_miech_test/t2v_metrics/R50: 88.5
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.44
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.440455320834644
MSRVTT_miech_test/v2t_metrics/R1: 18.9
MSRVTT_miech_test/v2t_metrics/R5: 48.6
MSRVTT_miech_test/v2t_metrics/R10: 62.9
MSRVTT_miech_test/v2t_metrics/R50: 87.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.8955
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.658907166033856
mnt_best : 38.440455320834644
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 6.23221 (QuantReg: 12.61442) QuantErr: 12.61442 batch_time=32.81059
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 7.58251 (QuantReg: 12.66460) QuantErr: 12.66460 batch_time=0.39939
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 6.90701 (QuantReg: 12.56349) QuantErr: 12.56349 batch_time=0.39072
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 6.90905 (QuantReg: 12.56117) QuantErr: 12.56117 batch_time=0.39600
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 9.14611 (QuantReg: 12.80264) QuantErr: 12.80264 batch_time=0.39179
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 6.14836 (QuantReg: 12.72223) QuantErr: 12.72223 batch_time=0.42241
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 6.39867 (QuantReg: 12.68338) QuantErr: 12.68338 batch_time=0.40587
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 7.11485 (QuantReg: 12.63764) QuantErr: 12.63764 batch_time=0.39774
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 9.35700 (QuantReg: 12.60250) QuantErr: 12.60250 batch_time=0.41131
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 5.69620 (QuantReg: 12.94583) QuantErr: 12.94583 batch_time=0.40267
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 7.77412 (QuantReg: 12.83961) QuantErr: 12.83961 batch_time=0.39344
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 7.35857 (QuantReg: 12.88034) QuantErr: 12.88034 batch_time=0.44703
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 7.72105 (QuantReg: 13.05387) QuantErr: 13.05387 batch_time=0.41018
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 7.79140 (QuantReg: 12.73355) QuantErr: 12.73355 batch_time=0.39514
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 9.57001 (QuantReg: 12.32803) QuantErr: 12.32803 batch_time=0.39152
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 6.64657 (QuantReg: 12.87786) QuantErr: 12.87786 batch_time=0.44024
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 7.15321 (QuantReg: 12.88130) QuantErr: 12.88130 batch_time=0.40701
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 7.33599 (QuantReg: 13.03074) QuantErr: 13.03074 batch_time=0.52297
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 7.85314 (QuantReg: 12.78380) QuantErr: 12.78380 batch_time=0.39053
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 7.12506 (QuantReg: 12.83935) QuantErr: 12.83935 batch_time=1.91643
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 7.47674 (QuantReg: 12.82886) QuantErr: 12.82886 batch_time=0.40039
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 7.45290 (QuantReg: 13.10509) QuantErr: 13.10509 batch_time=0.43127
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 6.65638 (QuantReg: 12.80432) QuantErr: 12.80432 batch_time=0.39039
Train Epoch: 11 codebook_update_time=0.45468
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch11.pth ...
Done in 10.285s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 7.24351321220398
quant_reg : 12.79260122680664
quant_err : 12.79260122680664
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 19.1
MSRVTT_miech_test/t2v_metrics/R5: 46.8
MSRVTT_miech_test/t2v_metrics/R10: 61.4
MSRVTT_miech_test/t2v_metrics/R50: 87.3
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.611999999999995
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.002823428254445
MSRVTT_miech_test/v2t_metrics/R1: 18.2
MSRVTT_miech_test/v2t_metrics/R5: 49.8
MSRVTT_miech_test/v2t_metrics/R10: 62.0
MSRVTT_miech_test/v2t_metrics/R50: 87.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.8855
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.302825043226505
mnt_best : 38.440455320834644
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 7.75213 (QuantReg: 12.61502) QuantErr: 12.61502 batch_time=30.60708
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 8.20796 (QuantReg: 12.68279) QuantErr: 12.68279 batch_time=0.42423
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 5.84435 (QuantReg: 12.82727) QuantErr: 12.82727 batch_time=0.44454
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 6.18951 (QuantReg: 12.71229) QuantErr: 12.71229 batch_time=0.40743
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 8.80280 (QuantReg: 12.53972) QuantErr: 12.53972 batch_time=0.40154
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 5.94572 (QuantReg: 12.55745) QuantErr: 12.55745 batch_time=0.40429
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 5.91257 (QuantReg: 12.58607) QuantErr: 12.58607 batch_time=0.41559
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 5.88237 (QuantReg: 12.69136) QuantErr: 12.69136 batch_time=0.42107
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 7.34632 (QuantReg: 12.68067) QuantErr: 12.68067 batch_time=0.64523
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 7.54191 (QuantReg: 12.36799) QuantErr: 12.36799 batch_time=0.41005
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 7.11271 (QuantReg: 12.99376) QuantErr: 12.99376 batch_time=0.41359
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 7.16062 (QuantReg: 12.79232) QuantErr: 12.79232 batch_time=0.43434
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 7.16783 (QuantReg: 12.53685) QuantErr: 12.53685 batch_time=0.75416
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 6.24167 (QuantReg: 12.86598) QuantErr: 12.86598 batch_time=1.33247
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 6.92331 (QuantReg: 12.70955) QuantErr: 12.70955 batch_time=0.40606
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 7.04610 (QuantReg: 12.66991) QuantErr: 12.66991 batch_time=0.42504
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 6.74936 (QuantReg: 12.82487) QuantErr: 12.82487 batch_time=0.39874
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 5.42318 (QuantReg: 12.77000) QuantErr: 12.77000 batch_time=0.40295
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 7.94761 (QuantReg: 12.62410) QuantErr: 12.62410 batch_time=0.39217
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 6.65994 (QuantReg: 13.23767) QuantErr: 13.23767 batch_time=0.39735
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 6.32958 (QuantReg: 12.92194) QuantErr: 12.92194 batch_time=0.43300
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 6.87943 (QuantReg: 12.69862) QuantErr: 12.69862 batch_time=0.40065
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 6.70328 (QuantReg: 13.28727) QuantErr: 13.28727 batch_time=0.40312
Train Epoch: 12 codebook_update_time=0.41613
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch12.pth ...
Done in 4.606s
removing stale ckpt [epoch 11] [took 0.09s]
epoch : 12
loss : 6.955071601867676
quant_reg : 12.803211044311523
quant_err : 12.803211044311523
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 17.4
MSRVTT_miech_test/t2v_metrics/R5: 46.9
MSRVTT_miech_test/t2v_metrics/R10: 62.0
MSRVTT_miech_test/t2v_metrics/R50: 88.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.119
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.98604781959713
MSRVTT_miech_test/v2t_metrics/R1: 18.8
MSRVTT_miech_test/v2t_metrics/R5: 48.3
MSRVTT_miech_test/v2t_metrics/R10: 62.5
MSRVTT_miech_test/v2t_metrics/R50: 87.6
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.006
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.42922850636274
mnt_best : 38.440455320834644
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 8.00680 (QuantReg: 12.80807) QuantErr: 12.80807 batch_time=29.63714
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 5.57565 (QuantReg: 12.72810) QuantErr: 12.72810 batch_time=0.42994
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 6.25076 (QuantReg: 12.74004) QuantErr: 12.74004 batch_time=0.39934
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.66475 (QuantReg: 12.55607) QuantErr: 12.55607 batch_time=0.40650
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 6.74009 (QuantReg: 12.65884) QuantErr: 12.65884 batch_time=0.39109
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 6.30482 (QuantReg: 12.21008) QuantErr: 12.21008 batch_time=0.39635
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 7.31389 (QuantReg: 12.91507) QuantErr: 12.91507 batch_time=0.46677
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 7.65376 (QuantReg: 12.69604) QuantErr: 12.69604 batch_time=0.41878
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 6.15424 (QuantReg: 12.80378) QuantErr: 12.80378 batch_time=0.41608
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 6.82071 (QuantReg: 12.55470) QuantErr: 12.55470 batch_time=0.39755
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 6.96746 (QuantReg: 12.70672) QuantErr: 12.70672 batch_time=0.41262
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 5.06358 (QuantReg: 13.05586) QuantErr: 13.05586 batch_time=0.41813
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 7.00041 (QuantReg: 12.94532) QuantErr: 12.94532 batch_time=0.40258
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 5.96446 (QuantReg: 12.82892) QuantErr: 12.82892 batch_time=1.34051
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 6.71627 (QuantReg: 12.66548) QuantErr: 12.66548 batch_time=0.40702
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 6.84561 (QuantReg: 12.49758) QuantErr: 12.49758 batch_time=0.63679
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 6.67455 (QuantReg: 13.05005) QuantErr: 13.05005 batch_time=0.41469
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 8.62417 (QuantReg: 13.05444) QuantErr: 13.05444 batch_time=0.39564
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 5.98583 (QuantReg: 13.07187) QuantErr: 13.07187 batch_time=0.40672
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 6.78169 (QuantReg: 12.66926) QuantErr: 12.66926 batch_time=0.39221
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 8.03541 (QuantReg: 12.98759) QuantErr: 12.98759 batch_time=0.40551
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 5.90882 (QuantReg: 13.04812) QuantErr: 13.04812 batch_time=0.50978
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 5.04740 (QuantReg: 13.17775) QuantErr: 13.17775 batch_time=0.38767
Train Epoch: 13 codebook_update_time=0.39790
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch13.pth ...
Done in 6.009s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch13.pth ...
Done in 11.933s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 6.671078981399536
quant_reg : 12.813359668731689
quant_err : 12.813359668731689
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 18.8
MSRVTT_miech_test/t2v_metrics/R5: 47.9
MSRVTT_miech_test/t2v_metrics/R10: 63.3
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.504
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.48566757229646
MSRVTT_miech_test/v2t_metrics/R1: 19.0
MSRVTT_miech_test/v2t_metrics/R5: 48.8
MSRVTT_miech_test/v2t_metrics/R10: 63.2
MSRVTT_miech_test/v2t_metrics/R50: 87.7
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.6155
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.84157516106784
mnt_best : 38.48566757229646
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 5.41910 (QuantReg: 12.67362) QuantErr: 12.67362 batch_time=29.65757
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 7.13767 (QuantReg: 12.54029) QuantErr: 12.54029 batch_time=0.43383
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 6.84805 (QuantReg: 12.80293) QuantErr: 12.80293 batch_time=0.42103
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 6.74500 (QuantReg: 12.61579) QuantErr: 12.61579 batch_time=0.40263
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 5.72762 (QuantReg: 12.73575) QuantErr: 12.73575 batch_time=0.38626
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 7.36359 (QuantReg: 12.96107) QuantErr: 12.96107 batch_time=0.39522
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 5.76429 (QuantReg: 13.03094) QuantErr: 13.03094 batch_time=0.44351
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 6.42421 (QuantReg: 12.66327) QuantErr: 12.66327 batch_time=0.44949
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 5.32382 (QuantReg: 12.97871) QuantErr: 12.97871 batch_time=0.42270
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 7.20532 (QuantReg: 13.15831) QuantErr: 13.15831 batch_time=0.43766
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 7.31882 (QuantReg: 12.64476) QuantErr: 12.64476 batch_time=0.42362
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 5.98668 (QuantReg: 12.69309) QuantErr: 12.69309 batch_time=0.41047
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 6.63371 (QuantReg: 12.77687) QuantErr: 12.77687 batch_time=0.42699
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 6.44503 (QuantReg: 12.99499) QuantErr: 12.99499 batch_time=2.65594
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 7.53452 (QuantReg: 12.92243) QuantErr: 12.92243 batch_time=0.39758
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 7.25048 (QuantReg: 12.83589) QuantErr: 12.83589 batch_time=0.64730
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 6.21372 (QuantReg: 12.94087) QuantErr: 12.94087 batch_time=0.38872
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 6.41736 (QuantReg: 13.09699) QuantErr: 13.09699 batch_time=0.44119
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 7.11315 (QuantReg: 12.76127) QuantErr: 12.76127 batch_time=0.40662
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 7.58450 (QuantReg: 12.72734) QuantErr: 12.72734 batch_time=0.40177
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 5.29212 (QuantReg: 12.79992) QuantErr: 12.79992 batch_time=0.38673
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 6.23398 (QuantReg: 13.12774) QuantErr: 13.12774 batch_time=0.39227
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 6.42232 (QuantReg: 13.00933) QuantErr: 13.00933 batch_time=0.39544
Train Epoch: 14 codebook_update_time=0.41635
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch14.pth ...
Done in 5.400s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch14.pth ...
Done in 12.225s
removing stale ckpt [epoch 13] [took 0.03s]
epoch : 14
loss : 6.566139265060425
quant_reg : 12.852768199920654
quant_err : 12.852768199920654
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 19.8
MSRVTT_miech_test/t2v_metrics/R5: 49.2
MSRVTT_miech_test/t2v_metrics/R10: 62.8
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.544
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.40306262535618
MSRVTT_miech_test/v2t_metrics/R1: 19.0
MSRVTT_miech_test/v2t_metrics/R5: 47.2
MSRVTT_miech_test/v2t_metrics/R10: 61.2
MSRVTT_miech_test/v2t_metrics/R50: 89.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.049
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.00280681021812
mnt_best : 39.40306262535618
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 7.26442 (QuantReg: 12.83354) QuantErr: 12.83354 batch_time=32.48766
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 5.56310 (QuantReg: 12.72239) QuantErr: 12.72239 batch_time=0.42200
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 4.99273 (QuantReg: 12.90024) QuantErr: 12.90024 batch_time=0.40383
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 6.98096 (QuantReg: 12.83415) QuantErr: 12.83415 batch_time=0.40525
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 6.16877 (QuantReg: 12.97118) QuantErr: 12.97118 batch_time=1.36293
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 5.20367 (QuantReg: 13.01758) QuantErr: 13.01758 batch_time=0.40008
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 5.00353 (QuantReg: 12.97217) QuantErr: 12.97217 batch_time=0.40124
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 7.01937 (QuantReg: 13.19083) QuantErr: 13.19083 batch_time=0.45082
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 7.76471 (QuantReg: 13.05806) QuantErr: 13.05806 batch_time=0.43499
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 7.43489 (QuantReg: 12.63867) QuantErr: 12.63867 batch_time=0.43037
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 6.43687 (QuantReg: 12.78239) QuantErr: 12.78239 batch_time=1.11554
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 6.56765 (QuantReg: 12.67868) QuantErr: 12.67868 batch_time=0.40326
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.91045 (QuantReg: 12.74588) QuantErr: 12.74588 batch_time=0.52260
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 6.99497 (QuantReg: 12.97610) QuantErr: 12.97610 batch_time=0.65661
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 6.46512 (QuantReg: 13.10224) QuantErr: 13.10224 batch_time=0.41632
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 5.81718 (QuantReg: 12.58477) QuantErr: 12.58477 batch_time=0.97406
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 6.18705 (QuantReg: 12.74498) QuantErr: 12.74498 batch_time=0.42091
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 5.40671 (QuantReg: 12.94444) QuantErr: 12.94444 batch_time=0.40312
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 6.29244 (QuantReg: 13.01442) QuantErr: 13.01442 batch_time=0.78304
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 4.40582 (QuantReg: 12.91537) QuantErr: 12.91537 batch_time=0.45780
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 6.07645 (QuantReg: 13.33785) QuantErr: 13.33785 batch_time=0.40464
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 6.64074 (QuantReg: 12.96900) QuantErr: 12.96900 batch_time=0.40889
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 4.94245 (QuantReg: 13.29122) QuantErr: 13.29122 batch_time=0.40759
Train Epoch: 15 codebook_update_time=0.71749
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch15.pth ...
Done in 5.972s
removing stale ckpt [epoch 14] [took 0.17s]
epoch : 15
loss : 6.290655345916748
quant_reg : 12.87828086090088
quant_err : 12.87828086090088
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 18.4
MSRVTT_miech_test/t2v_metrics/R5: 47.1
MSRVTT_miech_test/t2v_metrics/R10: 61.6
MSRVTT_miech_test/t2v_metrics/R50: 87.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.221
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.65359792356258
MSRVTT_miech_test/v2t_metrics/R1: 18.1
MSRVTT_miech_test/v2t_metrics/R5: 48.6
MSRVTT_miech_test/v2t_metrics/R10: 63.1
MSRVTT_miech_test/v2t_metrics/R50: 88.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.098
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.14591773013611
mnt_best : 39.40306262535618
not_improved_count: 1
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 6.33853 (QuantReg: 12.70792) QuantErr: 12.70792 batch_time=31.87988
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 5.65800 (QuantReg: 13.02724) QuantErr: 13.02724 batch_time=0.47382
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 6.88049 (QuantReg: 13.23194) QuantErr: 13.23194 batch_time=0.40267
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 5.98066 (QuantReg: 12.69251) QuantErr: 12.69251 batch_time=0.40961
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 5.43647 (QuantReg: 12.83436) QuantErr: 12.83436 batch_time=0.39488
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 6.08065 (QuantReg: 12.73471) QuantErr: 12.73471 batch_time=0.40821
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 6.34881 (QuantReg: 12.63976) QuantErr: 12.63976 batch_time=0.49925
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 6.47927 (QuantReg: 12.81815) QuantErr: 12.81815 batch_time=2.33310
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 6.14685 (QuantReg: 12.75630) QuantErr: 12.75630 batch_time=0.56125
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 6.30007 (QuantReg: 12.79685) QuantErr: 12.79685 batch_time=0.42571
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 7.33561 (QuantReg: 12.96419) QuantErr: 12.96419 batch_time=0.40702
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 5.31677 (QuantReg: 12.81240) QuantErr: 12.81240 batch_time=0.42244
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 5.64969 (QuantReg: 12.92088) QuantErr: 12.92088 batch_time=0.40372
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 6.79121 (QuantReg: 13.01323) QuantErr: 13.01323 batch_time=0.39178
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 6.24947 (QuantReg: 12.89592) QuantErr: 12.89592 batch_time=0.43565
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 7.11370 (QuantReg: 12.70207) QuantErr: 12.70207 batch_time=0.40888
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 5.97202 (QuantReg: 13.40416) QuantErr: 13.40416 batch_time=0.40699
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 5.92865 (QuantReg: 12.67877) QuantErr: 12.67877 batch_time=0.39190
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 5.64436 (QuantReg: 13.19504) QuantErr: 13.19504 batch_time=0.83662
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 7.89751 (QuantReg: 12.79161) QuantErr: 12.79161 batch_time=0.42641
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 5.24595 (QuantReg: 13.26579) QuantErr: 13.26579 batch_time=0.39905
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 7.56623 (QuantReg: 12.78615) QuantErr: 12.78615 batch_time=0.46260
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 7.04414 (QuantReg: 13.00375) QuantErr: 13.00375 batch_time=0.41870
Train Epoch: 16 codebook_update_time=0.42614
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch16.pth ...
Done in 6.355s
removing stale ckpt [epoch 15] [took 0.21s]
epoch : 16
loss : 6.066251697540284
quant_reg : 12.886809631347656
quant_err : 12.886809631347656
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 18.7
MSRVTT_miech_test/t2v_metrics/R5: 48.0
MSRVTT_miech_test/t2v_metrics/R10: 62.4
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.318
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.26095546752782
MSRVTT_miech_test/v2t_metrics/R1: 19.7
MSRVTT_miech_test/v2t_metrics/R5: 49.7
MSRVTT_miech_test/v2t_metrics/R10: 64.2
MSRVTT_miech_test/v2t_metrics/R50: 88.3
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.264
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.7605650489785
mnt_best : 39.40306262535618
not_improved_count: 2
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 6.41848 (QuantReg: 12.70818) QuantErr: 12.70818 batch_time=29.49727
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 4.71250 (QuantReg: 12.34892) QuantErr: 12.34892 batch_time=0.41902
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 5.98146 (QuantReg: 12.81002) QuantErr: 12.81002 batch_time=1.36175
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 5.41272 (QuantReg: 12.84713) QuantErr: 12.84713 batch_time=0.41938
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 6.72270 (QuantReg: 12.97453) QuantErr: 12.97453 batch_time=0.40778
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 5.88318 (QuantReg: 12.63791) QuantErr: 12.63791 batch_time=0.41159
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 6.16361 (QuantReg: 12.71667) QuantErr: 12.71667 batch_time=0.39841
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 6.20574 (QuantReg: 12.80159) QuantErr: 12.80159 batch_time=0.40027
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 5.39356 (QuantReg: 12.89148) QuantErr: 12.89148 batch_time=0.38926
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 6.94352 (QuantReg: 13.11692) QuantErr: 13.11692 batch_time=0.40930
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 5.83023 (QuantReg: 13.15109) QuantErr: 13.15109 batch_time=0.64665
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 7.26527 (QuantReg: 12.91320) QuantErr: 12.91320 batch_time=0.40650
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 6.20982 (QuantReg: 12.65449) QuantErr: 12.65449 batch_time=0.40503
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 6.41471 (QuantReg: 13.17655) QuantErr: 13.17655 batch_time=3.25359
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 5.29851 (QuantReg: 12.83221) QuantErr: 12.83221 batch_time=0.40402
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 4.79710 (QuantReg: 13.01221) QuantErr: 13.01221 batch_time=0.40836
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 7.99269 (QuantReg: 12.87910) QuantErr: 12.87910 batch_time=0.42090
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 5.81677 (QuantReg: 12.89763) QuantErr: 12.89763 batch_time=0.39773
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 5.19057 (QuantReg: 12.91610) QuantErr: 12.91610 batch_time=0.43355
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 5.85785 (QuantReg: 12.87201) QuantErr: 12.87201 batch_time=0.40351
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 5.13454 (QuantReg: 13.17515) QuantErr: 13.17515 batch_time=0.39497
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 5.76191 (QuantReg: 13.08790) QuantErr: 13.08790 batch_time=0.40346
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 5.58505 (QuantReg: 12.92857) QuantErr: 12.92857 batch_time=0.40955
Train Epoch: 17 codebook_update_time=0.42518
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch17.pth ...
Done in 4.887s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 5.982693190574646
quant_reg : 12.919715057373047
quant_err : 12.919715057373047
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 19.0
MSRVTT_miech_test/t2v_metrics/R5: 48.7
MSRVTT_miech_test/t2v_metrics/R10: 61.8
MSRVTT_miech_test/t2v_metrics/R50: 87.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.406
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.5262743009434
MSRVTT_miech_test/v2t_metrics/R1: 18.1
MSRVTT_miech_test/v2t_metrics/R5: 48.8
MSRVTT_miech_test/v2t_metrics/R10: 62.5
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.798
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.07671482911124
mnt_best : 39.40306262535618
not_improved_count: 3
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 5.08833 (QuantReg: 12.76469) QuantErr: 12.76469 batch_time=28.54726
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 7.39691 (QuantReg: 12.77684) QuantErr: 12.77684 batch_time=0.61032
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 5.08876 (QuantReg: 12.97323) QuantErr: 12.97323 batch_time=0.43953
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 6.71364 (QuantReg: 12.84135) QuantErr: 12.84135 batch_time=0.39174
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 5.44767 (QuantReg: 12.62852) QuantErr: 12.62852 batch_time=0.40091
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 5.16510 (QuantReg: 12.98937) QuantErr: 12.98937 batch_time=0.39299
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 4.60040 (QuantReg: 12.86418) QuantErr: 12.86418 batch_time=0.65962
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 5.04828 (QuantReg: 13.12074) QuantErr: 13.12074 batch_time=0.38955
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 5.12724 (QuantReg: 12.87836) QuantErr: 12.87836 batch_time=0.38896
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 4.42504 (QuantReg: 12.99546) QuantErr: 12.99546 batch_time=0.42833
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 4.74619 (QuantReg: 12.99942) QuantErr: 12.99942 batch_time=0.51962
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 5.15679 (QuantReg: 12.70933) QuantErr: 12.70933 batch_time=0.39434
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 6.41009 (QuantReg: 12.89111) QuantErr: 12.89111 batch_time=1.06730
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 5.79261 (QuantReg: 13.09463) QuantErr: 13.09463 batch_time=0.43344
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 5.29962 (QuantReg: 12.84764) QuantErr: 12.84764 batch_time=0.43905
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 6.60105 (QuantReg: 13.01006) QuantErr: 13.01006 batch_time=0.38883
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 5.86927 (QuantReg: 13.06829) QuantErr: 13.06829 batch_time=0.42604
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 4.72113 (QuantReg: 13.06340) QuantErr: 13.06340 batch_time=0.39801
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 6.78396 (QuantReg: 12.96646) QuantErr: 12.96646 batch_time=0.61112
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 5.94886 (QuantReg: 12.83607) QuantErr: 12.83607 batch_time=1.82138
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 5.61599 (QuantReg: 12.54996) QuantErr: 12.54996 batch_time=0.39917
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 6.37433 (QuantReg: 13.06063) QuantErr: 13.06063 batch_time=1.35946
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 5.55333 (QuantReg: 13.14573) QuantErr: 13.14573 batch_time=0.40863
Train Epoch: 18 codebook_update_time=0.41727
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch18.pth ...
Done in 5.320s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch18.pth ...
Done in 10.805s
removing stale ckpt [epoch 17] [took 0.03s]
epoch : 18
loss : 5.6984017162323
quant_reg : 12.956782176971435
quant_err : 12.956782176971435
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 20.8
MSRVTT_miech_test/t2v_metrics/R5: 48.5
MSRVTT_miech_test/t2v_metrics/R10: 62.0
MSRVTT_miech_test/t2v_metrics/R50: 87.3
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.441
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.69467535133183
MSRVTT_miech_test/v2t_metrics/R1: 17.6
MSRVTT_miech_test/v2t_metrics/R5: 48.6
MSRVTT_miech_test/v2t_metrics/R10: 64.9
MSRVTT_miech_test/v2t_metrics/R50: 88.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.895
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.14736498745409
mnt_best : 39.69467535133183
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 5.71018 (QuantReg: 13.04189) QuantErr: 13.04189 batch_time=31.84001
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 6.15522 (QuantReg: 12.68980) QuantErr: 12.68980 batch_time=0.40641
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 4.84386 (QuantReg: 12.76982) QuantErr: 12.76982 batch_time=0.41954
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 4.92876 (QuantReg: 12.89660) QuantErr: 12.89660 batch_time=0.40764
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 5.97753 (QuantReg: 13.02699) QuantErr: 13.02699 batch_time=0.40396
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 4.99155 (QuantReg: 12.85651) QuantErr: 12.85651 batch_time=0.39580
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 5.59758 (QuantReg: 12.57250) QuantErr: 12.57250 batch_time=0.41831
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 4.57738 (QuantReg: 13.16577) QuantErr: 13.16577 batch_time=0.41309
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 5.65703 (QuantReg: 12.86194) QuantErr: 12.86194 batch_time=0.42306
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 5.64090 (QuantReg: 12.97902) QuantErr: 12.97902 batch_time=0.41092
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 4.59775 (QuantReg: 13.08072) QuantErr: 13.08072 batch_time=0.45826
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 5.87112 (QuantReg: 13.29820) QuantErr: 13.29820 batch_time=0.41345
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 4.57489 (QuantReg: 12.97093) QuantErr: 12.97093 batch_time=0.42171
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 6.24869 (QuantReg: 13.00240) QuantErr: 13.00240 batch_time=0.40892
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 5.47125 (QuantReg: 12.90109) QuantErr: 12.90109 batch_time=0.41945
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 5.96234 (QuantReg: 12.97747) QuantErr: 12.97747 batch_time=0.42349
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 5.53093 (QuantReg: 13.05242) QuantErr: 13.05242 batch_time=0.42661
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 6.37512 (QuantReg: 13.24596) QuantErr: 13.24596 batch_time=0.55126
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 5.41521 (QuantReg: 13.11623) QuantErr: 13.11623 batch_time=0.51051
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 5.98955 (QuantReg: 13.17638) QuantErr: 13.17638 batch_time=0.39130
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 5.82096 (QuantReg: 13.22832) QuantErr: 13.22832 batch_time=0.59223
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 3.82603 (QuantReg: 13.21289) QuantErr: 13.21289 batch_time=0.41802
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 6.34380 (QuantReg: 12.75702) QuantErr: 12.75702 batch_time=0.40883
Train Epoch: 19 codebook_update_time=0.48118
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L1/checkpoint-epoch19.pth ...
Done in 5.025s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 5.612300939559937
quant_reg : 12.973538921356202
quant_err : 12.973538921356202
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 19.4
MSRVTT_miech_test/t2v_metrics/R5: 47.6
MSRVTT_miech_test/t2v_metrics/R10: 62.6
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.295
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.665859806125354
MSRVTT_miech_test/v2t_metrics/R1: 18.6