-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_bs256.txt
1853 lines (1853 loc) · 109 KB
/
HCQ_MSRVTT_1kA_bs256.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 875.6317975521088 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 51.9874062538147 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 36.77188491821289 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch0.pth ...
Done in 1.427s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch0.pth ...
Done in 2.851s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 487.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.3
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.544
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/125 256/32000 (1%)] Loss: 11.20562 (QuantReg: 22.45103) QuantErr: 22.45103 batch_time=29.33468
Train Epoch: 1 [17/125 4352/32000 (14%)] Loss: 8.83442 (QuantReg: 22.51726) QuantErr: 22.51726 batch_time=0.75327
Train Epoch: 1 [33/125 8448/32000 (26%)] Loss: 7.75064 (QuantReg: 22.53108) QuantErr: 22.53108 batch_time=0.75988
Train Epoch: 1 [49/125 12544/32000 (39%)] Loss: 7.02025 (QuantReg: 22.61122) QuantErr: 22.61122 batch_time=0.75752
Train Epoch: 1 [65/125 16640/32000 (52%)] Loss: 6.57318 (QuantReg: 22.62815) QuantErr: 22.62815 batch_time=2.18642
Train Epoch: 1 [81/125 20736/32000 (65%)] Loss: 6.39769 (QuantReg: 22.63561) QuantErr: 22.63561 batch_time=0.76663
Train Epoch: 1 [97/125 24832/32000 (78%)] Loss: 5.97622 (QuantReg: 22.63977) QuantErr: 22.63977 batch_time=0.97769
Train Epoch: 1 [113/125 28928/32000 (90%)] Loss: 5.64494 (QuantReg: 22.67752) QuantErr: 22.67752 batch_time=0.76635
Train Epoch: 1 codebook_update_time=1.82869
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch1.pth ...
Done in 3.764s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch1.pth ...
Done in 7.834s
epoch : 1
loss : 7.175728771209717
quant_reg : 22.602456665039064
quant_err : 22.602456665039064
learning_rate : 5e-05
n_samples : 32000
n_steps : 125
MSRVTT_jsfusion_test/t2v_metrics/R1: 9.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 28.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 76.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 46.626
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.237462785061304
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 30.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 43.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 77.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 14.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 44.985
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.044996603532354
mnt_best : 22.237462785061304
not_improved_count: 0
Train Epoch: 2 [1/125 256/32000 (1%)] Loss: 5.41043 (QuantReg: 11.52134) QuantErr: 11.52134 batch_time=31.02310
Train Epoch: 2 [17/125 4352/32000 (14%)] Loss: 5.40632 (QuantReg: 12.54937) QuantErr: 12.54937 batch_time=0.74226
Train Epoch: 2 [33/125 8448/32000 (26%)] Loss: 5.08989 (QuantReg: 13.12999) QuantErr: 13.12999 batch_time=0.91506
Train Epoch: 2 [49/125 12544/32000 (39%)] Loss: 5.16910 (QuantReg: 13.43021) QuantErr: 13.43021 batch_time=0.82254
Train Epoch: 2 [65/125 16640/32000 (52%)] Loss: 4.96524 (QuantReg: 13.87801) QuantErr: 13.87801 batch_time=2.52009
Train Epoch: 2 [81/125 20736/32000 (65%)] Loss: 5.17255 (QuantReg: 14.18652) QuantErr: 14.18652 batch_time=0.74301
Train Epoch: 2 [97/125 24832/32000 (78%)] Loss: 4.64596 (QuantReg: 14.94132) QuantErr: 14.94132 batch_time=0.85420
Train Epoch: 2 [113/125 28928/32000 (90%)] Loss: 4.74921 (QuantReg: 15.10593) QuantErr: 15.10593 batch_time=0.75038
Train Epoch: 2 codebook_update_time=1.79041
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch2.pth ...
Done in 3.898s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch2.pth ...
Done in 14.725s
removing stale ckpt [epoch 1] [took 0.05s]
removing stale ckpt [epoch 0] [took 0.05s]
epoch : 2
loss : 5.08022294998169
quant_reg : 13.701634033203124
quant_err : 13.701634033203124
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 36.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 82.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 10.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 37.059
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.864514973956958
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 38.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.832
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.726309467921734
mnt_best : 29.864514973956958
not_improved_count: 0
Train Epoch: 3 [1/125 256/32000 (1%)] Loss: 4.84651 (QuantReg: 11.96606) QuantErr: 11.96606 batch_time=32.29279
Train Epoch: 3 [17/125 4352/32000 (14%)] Loss: 4.60899 (QuantReg: 12.24570) QuantErr: 12.24570 batch_time=0.84052
Train Epoch: 3 [33/125 8448/32000 (26%)] Loss: 4.52964 (QuantReg: 12.41722) QuantErr: 12.41722 batch_time=0.86767
Train Epoch: 3 [49/125 12544/32000 (39%)] Loss: 4.42103 (QuantReg: 12.68839) QuantErr: 12.68839 batch_time=0.73920
Train Epoch: 3 [65/125 16640/32000 (52%)] Loss: 4.37677 (QuantReg: 12.46487) QuantErr: 12.46487 batch_time=1.52183
Train Epoch: 3 [81/125 20736/32000 (65%)] Loss: 4.17679 (QuantReg: 12.98517) QuantErr: 12.98517 batch_time=0.75323
Train Epoch: 3 [97/125 24832/32000 (78%)] Loss: 4.53278 (QuantReg: 13.06301) QuantErr: 13.06301 batch_time=0.74714
Train Epoch: 3 [113/125 28928/32000 (90%)] Loss: 3.73376 (QuantReg: 13.23227) QuantErr: 13.23227 batch_time=0.75004
Train Epoch: 3 codebook_update_time=1.66529
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch3.pth ...
Done in 3.707s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch3.pth ...
Done in 7.483s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 4.435342176437378
quant_reg : 12.735211463928223
quant_err : 12.735211463928223
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 375
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.831
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.58020301790745
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.558
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.06009984565856
mnt_best : 32.58020301790745
not_improved_count: 0
Train Epoch: 4 [1/125 256/32000 (1%)] Loss: 4.42509 (QuantReg: 12.06829) QuantErr: 12.06829 batch_time=33.54816
Train Epoch: 4 [17/125 4352/32000 (14%)] Loss: 3.84837 (QuantReg: 12.32931) QuantErr: 12.32931 batch_time=0.83057
Train Epoch: 4 [33/125 8448/32000 (26%)] Loss: 3.80875 (QuantReg: 12.46936) QuantErr: 12.46936 batch_time=0.75147
Train Epoch: 4 [49/125 12544/32000 (39%)] Loss: 3.68467 (QuantReg: 12.66700) QuantErr: 12.66700 batch_time=0.76335
Train Epoch: 4 [65/125 16640/32000 (52%)] Loss: 3.97227 (QuantReg: 12.96105) QuantErr: 12.96105 batch_time=0.96973
Train Epoch: 4 [81/125 20736/32000 (65%)] Loss: 3.92663 (QuantReg: 12.76507) QuantErr: 12.76507 batch_time=0.76663
Train Epoch: 4 [97/125 24832/32000 (78%)] Loss: 3.90290 (QuantReg: 13.09428) QuantErr: 13.09428 batch_time=0.76190
Train Epoch: 4 [113/125 28928/32000 (90%)] Loss: 3.84752 (QuantReg: 13.06852) QuantErr: 13.06852 batch_time=0.77221
Train Epoch: 4 codebook_update_time=1.83282
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch4.pth ...
Done in 3.780s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch4.pth ...
Done in 7.589s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 3.975587907791138
quant_reg : 12.673847160339356
quant_err : 12.673847160339356
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.456
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.21136857306636
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.036
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.200533738463356
mnt_best : 35.21136857306636
not_improved_count: 0
Train Epoch: 5 [1/125 256/32000 (1%)] Loss: 4.12166 (QuantReg: 12.28304) QuantErr: 12.28304 batch_time=33.58626
Train Epoch: 5 [17/125 4352/32000 (14%)] Loss: 4.16667 (QuantReg: 12.30247) QuantErr: 12.30247 batch_time=0.76364
Train Epoch: 5 [33/125 8448/32000 (26%)] Loss: 3.56564 (QuantReg: 12.59444) QuantErr: 12.59444 batch_time=0.77047
Train Epoch: 5 [49/125 12544/32000 (39%)] Loss: 3.85775 (QuantReg: 12.78645) QuantErr: 12.78645 batch_time=0.75204
Train Epoch: 5 [65/125 16640/32000 (52%)] Loss: 3.89427 (QuantReg: 12.54776) QuantErr: 12.54776 batch_time=0.96986
Train Epoch: 5 [81/125 20736/32000 (65%)] Loss: 3.82668 (QuantReg: 12.96520) QuantErr: 12.96520 batch_time=0.76132
Train Epoch: 5 [97/125 24832/32000 (78%)] Loss: 3.78395 (QuantReg: 12.69822) QuantErr: 12.69822 batch_time=0.77221
Train Epoch: 5 [113/125 28928/32000 (90%)] Loss: 3.54601 (QuantReg: 12.98527) QuantErr: 12.98527 batch_time=0.76464
Train Epoch: 5 codebook_update_time=1.65418
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch5.pth ...
Done in 3.938s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch5.pth ...
Done in 7.982s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 3.70991526222229
quant_reg : 12.763653953552247
quant_err : 12.763653953552247
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 625
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.109
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.15341932613529
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.25
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.712
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.588095106887614
mnt_best : 36.15341932613529
not_improved_count: 0
Train Epoch: 6 [1/125 256/32000 (1%)] Loss: 3.70805 (QuantReg: 12.80660) QuantErr: 12.80660 batch_time=39.11480
Train Epoch: 6 [17/125 4352/32000 (14%)] Loss: 3.39814 (QuantReg: 12.65313) QuantErr: 12.65313 batch_time=0.75000
Train Epoch: 6 [33/125 8448/32000 (26%)] Loss: 3.49766 (QuantReg: 13.04769) QuantErr: 13.04769 batch_time=0.76002
Train Epoch: 6 [49/125 12544/32000 (39%)] Loss: 3.15641 (QuantReg: 12.72586) QuantErr: 12.72586 batch_time=0.75350
Train Epoch: 6 [65/125 16640/32000 (52%)] Loss: 3.28426 (QuantReg: 13.00606) QuantErr: 13.00606 batch_time=2.13722
Train Epoch: 6 [81/125 20736/32000 (65%)] Loss: 3.70489 (QuantReg: 12.85954) QuantErr: 12.85954 batch_time=0.76682
Train Epoch: 6 [97/125 24832/32000 (78%)] Loss: 3.54616 (QuantReg: 13.21817) QuantErr: 13.21817 batch_time=0.75232
Train Epoch: 6 [113/125 28928/32000 (90%)] Loss: 3.11286 (QuantReg: 12.95079) QuantErr: 12.95079 batch_time=0.74996
Train Epoch: 6 codebook_update_time=1.66775
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch6.pth ...
Done in 3.855s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch6.pth ...
Done in 7.688s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 3.480328285217285
quant_reg : 12.877009132385254
quant_err : 12.877009132385254
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.824
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.26696090958165
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.1245
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.100174288440186
mnt_best : 36.26696090958165
not_improved_count: 0
Train Epoch: 7 [1/125 256/32000 (1%)] Loss: 3.19805 (QuantReg: 12.52387) QuantErr: 12.52387 batch_time=42.75564
Train Epoch: 7 [17/125 4352/32000 (14%)] Loss: 3.15768 (QuantReg: 12.61928) QuantErr: 12.61928 batch_time=3.70152
Train Epoch: 7 [33/125 8448/32000 (26%)] Loss: 3.59821 (QuantReg: 12.73985) QuantErr: 12.73985 batch_time=0.74571
Train Epoch: 7 [49/125 12544/32000 (39%)] Loss: 3.01964 (QuantReg: 12.97170) QuantErr: 12.97170 batch_time=0.74825
Train Epoch: 7 [65/125 16640/32000 (52%)] Loss: 3.05860 (QuantReg: 12.95887) QuantErr: 12.95887 batch_time=9.61617
Train Epoch: 7 [81/125 20736/32000 (65%)] Loss: 3.37847 (QuantReg: 12.89236) QuantErr: 12.89236 batch_time=3.95856
Train Epoch: 7 [97/125 24832/32000 (78%)] Loss: 3.13971 (QuantReg: 12.96136) QuantErr: 12.96136 batch_time=0.85947
Train Epoch: 7 [113/125 28928/32000 (90%)] Loss: 3.35174 (QuantReg: 13.30768) QuantErr: 13.30768 batch_time=0.84625
Train Epoch: 7 codebook_update_time=1.67845
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch7.pth ...
Done in 3.894s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch7.pth ...
Done in 7.909s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 3.253774677276611
quant_reg : 12.934579750061035
quant_err : 12.934579750061035
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 875
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.959
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.992563641799855
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.034
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.872156441892635
mnt_best : 37.992563641799855
not_improved_count: 0
Train Epoch: 8 [1/125 256/32000 (1%)] Loss: 3.22660 (QuantReg: 12.38415) QuantErr: 12.38415 batch_time=46.26583
Train Epoch: 8 [17/125 4352/32000 (14%)] Loss: 3.21903 (QuantReg: 12.90864) QuantErr: 12.90864 batch_time=0.76886
Train Epoch: 8 [33/125 8448/32000 (26%)] Loss: 3.15067 (QuantReg: 13.04508) QuantErr: 13.04508 batch_time=0.74818
Train Epoch: 8 [49/125 12544/32000 (39%)] Loss: 3.27221 (QuantReg: 12.74018) QuantErr: 12.74018 batch_time=0.92096
Train Epoch: 8 [65/125 16640/32000 (52%)] Loss: 3.20338 (QuantReg: 13.15736) QuantErr: 13.15736 batch_time=7.28924
Train Epoch: 8 [81/125 20736/32000 (65%)] Loss: 3.46892 (QuantReg: 12.93280) QuantErr: 12.93280 batch_time=0.75319
Train Epoch: 8 [97/125 24832/32000 (78%)] Loss: 2.90897 (QuantReg: 13.39285) QuantErr: 13.39285 batch_time=0.75731
Train Epoch: 8 [113/125 28928/32000 (90%)] Loss: 3.21843 (QuantReg: 13.35448) QuantErr: 13.35448 batch_time=0.82315
Train Epoch: 8 codebook_update_time=1.62879
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch8.pth ...
Done in 3.999s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch8.pth ...
Done in 8.071s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 3.1121463985443114
quant_reg : 13.024272911071778
quant_err : 13.024272911071778
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.462
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.77220772953492
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.8845
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.25215378827555
mnt_best : 39.77220772953492
not_improved_count: 0
Train Epoch: 9 [1/125 256/32000 (1%)] Loss: 2.73673 (QuantReg: 12.95376) QuantErr: 12.95376 batch_time=53.88581
Train Epoch: 9 [17/125 4352/32000 (14%)] Loss: 2.87602 (QuantReg: 13.20700) QuantErr: 13.20700 batch_time=0.75220
Train Epoch: 9 [33/125 8448/32000 (26%)] Loss: 2.86590 (QuantReg: 13.17877) QuantErr: 13.17877 batch_time=0.74632
Train Epoch: 9 [49/125 12544/32000 (39%)] Loss: 2.88650 (QuantReg: 13.07270) QuantErr: 13.07270 batch_time=0.83478
Train Epoch: 9 [65/125 16640/32000 (52%)] Loss: 3.09551 (QuantReg: 13.22497) QuantErr: 13.22497 batch_time=17.58522
Train Epoch: 9 [81/125 20736/32000 (65%)] Loss: 3.14179 (QuantReg: 13.20084) QuantErr: 13.20084 batch_time=0.75881
Train Epoch: 9 [97/125 24832/32000 (78%)] Loss: 2.62844 (QuantReg: 13.44975) QuantErr: 13.44975 batch_time=0.74570
Train Epoch: 9 [113/125 28928/32000 (90%)] Loss: 2.91841 (QuantReg: 13.34747) QuantErr: 13.34747 batch_time=0.76224
Train Epoch: 9 codebook_update_time=1.67060
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch9.pth ...
Done in 3.749s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch9.pth ...
Done in 7.475s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 2.9516396293640135
quant_reg : 13.166921783447266
quant_err : 13.166921783447266
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 1125
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.09
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.48268527755685
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.596
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.37510556640888
mnt_best : 40.48268527755685
not_improved_count: 0
Train Epoch: 10 [1/125 256/32000 (1%)] Loss: 2.66154 (QuantReg: 13.06406) QuantErr: 13.06406 batch_time=37.76255
Train Epoch: 10 [17/125 4352/32000 (14%)] Loss: 2.76634 (QuantReg: 13.09614) QuantErr: 13.09614 batch_time=0.78052
Train Epoch: 10 [33/125 8448/32000 (26%)] Loss: 3.20587 (QuantReg: 13.09699) QuantErr: 13.09699 batch_time=1.10674
Train Epoch: 10 [49/125 12544/32000 (39%)] Loss: 2.92529 (QuantReg: 13.21399) QuantErr: 13.21399 batch_time=0.75742
Train Epoch: 10 [65/125 16640/32000 (52%)] Loss: 2.84602 (QuantReg: 13.10221) QuantErr: 13.10221 batch_time=1.99954
Train Epoch: 10 [81/125 20736/32000 (65%)] Loss: 2.68987 (QuantReg: 13.13257) QuantErr: 13.13257 batch_time=0.75247
Train Epoch: 10 [97/125 24832/32000 (78%)] Loss: 2.87606 (QuantReg: 13.49540) QuantErr: 13.49540 batch_time=0.88060
Train Epoch: 10 [113/125 28928/32000 (90%)] Loss: 2.39163 (QuantReg: 13.30684) QuantErr: 13.30684 batch_time=0.79195
Train Epoch: 10 codebook_update_time=1.66138
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch10.pth ...
Done in 4.947s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch10.pth ...
Done in 9.993s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 2.849422588348389
quant_reg : 13.194122932434082
quant_err : 13.194122932434082
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.766
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.728110943681045
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.7555
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.491035756750776
mnt_best : 40.728110943681045
not_improved_count: 0
Train Epoch: 11 [1/125 256/32000 (1%)] Loss: 2.80282 (QuantReg: 12.86389) QuantErr: 12.86389 batch_time=46.64568
Train Epoch: 11 [17/125 4352/32000 (14%)] Loss: 2.92496 (QuantReg: 13.01901) QuantErr: 13.01901 batch_time=0.75006
Train Epoch: 11 [33/125 8448/32000 (26%)] Loss: 2.52689 (QuantReg: 13.18165) QuantErr: 13.18165 batch_time=0.83231
Train Epoch: 11 [49/125 12544/32000 (39%)] Loss: 2.82682 (QuantReg: 13.08979) QuantErr: 13.08979 batch_time=0.76585
Train Epoch: 11 [65/125 16640/32000 (52%)] Loss: 2.89093 (QuantReg: 13.38518) QuantErr: 13.38518 batch_time=6.55782
Train Epoch: 11 [81/125 20736/32000 (65%)] Loss: 2.63750 (QuantReg: 13.14111) QuantErr: 13.14111 batch_time=0.75138
Train Epoch: 11 [97/125 24832/32000 (78%)] Loss: 2.91965 (QuantReg: 13.09919) QuantErr: 13.09919 batch_time=0.84189
Train Epoch: 11 [113/125 28928/32000 (90%)] Loss: 2.56792 (QuantReg: 13.31429) QuantErr: 13.31429 batch_time=0.82096
Train Epoch: 11 codebook_update_time=1.77519
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch11.pth ...
Done in 5.175s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch11.pth ...
Done in 11.009s
removing stale ckpt [epoch 10] [took 0.03s]
epoch : 11
loss : 2.7412811489105224
quant_reg : 13.224890098571777
quant_err : 13.224890098571777
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 1375
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.491
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.0809864635199
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.648
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.91317380333002
mnt_best : 41.0809864635199
not_improved_count: 0
Train Epoch: 12 [1/125 256/32000 (1%)] Loss: 2.55493 (QuantReg: 13.23746) QuantErr: 13.23746 batch_time=48.81538
Train Epoch: 12 [17/125 4352/32000 (14%)] Loss: 2.68960 (QuantReg: 13.39561) QuantErr: 13.39561 batch_time=1.51875
Train Epoch: 12 [33/125 8448/32000 (26%)] Loss: 2.58000 (QuantReg: 13.42396) QuantErr: 13.42396 batch_time=1.28728
Train Epoch: 12 [49/125 12544/32000 (39%)] Loss: 2.86778 (QuantReg: 13.14293) QuantErr: 13.14293 batch_time=0.75683
Train Epoch: 12 [65/125 16640/32000 (52%)] Loss: 2.66997 (QuantReg: 13.19487) QuantErr: 13.19487 batch_time=10.22465
Train Epoch: 12 [81/125 20736/32000 (65%)] Loss: 2.74792 (QuantReg: 13.11519) QuantErr: 13.11519 batch_time=0.78312
Train Epoch: 12 [97/125 24832/32000 (78%)] Loss: 2.46802 (QuantReg: 13.19385) QuantErr: 13.19385 batch_time=1.24494
Train Epoch: 12 [113/125 28928/32000 (90%)] Loss: 2.71051 (QuantReg: 13.31606) QuantErr: 13.31606 batch_time=0.88271
Train Epoch: 12 codebook_update_time=1.85466
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch12.pth ...
Done in 5.360s
removing stale ckpt [epoch 11] [took 0.13s]
epoch : 12
loss : 2.6172759857177734
quant_reg : 13.284943115234375
quant_err : 13.284943115234375
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.893
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.03280641944853
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.6535
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.982382278052874
mnt_best : 41.0809864635199
not_improved_count: 1
Train Epoch: 13 [1/125 256/32000 (1%)] Loss: 2.74042 (QuantReg: 13.19495) QuantErr: 13.19495 batch_time=55.66265
Train Epoch: 13 [17/125 4352/32000 (14%)] Loss: 2.43176 (QuantReg: 13.15541) QuantErr: 13.15541 batch_time=0.75423
Train Epoch: 13 [33/125 8448/32000 (26%)] Loss: 2.44356 (QuantReg: 13.13887) QuantErr: 13.13887 batch_time=1.01809
Train Epoch: 13 [49/125 12544/32000 (39%)] Loss: 2.60980 (QuantReg: 13.17964) QuantErr: 13.17964 batch_time=0.77481
Train Epoch: 13 [65/125 16640/32000 (52%)] Loss: 2.19902 (QuantReg: 13.52716) QuantErr: 13.52716 batch_time=16.29750
Train Epoch: 13 [81/125 20736/32000 (65%)] Loss: 2.30786 (QuantReg: 13.36950) QuantErr: 13.36950 batch_time=0.77766
Train Epoch: 13 [97/125 24832/32000 (78%)] Loss: 2.28373 (QuantReg: 13.20085) QuantErr: 13.20085 batch_time=1.02691
Train Epoch: 13 [113/125 28928/32000 (90%)] Loss: 2.67598 (QuantReg: 13.45185) QuantErr: 13.45185 batch_time=0.94074
Train Epoch: 13 codebook_update_time=1.70998
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch13.pth ...
Done in 5.275s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 2.5465402507781985
quant_reg : 13.354731651306153
quant_err : 13.354731651306153
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 1625
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.505
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.3776278774056
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.561
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.752703997361365
mnt_best : 41.0809864635199
not_improved_count: 2
Train Epoch: 14 [1/125 256/32000 (1%)] Loss: 2.27782 (QuantReg: 13.58788) QuantErr: 13.58788 batch_time=45.92857
Train Epoch: 14 [17/125 4352/32000 (14%)] Loss: 2.47978 (QuantReg: 13.09658) QuantErr: 13.09658 batch_time=0.78354
Train Epoch: 14 [33/125 8448/32000 (26%)] Loss: 2.61238 (QuantReg: 13.50651) QuantErr: 13.50651 batch_time=2.11738
Train Epoch: 14 [49/125 12544/32000 (39%)] Loss: 2.53085 (QuantReg: 13.19083) QuantErr: 13.19083 batch_time=0.75566
Train Epoch: 14 [65/125 16640/32000 (52%)] Loss: 2.37278 (QuantReg: 13.54847) QuantErr: 13.54847 batch_time=5.22456
Train Epoch: 14 [81/125 20736/32000 (65%)] Loss: 2.14955 (QuantReg: 13.39153) QuantErr: 13.39153 batch_time=0.76668
Train Epoch: 14 [97/125 24832/32000 (78%)] Loss: 2.35139 (QuantReg: 13.42149) QuantErr: 13.42149 batch_time=2.21766
Train Epoch: 14 [113/125 28928/32000 (90%)] Loss: 2.52338 (QuantReg: 13.54542) QuantErr: 13.54542 batch_time=1.00241
Train Epoch: 14 codebook_update_time=1.86001
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch14.pth ...
Done in 5.329s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch14.pth ...
Done in 10.263s
removing stale ckpt [epoch 13] [took 0.16s]
epoch : 14
loss : 2.4709984607696533
quant_reg : 13.389761093139649
quant_err : 13.389761093139649
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.253
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.71906954067978
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.8875
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.56535885888369
mnt_best : 41.71906954067978
not_improved_count: 0
Train Epoch: 15 [1/125 256/32000 (1%)] Loss: 2.30085 (QuantReg: 13.33228) QuantErr: 13.33228 batch_time=65.85663
Train Epoch: 15 [17/125 4352/32000 (14%)] Loss: 2.55275 (QuantReg: 13.19432) QuantErr: 13.19432 batch_time=0.74663
Train Epoch: 15 [33/125 8448/32000 (26%)] Loss: 2.04793 (QuantReg: 13.43840) QuantErr: 13.43840 batch_time=0.76169
Train Epoch: 15 [49/125 12544/32000 (39%)] Loss: 2.35929 (QuantReg: 13.58166) QuantErr: 13.58166 batch_time=0.74338
Train Epoch: 15 [65/125 16640/32000 (52%)] Loss: 2.53458 (QuantReg: 13.62448) QuantErr: 13.62448 batch_time=18.34048
Train Epoch: 15 [81/125 20736/32000 (65%)] Loss: 2.51677 (QuantReg: 13.37210) QuantErr: 13.37210 batch_time=0.78194
Train Epoch: 15 [97/125 24832/32000 (78%)] Loss: 2.46177 (QuantReg: 13.60930) QuantErr: 13.60930 batch_time=0.79236
Train Epoch: 15 [113/125 28928/32000 (90%)] Loss: 2.06730 (QuantReg: 13.79082) QuantErr: 13.79082 batch_time=0.75006
Train Epoch: 15 codebook_update_time=1.77127
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch15.pth ...
Done in 4.362s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch15.pth ...
Done in 8.902s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 2.4321523580551148
quant_reg : 13.445170486450195
quant_err : 13.445170486450195
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 1875
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.146
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.203456325399195
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.132
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.35685221653413
mnt_best : 42.203456325399195
not_improved_count: 0
Train Epoch: 16 [1/125 256/32000 (1%)] Loss: 2.64770 (QuantReg: 13.36431) QuantErr: 13.36431 batch_time=54.01168
Train Epoch: 16 [17/125 4352/32000 (14%)] Loss: 2.58541 (QuantReg: 13.01781) QuantErr: 13.01781 batch_time=0.88948
Train Epoch: 16 [33/125 8448/32000 (26%)] Loss: 2.25600 (QuantReg: 13.57706) QuantErr: 13.57706 batch_time=0.77231
Train Epoch: 16 [49/125 12544/32000 (39%)] Loss: 2.55840 (QuantReg: 13.56935) QuantErr: 13.56935 batch_time=0.76040
Train Epoch: 16 [65/125 16640/32000 (52%)] Loss: 2.32697 (QuantReg: 13.54307) QuantErr: 13.54307 batch_time=8.73092
Train Epoch: 16 [81/125 20736/32000 (65%)] Loss: 2.25019 (QuantReg: 13.72236) QuantErr: 13.72236 batch_time=0.75744
Train Epoch: 16 [97/125 24832/32000 (78%)] Loss: 2.50120 (QuantReg: 13.55775) QuantErr: 13.55775 batch_time=0.73432
Train Epoch: 16 [113/125 28928/32000 (90%)] Loss: 2.50146 (QuantReg: 13.59549) QuantErr: 13.59549 batch_time=0.74200
Train Epoch: 16 codebook_update_time=1.67280
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch16.pth ...
Done in 5.030s
removing stale ckpt [epoch 15] [took 0.02s]
epoch : 16
loss : 2.3919539937973022
quant_reg : 13.497487312316894
quant_err : 13.497487312316894
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.451
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.057065287574105
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.671
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.44154812711669
mnt_best : 42.203456325399195
not_improved_count: 1
Train Epoch: 17 [1/125 256/32000 (1%)] Loss: 2.28034 (QuantReg: 13.53736) QuantErr: 13.53736 batch_time=36.94950
Train Epoch: 17 [17/125 4352/32000 (14%)] Loss: 2.28003 (QuantReg: 13.71474) QuantErr: 13.71474 batch_time=0.74891
Train Epoch: 17 [33/125 8448/32000 (26%)] Loss: 2.28011 (QuantReg: 13.50168) QuantErr: 13.50168 batch_time=0.76445
Train Epoch: 17 [49/125 12544/32000 (39%)] Loss: 2.17158 (QuantReg: 13.53873) QuantErr: 13.53873 batch_time=0.86051
Train Epoch: 17 [65/125 16640/32000 (52%)] Loss: 2.24796 (QuantReg: 13.87566) QuantErr: 13.87566 batch_time=0.74761
Train Epoch: 17 [81/125 20736/32000 (65%)] Loss: 2.42896 (QuantReg: 13.55810) QuantErr: 13.55810 batch_time=0.80704
Train Epoch: 17 [97/125 24832/32000 (78%)] Loss: 2.36684 (QuantReg: 13.74431) QuantErr: 13.74431 batch_time=0.91539
Train Epoch: 17 [113/125 28928/32000 (90%)] Loss: 2.45648 (QuantReg: 13.31039) QuantErr: 13.31039 batch_time=0.85620
Train Epoch: 17 codebook_update_time=2.38039
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch17.pth ...
Done in 5.852s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch17.pth ...
Done in 12.353s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.325145471572876
quant_reg : 13.532592468261718
quant_err : 13.532592468261718
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 2125
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.334
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.25520130169544
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.889
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.90404002356713
mnt_best : 42.25520130169544
not_improved_count: 0
Train Epoch: 18 [1/125 256/32000 (1%)] Loss: 2.39643 (QuantReg: 13.16110) QuantErr: 13.16110 batch_time=47.82382
Train Epoch: 18 [17/125 4352/32000 (14%)] Loss: 2.51984 (QuantReg: 13.51346) QuantErr: 13.51346 batch_time=0.75654
Train Epoch: 18 [33/125 8448/32000 (26%)] Loss: 2.05286 (QuantReg: 13.41271) QuantErr: 13.41271 batch_time=0.75505
Train Epoch: 18 [49/125 12544/32000 (39%)] Loss: 2.23348 (QuantReg: 13.47570) QuantErr: 13.47570 batch_time=0.91650
Train Epoch: 18 [65/125 16640/32000 (52%)] Loss: 2.38817 (QuantReg: 13.49854) QuantErr: 13.49854 batch_time=6.00702
Train Epoch: 18 [81/125 20736/32000 (65%)] Loss: 2.15119 (QuantReg: 13.68169) QuantErr: 13.68169 batch_time=0.84022
Train Epoch: 18 [97/125 24832/32000 (78%)] Loss: 2.27120 (QuantReg: 13.48450) QuantErr: 13.48450 batch_time=0.93627
Train Epoch: 18 [113/125 28928/32000 (90%)] Loss: 2.18489 (QuantReg: 13.57592) QuantErr: 13.57592 batch_time=0.85324
Train Epoch: 18 codebook_update_time=1.63640
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch18.pth ...
Done in 5.099s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch18.pth ...
Done in 10.014s
removing stale ckpt [epoch 17] [took 0.23s]
epoch : 18
loss : 2.26050780582428
quant_reg : 13.540499671936034
quant_err : 13.540499671936034
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.281
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.01394608034441
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.1045
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.09861089437567
mnt_best : 43.01394608034441
not_improved_count: 0
Train Epoch: 19 [1/125 256/32000 (1%)] Loss: 2.23387 (QuantReg: 13.69227) QuantErr: 13.69227 batch_time=51.53053
Train Epoch: 19 [17/125 4352/32000 (14%)] Loss: 2.26368 (QuantReg: 13.68560) QuantErr: 13.68560 batch_time=3.81061
Train Epoch: 19 [33/125 8448/32000 (26%)] Loss: 1.96937 (QuantReg: 13.80547) QuantErr: 13.80547 batch_time=1.01737
Train Epoch: 19 [49/125 12544/32000 (39%)] Loss: 2.15830 (QuantReg: 13.60329) QuantErr: 13.60329 batch_time=0.75684
Train Epoch: 19 [65/125 16640/32000 (52%)] Loss: 2.47212 (QuantReg: 13.36186) QuantErr: 13.36186 batch_time=6.68474
Train Epoch: 19 [81/125 20736/32000 (65%)] Loss: 2.28046 (QuantReg: 13.60723) QuantErr: 13.60723 batch_time=3.34255
Train Epoch: 19 [97/125 24832/32000 (78%)] Loss: 2.14118 (QuantReg: 13.76014) QuantErr: 13.76014 batch_time=0.94952
Train Epoch: 19 [113/125 28928/32000 (90%)] Loss: 2.17931 (QuantReg: 13.59145) QuantErr: 13.59145 batch_time=0.87809
Train Epoch: 19 codebook_update_time=1.66252
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch19.pth ...
Done in 4.381s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 2.235119789123535
quant_reg : 13.602435302734374
quant_err : 13.602435302734374
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 2375
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.033
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.82595232155793
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.6695
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.00839784611977
mnt_best : 43.01394608034441
not_improved_count: 1
Train Epoch: 20 [1/125 256/32000 (1%)] Loss: 2.05597 (QuantReg: 13.86726) QuantErr: 13.86726 batch_time=51.88018
Train Epoch: 20 [17/125 4352/32000 (14%)] Loss: 2.36028 (QuantReg: 13.47959) QuantErr: 13.47959 batch_time=0.76106
Train Epoch: 20 [33/125 8448/32000 (26%)] Loss: 2.43594 (QuantReg: 13.64219) QuantErr: 13.64219 batch_time=0.75707
Train Epoch: 20 [49/125 12544/32000 (39%)] Loss: 2.16760 (QuantReg: 13.71751) QuantErr: 13.71751 batch_time=0.75839
Train Epoch: 20 [65/125 16640/32000 (52%)] Loss: 2.27383 (QuantReg: 13.68169) QuantErr: 13.68169 batch_time=9.14413
Train Epoch: 20 [81/125 20736/32000 (65%)] Loss: 2.44434 (QuantReg: 13.54818) QuantErr: 13.54818 batch_time=0.89289
Train Epoch: 20 [97/125 24832/32000 (78%)] Loss: 1.84926 (QuantReg: 13.43291) QuantErr: 13.43291 batch_time=0.89206
Train Epoch: 20 [113/125 28928/32000 (90%)] Loss: 2.27553 (QuantReg: 13.73527) QuantErr: 13.73527 batch_time=0.75308
Train Epoch: 20 codebook_update_time=1.67893
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch20.pth ...
Done in 5.002s
removing stale ckpt [epoch 19] [took 0.01s]
epoch : 20
loss : 2.2057833862304688
quant_reg : 13.616562995910645
quant_err : 13.616562995910645
learning_rate : 1.8867680126765363e-05
n_samples : 640000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.996
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.866731433390775
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.7155
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.448837714337735
mnt_best : 43.01394608034441
not_improved_count: 2
Train Epoch: 21 [1/125 256/32000 (1%)] Loss: 2.27857 (QuantReg: 13.69720) QuantErr: 13.69720 batch_time=41.56980
Train Epoch: 21 [17/125 4352/32000 (14%)] Loss: 1.91918 (QuantReg: 13.41325) QuantErr: 13.41325 batch_time=7.52535
Train Epoch: 21 [33/125 8448/32000 (26%)] Loss: 1.97792 (QuantReg: 13.30781) QuantErr: 13.30781 batch_time=0.95541
Train Epoch: 21 [49/125 12544/32000 (39%)] Loss: 2.29834 (QuantReg: 13.40190) QuantErr: 13.40190 batch_time=0.92751
Train Epoch: 21 [65/125 16640/32000 (52%)] Loss: 2.01503 (QuantReg: 13.74132) QuantErr: 13.74132 batch_time=1.27645
Train Epoch: 21 [81/125 20736/32000 (65%)] Loss: 2.19823 (QuantReg: 13.74048) QuantErr: 13.74048 batch_time=5.52607
Train Epoch: 21 [97/125 24832/32000 (78%)] Loss: 2.07479 (QuantReg: 13.78504) QuantErr: 13.78504 batch_time=0.75899
Train Epoch: 21 [113/125 28928/32000 (90%)] Loss: 2.27287 (QuantReg: 13.73755) QuantErr: 13.73755 batch_time=0.74377
Train Epoch: 21 codebook_update_time=1.72130
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch21.pth ...
Done in 6.998s
removing stale ckpt [epoch 20] [took 0.01s]
epoch : 21
loss : 2.1600681066513063
quant_reg : 13.638440368652343
quant_err : 13.638440368652343
learning_rate : 1.7924296120427095e-05
n_samples : 672000
n_steps : 2625
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.396
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.990342161271734
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.973
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.067382980806556
mnt_best : 43.01394608034441
not_improved_count: 3
Train Epoch: 22 [1/125 256/32000 (1%)] Loss: 2.40051 (QuantReg: 13.60661) QuantErr: 13.60661 batch_time=52.92273
Train Epoch: 22 [17/125 4352/32000 (14%)] Loss: 2.16685 (QuantReg: 13.56999) QuantErr: 13.56999 batch_time=0.93857
Train Epoch: 22 [33/125 8448/32000 (26%)] Loss: 2.08345 (QuantReg: 13.63505) QuantErr: 13.63505 batch_time=0.97438
Train Epoch: 22 [49/125 12544/32000 (39%)] Loss: 2.25411 (QuantReg: 13.63555) QuantErr: 13.63555 batch_time=0.97065
Train Epoch: 22 [65/125 16640/32000 (52%)] Loss: 2.02546 (QuantReg: 13.71772) QuantErr: 13.71772 batch_time=9.33312
Train Epoch: 22 [81/125 20736/32000 (65%)] Loss: 1.90465 (QuantReg: 13.83156) QuantErr: 13.83156 batch_time=0.76796
Train Epoch: 22 [97/125 24832/32000 (78%)] Loss: 2.32626 (QuantReg: 13.68186) QuantErr: 13.68186 batch_time=0.75297
Train Epoch: 22 [113/125 28928/32000 (90%)] Loss: 1.80371 (QuantReg: 13.89478) QuantErr: 13.89478 batch_time=0.74063
Train Epoch: 22 codebook_update_time=1.85167
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch22.pth ...
Done in 4.466s
removing stale ckpt [epoch 21] [took 0.01s]
epoch : 22
loss : 2.130122130393982
quant_reg : 13.644888298034669
quant_err : 13.644888298034669
learning_rate : 1.702808131440574e-05
n_samples : 704000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.788
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.19606740679588
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.2085
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.96688072440102
mnt_best : 43.01394608034441
not_improved_count: 4
Train Epoch: 23 [1/125 256/32000 (1%)] Loss: 1.98128 (QuantReg: 13.72740) QuantErr: 13.72740 batch_time=48.59527
Train Epoch: 23 [17/125 4352/32000 (14%)] Loss: 1.88667 (QuantReg: 13.72806) QuantErr: 13.72806 batch_time=0.73947
Train Epoch: 23 [33/125 8448/32000 (26%)] Loss: 2.27047 (QuantReg: 13.83532) QuantErr: 13.83532 batch_time=0.74698
Train Epoch: 23 [49/125 12544/32000 (39%)] Loss: 2.29494 (QuantReg: 13.76962) QuantErr: 13.76962 batch_time=0.77444
Train Epoch: 23 [65/125 16640/32000 (52%)] Loss: 2.16455 (QuantReg: 13.99452) QuantErr: 13.99452 batch_time=3.96163
Train Epoch: 23 [81/125 20736/32000 (65%)] Loss: 2.28354 (QuantReg: 13.66294) QuantErr: 13.66294 batch_time=1.43473
Train Epoch: 23 [97/125 24832/32000 (78%)] Loss: 2.31905 (QuantReg: 13.64249) QuantErr: 13.64249 batch_time=0.86634
Train Epoch: 23 [113/125 28928/32000 (90%)] Loss: 2.22036 (QuantReg: 13.62432) QuantErr: 13.62432 batch_time=0.74102
Train Epoch: 23 codebook_update_time=1.66718
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch23.pth ...
Done in 6.497s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch23.pth ...
Done in 11.573s
removing stale ckpt [epoch 22] [took 0.01s]
epoch : 23
loss : 2.116029992103577
quant_reg : 13.705629737854004
quant_err : 13.705629737854004
learning_rate : 1.6176677248685452e-05
n_samples : 736000
n_steps : 2875
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.426
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.58974732972494
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.888
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.86031880864897
mnt_best : 43.58974732972494
not_improved_count: 0
Train Epoch: 24 [1/125 256/32000 (1%)] Loss: 2.03641 (QuantReg: 13.75127) QuantErr: 13.75127 batch_time=47.65060
Train Epoch: 24 [17/125 4352/32000 (14%)] Loss: 1.85670 (QuantReg: 13.70551) QuantErr: 13.70551 batch_time=0.76218
Train Epoch: 24 [33/125 8448/32000 (26%)] Loss: 2.24002 (QuantReg: 13.66437) QuantErr: 13.66437 batch_time=0.75268
Train Epoch: 24 [49/125 12544/32000 (39%)] Loss: 1.97995 (QuantReg: 13.80650) QuantErr: 13.80650 batch_time=0.87137
Train Epoch: 24 [65/125 16640/32000 (52%)] Loss: 2.02192 (QuantReg: 13.80353) QuantErr: 13.80353 batch_time=9.40578
Train Epoch: 24 [81/125 20736/32000 (65%)] Loss: 1.99078 (QuantReg: 13.71151) QuantErr: 13.71151 batch_time=0.92933
Train Epoch: 24 [97/125 24832/32000 (78%)] Loss: 2.15788 (QuantReg: 13.52409) QuantErr: 13.52409 batch_time=0.77475
Train Epoch: 24 [113/125 28928/32000 (90%)] Loss: 2.07992 (QuantReg: 13.78923) QuantErr: 13.78923 batch_time=0.72912
Train Epoch: 24 codebook_update_time=1.74064
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch24.pth ...
Done in 5.659s
removing stale ckpt [epoch 23] [took 0.01s]
epoch : 24
loss : 2.0664731254577635
quant_reg : 13.728834701538085
quant_err : 13.728834701538085
learning_rate : 1.5367843386251178e-05
n_samples : 768000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.944
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.93612510190617
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.657
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.421054909525225
mnt_best : 43.58974732972494
not_improved_count: 1
Train Epoch: 25 [1/125 256/32000 (1%)] Loss: 2.11828 (QuantReg: 13.44857) QuantErr: 13.44857 batch_time=42.31836
Train Epoch: 25 [17/125 4352/32000 (14%)] Loss: 2.14122 (QuantReg: 13.58167) QuantErr: 13.58167 batch_time=0.77997
Train Epoch: 25 [33/125 8448/32000 (26%)] Loss: 1.89318 (QuantReg: 13.83132) QuantErr: 13.83132 batch_time=0.90484
Train Epoch: 25 [49/125 12544/32000 (39%)] Loss: 1.86512 (QuantReg: 13.67218) QuantErr: 13.67218 batch_time=0.75089
Train Epoch: 25 [65/125 16640/32000 (52%)] Loss: 1.94617 (QuantReg: 13.81431) QuantErr: 13.81431 batch_time=3.49714
Train Epoch: 25 [81/125 20736/32000 (65%)] Loss: 2.03460 (QuantReg: 13.88055) QuantErr: 13.88055 batch_time=0.74344
Train Epoch: 25 [97/125 24832/32000 (78%)] Loss: 1.99960 (QuantReg: 13.64193) QuantErr: 13.64193 batch_time=0.77424
Train Epoch: 25 [113/125 28928/32000 (90%)] Loss: 1.94388 (QuantReg: 13.87327) QuantErr: 13.87327 batch_time=0.74706
Train Epoch: 25 codebook_update_time=1.76402
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch25.pth ...
Done in 4.107s
removing stale ckpt [epoch 24] [took 0.01s]
epoch : 25
loss : 2.026697339057922
quant_reg : 13.728133735656739
quant_err : 13.728133735656739
learning_rate : 1.4599451216938618e-05
n_samples : 800000
n_steps : 3125
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.349
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.31521030454747
MSRVTT_jsfusion_test/v2t_metrics/R1: 25.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.255
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.48177831727137
mnt_best : 43.58974732972494
not_improved_count: 2
Train Epoch: 26 [1/125 256/32000 (1%)] Loss: 2.12863 (QuantReg: 13.68475) QuantErr: 13.68475 batch_time=34.34451
Train Epoch: 26 [17/125 4352/32000 (14%)] Loss: 2.16089 (QuantReg: 13.73365) QuantErr: 13.73365 batch_time=0.74461
Train Epoch: 26 [33/125 8448/32000 (26%)] Loss: 2.34294 (QuantReg: 13.58987) QuantErr: 13.58987 batch_time=1.01248
Train Epoch: 26 [49/125 12544/32000 (39%)] Loss: 1.91326 (QuantReg: 13.83023) QuantErr: 13.83023 batch_time=0.75108
Train Epoch: 26 [65/125 16640/32000 (52%)] Loss: 2.17730 (QuantReg: 13.79836) QuantErr: 13.79836 batch_time=1.43690
Train Epoch: 26 [81/125 20736/32000 (65%)] Loss: 1.92469 (QuantReg: 13.65987) QuantErr: 13.65987 batch_time=0.76525
Train Epoch: 26 [97/125 24832/32000 (78%)] Loss: 2.15192 (QuantReg: 13.79074) QuantErr: 13.79074 batch_time=0.78889
Train Epoch: 26 [113/125 28928/32000 (90%)] Loss: 1.94903 (QuantReg: 13.75730) QuantErr: 13.75730 batch_time=0.74488
Train Epoch: 26 codebook_update_time=1.71413
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch26.pth ...
Done in 25.053s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch26.pth ...
Done in 30.489s
removing stale ckpt [epoch 25] [took 0.03s]
epoch : 26
loss : 2.0662539710998535
quant_reg : 13.732690795898437
quant_err : 13.732690795898437
learning_rate : 1.3869478656091687e-05
n_samples : 832000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.44
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.63019486963897
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.601
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.24915017278041
mnt_best : 43.63019486963897
not_improved_count: 0
Train Epoch: 27 [1/125 256/32000 (1%)] Loss: 2.00068 (QuantReg: 13.69935) QuantErr: 13.69935 batch_time=42.26766
Train Epoch: 27 [17/125 4352/32000 (14%)] Loss: 2.04031 (QuantReg: 13.86800) QuantErr: 13.86800 batch_time=0.79041
Train Epoch: 27 [33/125 8448/32000 (26%)] Loss: 1.85370 (QuantReg: 14.00690) QuantErr: 14.00690 batch_time=0.74007
Train Epoch: 27 [49/125 12544/32000 (39%)] Loss: 1.87281 (QuantReg: 13.99162) QuantErr: 13.99162 batch_time=0.80312
Train Epoch: 27 [65/125 16640/32000 (52%)] Loss: 2.08946 (QuantReg: 13.78791) QuantErr: 13.78791 batch_time=6.17820
Train Epoch: 27 [81/125 20736/32000 (65%)] Loss: 1.74572 (QuantReg: 13.99887) QuantErr: 13.99887 batch_time=0.90571
Train Epoch: 27 [97/125 24832/32000 (78%)] Loss: 2.02532 (QuantReg: 13.64239) QuantErr: 13.64239 batch_time=0.73298
Train Epoch: 27 [113/125 28928/32000 (90%)] Loss: 2.18367 (QuantReg: 13.65386) QuantErr: 13.65386 batch_time=0.75822
Train Epoch: 27 codebook_update_time=1.72213
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch27.pth ...
Done in 4.874s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs256/checkpoint-epoch27.pth ...
Done in 10.298s
removing stale ckpt [epoch 26] [took 0.01s]
epoch : 27
loss : 2.018105319023132
quant_reg : 13.784707397460938
quant_err : 13.784707397460938
learning_rate : 1.3176004723287102e-05
n_samples : 864000
n_steps : 3375
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.5