-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_M16.txt
2599 lines (2599 loc) · 192 KB
/
HCQ_MSRVTT_1kA_M16.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 977.1309831142426 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 88.7695951461792 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 80.00765323638916 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch0.pth ...
Done in 1.859s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch0.pth ...
Done in 3.755s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 1.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 512.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 499.287
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3684031498640387
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 4.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 496.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 496.402
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.3684031498640387
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.82110 (QuantReg: 16.73714) QuantErr: 16.73714 batch_time=25.04498
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.64018 (QuantReg: 16.76336) QuantErr: 16.76336 batch_time=0.41023
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.61753 (QuantReg: 16.66730) QuantErr: 16.66730 batch_time=0.46857
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.30028 (QuantReg: 16.69251) QuantErr: 16.69251 batch_time=0.48090
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.44799 (QuantReg: 16.70835) QuantErr: 16.70835 batch_time=0.44847
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.71118 (QuantReg: 16.71692) QuantErr: 16.71692 batch_time=0.44599
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.87846 (QuantReg: 16.70773) QuantErr: 16.70773 batch_time=0.48156
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 4.92653 (QuantReg: 16.72494) QuantErr: 16.72494 batch_time=0.46949
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.49199 (QuantReg: 16.71083) QuantErr: 16.71083 batch_time=0.45686
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.31307 (QuantReg: 16.70510) QuantErr: 16.70510 batch_time=0.46504
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.64775 (QuantReg: 16.73609) QuantErr: 16.73609 batch_time=0.48284
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.68742 (QuantReg: 16.73212) QuantErr: 16.73212 batch_time=0.45341
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.11954 (QuantReg: 16.72249) QuantErr: 16.72249 batch_time=0.45579
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.38143 (QuantReg: 16.70942) QuantErr: 16.70942 batch_time=0.45342
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.68684 (QuantReg: 16.73557) QuantErr: 16.73557 batch_time=0.45364
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.08358 (QuantReg: 16.72569) QuantErr: 16.72569 batch_time=0.44574
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.90267 (QuantReg: 16.72654) QuantErr: 16.72654 batch_time=0.44444
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.51615 (QuantReg: 16.70028) QuantErr: 16.70028 batch_time=0.45657
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.12579 (QuantReg: 16.71182) QuantErr: 16.71182 batch_time=0.45446
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.47296 (QuantReg: 16.72617) QuantErr: 16.72617 batch_time=0.46555
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.59700 (QuantReg: 16.72669) QuantErr: 16.72669 batch_time=0.48669
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.19290 (QuantReg: 16.70918) QuantErr: 16.70918 batch_time=0.47272
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.73315 (QuantReg: 16.72744) QuantErr: 16.72744 batch_time=0.46275
Train Epoch: 1 codebook_update_time=1.00416
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch1.pth ...
Done in 4.473s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch1.pth ...
Done in 15.655s
epoch : 1
loss : 5.360958498001098
quant_reg : 16.717606704711915
quant_err : 16.717606704711915
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 9.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 30.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 43.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 77.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 45.427
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.981782676089704
MSRVTT_jsfusion_test/v2t_metrics/R1: 9.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 30.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 43.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 77.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 14.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 43.6735
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.01593099180736
mnt_best : 22.981782676089704
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.17856 (QuantReg: 7.07010) QuantErr: 7.07010 batch_time=36.35618
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.05235 (QuantReg: 7.23058) QuantErr: 7.23058 batch_time=0.47386
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.06060 (QuantReg: 7.51461) QuantErr: 7.51461 batch_time=0.69575
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.77545 (QuantReg: 7.39475) QuantErr: 7.39475 batch_time=0.44706
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.79259 (QuantReg: 7.78440) QuantErr: 7.78440 batch_time=0.73004
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.96451 (QuantReg: 7.73180) QuantErr: 7.73180 batch_time=0.52564
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.72007 (QuantReg: 7.54142) QuantErr: 7.54142 batch_time=0.44315
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.75541 (QuantReg: 7.92179) QuantErr: 7.92179 batch_time=0.44543
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.65107 (QuantReg: 7.97263) QuantErr: 7.97263 batch_time=0.44465
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.27989 (QuantReg: 7.87211) QuantErr: 7.87211 batch_time=0.44308
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.68374 (QuantReg: 7.72146) QuantErr: 7.72146 batch_time=0.47854
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.21620 (QuantReg: 8.11856) QuantErr: 8.11856 batch_time=0.48741
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.59587 (QuantReg: 8.23462) QuantErr: 8.23462 batch_time=0.48379
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.99033 (QuantReg: 8.26824) QuantErr: 8.26824 batch_time=3.33394
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.96042 (QuantReg: 8.38836) QuantErr: 8.38836 batch_time=0.49218
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.45107 (QuantReg: 8.55118) QuantErr: 8.55118 batch_time=0.45411
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.78874 (QuantReg: 8.43669) QuantErr: 8.43669 batch_time=0.47990
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.60184 (QuantReg: 8.77041) QuantErr: 8.77041 batch_time=0.47254
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.29987 (QuantReg: 8.86511) QuantErr: 8.86511 batch_time=1.68164
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.32591 (QuantReg: 8.89922) QuantErr: 8.89922 batch_time=0.44211
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.47852 (QuantReg: 8.77727) QuantErr: 8.77727 batch_time=0.47104
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.51597 (QuantReg: 8.67657) QuantErr: 8.67657 batch_time=0.44402
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.78719 (QuantReg: 8.81221) QuantErr: 8.81221 batch_time=0.47254
Train Epoch: 2 codebook_update_time=0.90375
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch2.pth ...
Done in 4.266s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch2.pth ...
Done in 8.580s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 3.6905451164245604
quant_reg : 8.11965069580078
quant_err : 8.11965069580078
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 36.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 81.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 10.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 37.8535
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.18829720804664
MSRVTT_jsfusion_test/v2t_metrics/R1: 12.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 36.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 80.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 10.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.4735
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.42591283678866
mnt_best : 28.18829720804664
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.25432 (QuantReg: 7.30566) QuantErr: 7.30566 batch_time=36.50067
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.16206 (QuantReg: 7.41762) QuantErr: 7.41762 batch_time=0.43615
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.29178 (QuantReg: 7.46557) QuantErr: 7.46557 batch_time=0.44143
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.85109 (QuantReg: 7.36310) QuantErr: 7.36310 batch_time=1.40228
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.89033 (QuantReg: 7.54158) QuantErr: 7.54158 batch_time=0.46383
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.40687 (QuantReg: 7.37454) QuantErr: 7.37454 batch_time=0.44363
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.60881 (QuantReg: 7.72677) QuantErr: 7.72677 batch_time=0.45282
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.46353 (QuantReg: 7.72316) QuantErr: 7.72316 batch_time=0.49152
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.46972 (QuantReg: 7.50461) QuantErr: 7.50461 batch_time=0.46765
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.62161 (QuantReg: 7.81884) QuantErr: 7.81884 batch_time=0.43859
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.44649 (QuantReg: 7.69655) QuantErr: 7.69655 batch_time=0.44095
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.18363 (QuantReg: 7.61112) QuantErr: 7.61112 batch_time=0.44695
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.86042 (QuantReg: 7.54716) QuantErr: 7.54716 batch_time=0.43549
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.31035 (QuantReg: 7.54675) QuantErr: 7.54675 batch_time=0.43824
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.55728 (QuantReg: 7.36962) QuantErr: 7.36962 batch_time=0.45293
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.76955 (QuantReg: 7.68143) QuantErr: 7.68143 batch_time=0.43993
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.23213 (QuantReg: 8.05326) QuantErr: 8.05326 batch_time=0.43910
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.87656 (QuantReg: 7.97909) QuantErr: 7.97909 batch_time=0.43559
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.95212 (QuantReg: 7.85022) QuantErr: 7.85022 batch_time=0.47395
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.08267 (QuantReg: 7.86292) QuantErr: 7.86292 batch_time=0.44677
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.04164 (QuantReg: 7.94750) QuantErr: 7.94750 batch_time=0.46298
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.04666 (QuantReg: 8.13113) QuantErr: 8.13113 batch_time=0.47259
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.12181 (QuantReg: 8.09111) QuantErr: 8.09111 batch_time=0.44789
Train Epoch: 3 codebook_update_time=0.92569
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch3.pth ...
Done in 4.784s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch3.pth ...
Done in 9.064s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.1337384243011472
quant_reg : 7.684354175567627
quant_err : 7.684354175567627
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 54.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.83
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.79785221431743
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.506
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.782926636572196
mnt_best : 30.79785221431743
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.98140 (QuantReg: 7.51392) QuantErr: 7.51392 batch_time=32.90996
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.73426 (QuantReg: 7.46782) QuantErr: 7.46782 batch_time=0.45289
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.88312 (QuantReg: 7.52713) QuantErr: 7.52713 batch_time=0.48684
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.69430 (QuantReg: 7.61402) QuantErr: 7.61402 batch_time=0.44551
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.72143 (QuantReg: 7.63271) QuantErr: 7.63271 batch_time=0.44826
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.98000 (QuantReg: 7.49240) QuantErr: 7.49240 batch_time=0.48610
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.56225 (QuantReg: 7.38327) QuantErr: 7.38327 batch_time=0.45242
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.78896 (QuantReg: 7.60373) QuantErr: 7.60373 batch_time=0.61568
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.03004 (QuantReg: 7.44152) QuantErr: 7.44152 batch_time=0.45108
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.47788 (QuantReg: 7.60495) QuantErr: 7.60495 batch_time=0.46416
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.71440 (QuantReg: 7.98534) QuantErr: 7.98534 batch_time=0.47555
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.53080 (QuantReg: 8.11319) QuantErr: 8.11319 batch_time=0.44849
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.98139 (QuantReg: 7.81144) QuantErr: 7.81144 batch_time=0.44864
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.59754 (QuantReg: 7.82784) QuantErr: 7.82784 batch_time=1.99813
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.62863 (QuantReg: 7.82190) QuantErr: 7.82190 batch_time=0.44527
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.84348 (QuantReg: 7.83703) QuantErr: 7.83703 batch_time=0.46181
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.76617 (QuantReg: 7.74640) QuantErr: 7.74640 batch_time=0.45635
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.80449 (QuantReg: 7.92095) QuantErr: 7.92095 batch_time=0.48159
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.57138 (QuantReg: 8.25018) QuantErr: 8.25018 batch_time=0.44430
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.41443 (QuantReg: 7.79298) QuantErr: 7.79298 batch_time=0.46127
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.37676 (QuantReg: 8.25281) QuantErr: 8.25281 batch_time=1.17584
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.19118 (QuantReg: 8.26304) QuantErr: 8.26304 batch_time=0.45785
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.84503 (QuantReg: 7.92077) QuantErr: 7.92077 batch_time=0.46952
Train Epoch: 4 codebook_update_time=0.92037
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch4.pth ...
Done in 4.190s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch4.pth ...
Done in 8.467s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 2.8401441869735717
quant_reg : 7.714178134918213
quant_err : 7.714178134918213
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.199
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.91697402337231
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.0345
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.19023276818406
mnt_best : 32.91697402337231
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.84343 (QuantReg: 7.40473) QuantErr: 7.40473 batch_time=36.74361
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.08474 (QuantReg: 7.73324) QuantErr: 7.73324 batch_time=0.48235
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.22477 (QuantReg: 7.42276) QuantErr: 7.42276 batch_time=0.46293
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.52743 (QuantReg: 7.74114) QuantErr: 7.74114 batch_time=0.44673
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.70244 (QuantReg: 7.59656) QuantErr: 7.59656 batch_time=0.58051
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.40621 (QuantReg: 7.85106) QuantErr: 7.85106 batch_time=0.71917
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.71777 (QuantReg: 7.57189) QuantErr: 7.57189 batch_time=0.57021
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.07805 (QuantReg: 7.42643) QuantErr: 7.42643 batch_time=0.50055
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.48010 (QuantReg: 7.79645) QuantErr: 7.79645 batch_time=0.49287
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.62184 (QuantReg: 7.65526) QuantErr: 7.65526 batch_time=0.48636
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.64139 (QuantReg: 7.84625) QuantErr: 7.84625 batch_time=0.45434
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.60812 (QuantReg: 7.63594) QuantErr: 7.63594 batch_time=0.49165
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.52040 (QuantReg: 7.93564) QuantErr: 7.93564 batch_time=0.45264
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.25655 (QuantReg: 7.64877) QuantErr: 7.64877 batch_time=0.46468
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.93204 (QuantReg: 7.94341) QuantErr: 7.94341 batch_time=0.50561
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.30359 (QuantReg: 7.95605) QuantErr: 7.95605 batch_time=0.46440
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.67990 (QuantReg: 7.84156) QuantErr: 7.84156 batch_time=0.55625
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.59554 (QuantReg: 7.81793) QuantErr: 7.81793 batch_time=0.61635
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.70282 (QuantReg: 8.03390) QuantErr: 8.03390 batch_time=0.44791
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.60460 (QuantReg: 7.77202) QuantErr: 7.77202 batch_time=3.26850
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.84069 (QuantReg: 7.95652) QuantErr: 7.95652 batch_time=0.46011
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.41964 (QuantReg: 7.81619) QuantErr: 7.81619 batch_time=0.58331
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.79024 (QuantReg: 8.03291) QuantErr: 8.03291 batch_time=1.18846
Train Epoch: 5 codebook_update_time=0.88482
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch5.pth ...
Done in 8.235s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch5.pth ...
Done in 12.203s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.5731352038383486
quant_reg : 7.793773090362548
quant_err : 7.793773090362548
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.589
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.38288936566524
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.736
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.72559992214129
mnt_best : 35.38288936566524
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.73209 (QuantReg: 7.63435) QuantErr: 7.63435 batch_time=38.31992
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.70332 (QuantReg: 7.32870) QuantErr: 7.32870 batch_time=0.45175
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.46937 (QuantReg: 7.60548) QuantErr: 7.60548 batch_time=0.49591
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.45086 (QuantReg: 7.68052) QuantErr: 7.68052 batch_time=0.45794
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.28267 (QuantReg: 7.55919) QuantErr: 7.55919 batch_time=0.44542
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.24047 (QuantReg: 7.59163) QuantErr: 7.59163 batch_time=0.45432
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.68470 (QuantReg: 7.90256) QuantErr: 7.90256 batch_time=0.44209
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.37717 (QuantReg: 7.94537) QuantErr: 7.94537 batch_time=0.46246
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.46217 (QuantReg: 7.89894) QuantErr: 7.89894 batch_time=0.43859
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.20793 (QuantReg: 7.70708) QuantErr: 7.70708 batch_time=0.47525
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.32407 (QuantReg: 8.18979) QuantErr: 8.18979 batch_time=0.47164
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.38876 (QuantReg: 7.71781) QuantErr: 7.71781 batch_time=0.45535
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.27319 (QuantReg: 7.90885) QuantErr: 7.90885 batch_time=0.46228
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.22938 (QuantReg: 7.75033) QuantErr: 7.75033 batch_time=0.46699
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.62557 (QuantReg: 8.13561) QuantErr: 8.13561 batch_time=0.44807
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.34782 (QuantReg: 7.90916) QuantErr: 7.90916 batch_time=0.48076
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.12306 (QuantReg: 8.04376) QuantErr: 8.04376 batch_time=0.46825
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.50928 (QuantReg: 7.85645) QuantErr: 7.85645 batch_time=0.52783
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.64202 (QuantReg: 8.03232) QuantErr: 8.03232 batch_time=0.48867
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.41713 (QuantReg: 7.99835) QuantErr: 7.99835 batch_time=0.44871
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.21861 (QuantReg: 7.97663) QuantErr: 7.97663 batch_time=0.46468
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.20124 (QuantReg: 7.83934) QuantErr: 7.83934 batch_time=0.57081
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.03774 (QuantReg: 8.06871) QuantErr: 8.06871 batch_time=0.45765
Train Epoch: 6 codebook_update_time=0.87536
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch6.pth ...
Done in 4.280s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.3839840040206908
quant_reg : 7.896318431854248
quant_err : 7.896318431854248
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.117999999999995
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.334089362390685
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.5055
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.785487959794395
mnt_best : 35.38288936566524
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.41114 (QuantReg: 7.43145) QuantErr: 7.43145 batch_time=32.59956
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.42890 (QuantReg: 7.69262) QuantErr: 7.69262 batch_time=0.45169
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.20282 (QuantReg: 7.51023) QuantErr: 7.51023 batch_time=0.45279
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.97075 (QuantReg: 7.76075) QuantErr: 7.76075 batch_time=0.48949
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.01953 (QuantReg: 7.78327) QuantErr: 7.78327 batch_time=0.61654
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.08936 (QuantReg: 8.09592) QuantErr: 8.09592 batch_time=0.47193
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.24912 (QuantReg: 8.03430) QuantErr: 8.03430 batch_time=0.47548
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.18846 (QuantReg: 7.96161) QuantErr: 7.96161 batch_time=0.57220
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.28660 (QuantReg: 8.04733) QuantErr: 8.04733 batch_time=0.43921
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.75918 (QuantReg: 7.90136) QuantErr: 7.90136 batch_time=0.45273
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.75925 (QuantReg: 7.98154) QuantErr: 7.98154 batch_time=0.46976
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.18354 (QuantReg: 7.99777) QuantErr: 7.99777 batch_time=0.49739
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.13387 (QuantReg: 7.96856) QuantErr: 7.96856 batch_time=0.46468
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.29913 (QuantReg: 7.77976) QuantErr: 7.77976 batch_time=0.62227
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.69936 (QuantReg: 7.80351) QuantErr: 7.80351 batch_time=0.48524
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.26615 (QuantReg: 7.93780) QuantErr: 7.93780 batch_time=0.46715
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.71864 (QuantReg: 8.14328) QuantErr: 8.14328 batch_time=0.46477
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.29007 (QuantReg: 7.92044) QuantErr: 7.92044 batch_time=0.45485
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.64346 (QuantReg: 8.05156) QuantErr: 8.05156 batch_time=0.47388
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.46698 (QuantReg: 8.28750) QuantErr: 8.28750 batch_time=0.50098
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.17877 (QuantReg: 8.23953) QuantErr: 8.23953 batch_time=0.46695
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.17780 (QuantReg: 8.47929) QuantErr: 8.47929 batch_time=0.48465
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.22082 (QuantReg: 8.12025) QuantErr: 8.12025 batch_time=0.46691
Train Epoch: 7 codebook_update_time=0.88403
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch7.pth ...
Done in 4.227s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch7.pth ...
Done in 8.351s
removing stale ckpt [epoch 6] [took 0.04s]
epoch : 7
loss : 2.2156121048927306
quant_reg : 7.960436546325684
quant_err : 7.960436546325684
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.4295
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.192743984874944
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.6745
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.367547156978475
mnt_best : 36.192743984874944
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.42256 (QuantReg: 7.87193) QuantErr: 7.87193 batch_time=34.85593
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.37848 (QuantReg: 7.84394) QuantErr: 7.84394 batch_time=0.43838
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.62011 (QuantReg: 8.05914) QuantErr: 8.05914 batch_time=0.46885
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.15380 (QuantReg: 7.84345) QuantErr: 7.84345 batch_time=0.48578
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.25519 (QuantReg: 8.12880) QuantErr: 8.12880 batch_time=0.53014
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.93180 (QuantReg: 7.85059) QuantErr: 7.85059 batch_time=0.46496
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.11908 (QuantReg: 8.08833) QuantErr: 8.08833 batch_time=1.31324
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.37190 (QuantReg: 8.18337) QuantErr: 8.18337 batch_time=0.47870
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.34704 (QuantReg: 7.85834) QuantErr: 7.85834 batch_time=0.45209
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.52968 (QuantReg: 8.05306) QuantErr: 8.05306 batch_time=0.45437
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.38847 (QuantReg: 8.17402) QuantErr: 8.17402 batch_time=0.45334
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.06504 (QuantReg: 7.93510) QuantErr: 7.93510 batch_time=0.45771
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.26733 (QuantReg: 7.95356) QuantErr: 7.95356 batch_time=0.45676
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.29539 (QuantReg: 8.35352) QuantErr: 8.35352 batch_time=2.38711
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.05897 (QuantReg: 8.31850) QuantErr: 8.31850 batch_time=0.44797
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.88647 (QuantReg: 7.94541) QuantErr: 7.94541 batch_time=0.46540
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.01599 (QuantReg: 7.89414) QuantErr: 7.89414 batch_time=0.49184
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.64996 (QuantReg: 8.10254) QuantErr: 8.10254 batch_time=0.45454
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.28161 (QuantReg: 8.14208) QuantErr: 8.14208 batch_time=0.44925
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.27110 (QuantReg: 7.88931) QuantErr: 7.88931 batch_time=0.45874
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.22791 (QuantReg: 8.02595) QuantErr: 8.02595 batch_time=0.47074
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.67556 (QuantReg: 8.16579) QuantErr: 8.16579 batch_time=0.46945
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.26150 (QuantReg: 8.17115) QuantErr: 8.17115 batch_time=0.45171
Train Epoch: 8 codebook_update_time=0.90478
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch8.pth ...
Done in 4.453s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch8.pth ...
Done in 8.680s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 2.114922342300415
quant_reg : 8.043782474517823
quant_err : 8.043782474517823
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.305499999999995
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.52602135203659
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.8315
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.29496274699176
mnt_best : 36.52602135203659
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.73069 (QuantReg: 8.19795) QuantErr: 8.19795 batch_time=41.60326
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.23184 (QuantReg: 8.09839) QuantErr: 8.09839 batch_time=0.46624
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.81314 (QuantReg: 7.81098) QuantErr: 7.81098 batch_time=0.44866
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.10036 (QuantReg: 8.01999) QuantErr: 8.01999 batch_time=0.46990
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.97078 (QuantReg: 8.18474) QuantErr: 8.18474 batch_time=0.44819
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.72047 (QuantReg: 8.21229) QuantErr: 8.21229 batch_time=0.45473
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.73155 (QuantReg: 8.26590) QuantErr: 8.26590 batch_time=0.43431
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.94040 (QuantReg: 8.22992) QuantErr: 8.22992 batch_time=0.44354
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.14177 (QuantReg: 7.85790) QuantErr: 7.85790 batch_time=0.44265
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.88369 (QuantReg: 8.09215) QuantErr: 8.09215 batch_time=0.44898
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.05735 (QuantReg: 8.04698) QuantErr: 8.04698 batch_time=0.44872
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.64974 (QuantReg: 8.04477) QuantErr: 8.04477 batch_time=0.46123
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.98417 (QuantReg: 8.09004) QuantErr: 8.09004 batch_time=0.45375
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.86997 (QuantReg: 7.96634) QuantErr: 7.96634 batch_time=0.45272
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.09359 (QuantReg: 7.79243) QuantErr: 7.79243 batch_time=0.47782
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.67689 (QuantReg: 8.29142) QuantErr: 8.29142 batch_time=0.90975
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.05238 (QuantReg: 8.38871) QuantErr: 8.38871 batch_time=0.46364
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.47127 (QuantReg: 8.20511) QuantErr: 8.20511 batch_time=0.45604
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.95709 (QuantReg: 8.25575) QuantErr: 8.25575 batch_time=0.43753
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.88702 (QuantReg: 8.27358) QuantErr: 8.27358 batch_time=0.45410
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.73269 (QuantReg: 8.42503) QuantErr: 8.42503 batch_time=0.51729
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.79981 (QuantReg: 7.91370) QuantErr: 7.91370 batch_time=0.46567
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.89317 (QuantReg: 7.93073) QuantErr: 7.93073 batch_time=0.43920
Train Epoch: 9 codebook_update_time=0.96092
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch9.pth ...
Done in 4.387s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch9.pth ...
Done in 15.858s
removing stale ckpt [epoch 8] [took 0.10s]
epoch : 9
loss : 1.9812697615623474
quant_reg : 8.09144174194336
quant_err : 8.09144174194336
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.838
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.69623661092081
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.009
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.17392177735001
mnt_best : 37.69623661092081
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.78292 (QuantReg: 8.17134) QuantErr: 8.17134 batch_time=31.73486
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.93747 (QuantReg: 8.00046) QuantErr: 8.00046 batch_time=0.44674
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.25330 (QuantReg: 8.27331) QuantErr: 8.27331 batch_time=0.45446
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.06259 (QuantReg: 7.94308) QuantErr: 7.94308 batch_time=0.44541
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.64291 (QuantReg: 7.98879) QuantErr: 7.98879 batch_time=0.45323
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.99533 (QuantReg: 8.16954) QuantErr: 8.16954 batch_time=0.46967
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.34338 (QuantReg: 7.94608) QuantErr: 7.94608 batch_time=0.44754
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.97780 (QuantReg: 8.16412) QuantErr: 8.16412 batch_time=0.44462
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.03299 (QuantReg: 8.04412) QuantErr: 8.04412 batch_time=0.45942
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.89096 (QuantReg: 8.14646) QuantErr: 8.14646 batch_time=0.45831
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.63765 (QuantReg: 8.09180) QuantErr: 8.09180 batch_time=0.43928
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.97721 (QuantReg: 8.25388) QuantErr: 8.25388 batch_time=0.46142
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.90680 (QuantReg: 8.00515) QuantErr: 8.00515 batch_time=0.47225
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.54633 (QuantReg: 8.43653) QuantErr: 8.43653 batch_time=2.78371
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.90402 (QuantReg: 8.05058) QuantErr: 8.05058 batch_time=3.07693
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.16241 (QuantReg: 8.40249) QuantErr: 8.40249 batch_time=0.47937
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.09362 (QuantReg: 8.22207) QuantErr: 8.22207 batch_time=0.48053
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.87136 (QuantReg: 8.31731) QuantErr: 8.31731 batch_time=0.45182
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.57149 (QuantReg: 8.33089) QuantErr: 8.33089 batch_time=1.75746
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.50613 (QuantReg: 8.51709) QuantErr: 8.51709 batch_time=0.44218
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.94974 (QuantReg: 8.20633) QuantErr: 8.20633 batch_time=0.45352
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.17365 (QuantReg: 8.13833) QuantErr: 8.13833 batch_time=0.43849
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.02258 (QuantReg: 8.37713) QuantErr: 8.37713 batch_time=0.47756
Train Epoch: 10 codebook_update_time=0.95541
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch10.pth ...
Done in 4.896s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch10.pth ...
Done in 9.331s
removing stale ckpt [epoch 9] [took 0.33s]
epoch : 10
loss : 1.8878284311294555
quant_reg : 8.198499477386475
quant_err : 8.198499477386475
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.925
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.861997584535814
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.3205
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.210784247616836
mnt_best : 37.861997584535814
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.85660 (QuantReg: 7.97208) QuantErr: 7.97208 batch_time=35.56270
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.09944 (QuantReg: 7.87056) QuantErr: 7.87056 batch_time=0.44856
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.12081 (QuantReg: 8.09329) QuantErr: 8.09329 batch_time=0.47266
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.98793 (QuantReg: 8.21986) QuantErr: 8.21986 batch_time=0.46954
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.02848 (QuantReg: 8.11558) QuantErr: 8.11558 batch_time=0.49174
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.37420 (QuantReg: 8.56297) QuantErr: 8.56297 batch_time=0.44704
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.54496 (QuantReg: 8.20452) QuantErr: 8.20452 batch_time=0.45911
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.02218 (QuantReg: 7.95975) QuantErr: 7.95975 batch_time=0.45051
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.73711 (QuantReg: 8.13231) QuantErr: 8.13231 batch_time=0.45406
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.68343 (QuantReg: 8.29293) QuantErr: 8.29293 batch_time=0.47250
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.54917 (QuantReg: 8.30648) QuantErr: 8.30648 batch_time=0.44104
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.64828 (QuantReg: 8.26935) QuantErr: 8.26935 batch_time=0.45276
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.93075 (QuantReg: 8.09760) QuantErr: 8.09760 batch_time=0.46138
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.79477 (QuantReg: 8.22699) QuantErr: 8.22699 batch_time=0.47233
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.45886 (QuantReg: 8.31362) QuantErr: 8.31362 batch_time=0.45579
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.21479 (QuantReg: 7.94167) QuantErr: 7.94167 batch_time=0.47561
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.72146 (QuantReg: 8.23365) QuantErr: 8.23365 batch_time=0.45644
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.75066 (QuantReg: 8.46665) QuantErr: 8.46665 batch_time=0.44615
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.80791 (QuantReg: 8.17243) QuantErr: 8.17243 batch_time=0.44858
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.68795 (QuantReg: 8.33091) QuantErr: 8.33091 batch_time=0.49454
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.75633 (QuantReg: 8.18197) QuantErr: 8.18197 batch_time=0.45416
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.77584 (QuantReg: 8.34552) QuantErr: 8.34552 batch_time=0.48596
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.70817 (QuantReg: 8.44257) QuantErr: 8.44257 batch_time=0.45219
Train Epoch: 11 codebook_update_time=0.91021
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch11.pth ...
Done in 6.206s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch11.pth ...
Done in 11.473s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 1.804715036392212
quant_reg : 8.225706531524658
quant_err : 8.225706531524658
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.497
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.5363641357051
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.3935
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.53687410009514
mnt_best : 38.5363641357051
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.59642 (QuantReg: 8.07844) QuantErr: 8.07844 batch_time=37.07456
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.72010 (QuantReg: 8.18505) QuantErr: 8.18505 batch_time=0.51278
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.91244 (QuantReg: 8.24072) QuantErr: 8.24072 batch_time=0.45375
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.98730 (QuantReg: 8.25162) QuantErr: 8.25162 batch_time=0.46322
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.64579 (QuantReg: 8.16204) QuantErr: 8.16204 batch_time=0.51685
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.70413 (QuantReg: 8.32369) QuantErr: 8.32369 batch_time=0.44882
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.97089 (QuantReg: 8.46781) QuantErr: 8.46781 batch_time=0.47213
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.88689 (QuantReg: 7.95257) QuantErr: 7.95257 batch_time=0.46881
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.74297 (QuantReg: 8.20010) QuantErr: 8.20010 batch_time=0.46374
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.71866 (QuantReg: 8.27561) QuantErr: 8.27561 batch_time=0.46713
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.52796 (QuantReg: 8.22826) QuantErr: 8.22826 batch_time=0.45893
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.51365 (QuantReg: 8.40314) QuantErr: 8.40314 batch_time=0.44180
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.60652 (QuantReg: 8.13683) QuantErr: 8.13683 batch_time=0.44271
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.80618 (QuantReg: 8.07737) QuantErr: 8.07737 batch_time=0.95506
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.97433 (QuantReg: 8.56723) QuantErr: 8.56723 batch_time=0.45224
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.71885 (QuantReg: 8.38036) QuantErr: 8.38036 batch_time=0.47661
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.87354 (QuantReg: 8.49310) QuantErr: 8.49310 batch_time=0.43743
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.46117 (QuantReg: 8.38833) QuantErr: 8.38833 batch_time=0.48105
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.89281 (QuantReg: 8.24294) QuantErr: 8.24294 batch_time=0.43996
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.50662 (QuantReg: 8.40176) QuantErr: 8.40176 batch_time=0.46463
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.34865 (QuantReg: 8.58294) QuantErr: 8.58294 batch_time=0.47889
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.79546 (QuantReg: 8.57229) QuantErr: 8.57229 batch_time=0.47052
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.61079 (QuantReg: 8.35874) QuantErr: 8.35874 batch_time=0.46962
Train Epoch: 12 codebook_update_time=0.92684
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch12.pth ...
Done in 5.945s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch12.pth ...
Done in 11.630s
removing stale ckpt [epoch 11] [took 0.07s]
epoch : 12
loss : 1.740939169883728
quant_reg : 8.27195954322815
quant_err : 8.27195954322815
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.276
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.353137271010894
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.0355
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.25527413852921
mnt_best : 39.353137271010894
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.78293 (QuantReg: 8.08845) QuantErr: 8.08845 batch_time=37.18395
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.30865 (QuantReg: 8.58026) QuantErr: 8.58026 batch_time=0.46873
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.71605 (QuantReg: 8.15669) QuantErr: 8.15669 batch_time=0.48478
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.38429 (QuantReg: 8.37914) QuantErr: 8.37914 batch_time=0.53062
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.82118 (QuantReg: 8.35863) QuantErr: 8.35863 batch_time=0.43604
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.86181 (QuantReg: 8.05608) QuantErr: 8.05608 batch_time=0.44702
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.68470 (QuantReg: 8.34985) QuantErr: 8.34985 batch_time=0.45107
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.73175 (QuantReg: 8.50002) QuantErr: 8.50002 batch_time=0.46385
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.62947 (QuantReg: 8.42108) QuantErr: 8.42108 batch_time=0.46558
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.53610 (QuantReg: 8.27346) QuantErr: 8.27346 batch_time=0.45151
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.56044 (QuantReg: 8.46809) QuantErr: 8.46809 batch_time=0.44685
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.50109 (QuantReg: 8.61435) QuantErr: 8.61435 batch_time=0.49290
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.67367 (QuantReg: 8.31978) QuantErr: 8.31978 batch_time=0.44885
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.72590 (QuantReg: 8.46152) QuantErr: 8.46152 batch_time=0.47764
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.49699 (QuantReg: 8.41980) QuantErr: 8.41980 batch_time=0.48756
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.00651 (QuantReg: 8.34838) QuantErr: 8.34838 batch_time=0.45994
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.61158 (QuantReg: 8.51261) QuantErr: 8.51261 batch_time=0.48445
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.92151 (QuantReg: 8.28775) QuantErr: 8.28775 batch_time=0.45397
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.51066 (QuantReg: 8.50255) QuantErr: 8.50255 batch_time=0.49436
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.84820 (QuantReg: 8.32992) QuantErr: 8.32992 batch_time=0.46721
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.31368 (QuantReg: 8.48510) QuantErr: 8.48510 batch_time=0.47719
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.76460 (QuantReg: 8.41805) QuantErr: 8.41805 batch_time=0.44394
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.93367 (QuantReg: 8.29197) QuantErr: 8.29197 batch_time=0.70845
Train Epoch: 13 codebook_update_time=0.92053
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch13.pth ...
Done in 7.481s
removing stale ckpt [epoch 12] [took 0.11s]
epoch : 13
loss : 1.6926685357093811
quant_reg : 8.351317193984986
quant_err : 8.351317193984986
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.034
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.96814229398669
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.081
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.55045950185835
mnt_best : 39.353137271010894
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.56334 (QuantReg: 8.36351) QuantErr: 8.36351 batch_time=37.45597
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.52946 (QuantReg: 8.24256) QuantErr: 8.24256 batch_time=0.46801
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.60716 (QuantReg: 8.24377) QuantErr: 8.24377 batch_time=0.48965
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.52142 (QuantReg: 8.54488) QuantErr: 8.54488 batch_time=0.47499
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.55248 (QuantReg: 8.37206) QuantErr: 8.37206 batch_time=0.45033
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.94114 (QuantReg: 8.06822) QuantErr: 8.06822 batch_time=0.44140
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.82616 (QuantReg: 8.28367) QuantErr: 8.28367 batch_time=0.44592
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.60094 (QuantReg: 8.26544) QuantErr: 8.26544 batch_time=0.44057
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.48610 (QuantReg: 8.03948) QuantErr: 8.03948 batch_time=0.48284
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.37630 (QuantReg: 8.31670) QuantErr: 8.31670 batch_time=0.44760
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.09721 (QuantReg: 8.27077) QuantErr: 8.27077 batch_time=0.47838
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.51183 (QuantReg: 8.27569) QuantErr: 8.27569 batch_time=0.45437
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.45751 (QuantReg: 8.47719) QuantErr: 8.47719 batch_time=0.47017
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.81091 (QuantReg: 8.19358) QuantErr: 8.19358 batch_time=0.49331
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.53318 (QuantReg: 8.30438) QuantErr: 8.30438 batch_time=0.46989
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.87129 (QuantReg: 8.29864) QuantErr: 8.29864 batch_time=0.64976
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.53609 (QuantReg: 8.56568) QuantErr: 8.56568 batch_time=0.52888
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.81726 (QuantReg: 8.21261) QuantErr: 8.21261 batch_time=0.44504
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.59532 (QuantReg: 8.52404) QuantErr: 8.52404 batch_time=0.45971
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.64842 (QuantReg: 8.27141) QuantErr: 8.27141 batch_time=0.48547
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.76628 (QuantReg: 8.30976) QuantErr: 8.30976 batch_time=0.47081
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.38097 (QuantReg: 8.73403) QuantErr: 8.73403 batch_time=0.44717
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.27847 (QuantReg: 8.58795) QuantErr: 8.58795 batch_time=0.46290
Train Epoch: 14 codebook_update_time=0.93692
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch14.pth ...
Done in 5.535s
removing stale ckpt [epoch 13] [took 0.07s]
epoch : 14
loss : 1.6268961129188537
quant_reg : 8.348859016418457
quant_err : 8.348859016418457
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.518
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.27659603571831
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.6995
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.04639492852869
mnt_best : 39.353137271010894
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.52829 (QuantReg: 8.38101) QuantErr: 8.38101 batch_time=33.40262
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.14009 (QuantReg: 8.33906) QuantErr: 8.33906 batch_time=0.44126
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.65475 (QuantReg: 8.40009) QuantErr: 8.40009 batch_time=0.47784
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.39076 (QuantReg: 8.20099) QuantErr: 8.20099 batch_time=0.45596
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.80330 (QuantReg: 8.36626) QuantErr: 8.36626 batch_time=0.44272
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.66318 (QuantReg: 8.32972) QuantErr: 8.32972 batch_time=0.45014
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.77957 (QuantReg: 8.64868) QuantErr: 8.64868 batch_time=1.18599
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.32755 (QuantReg: 8.22242) QuantErr: 8.22242 batch_time=0.44507
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.88597 (QuantReg: 8.45802) QuantErr: 8.45802 batch_time=0.46188
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.54353 (QuantReg: 8.42413) QuantErr: 8.42413 batch_time=0.44439
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.86701 (QuantReg: 8.44006) QuantErr: 8.44006 batch_time=0.44962
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.31829 (QuantReg: 8.62650) QuantErr: 8.62650 batch_time=0.45721
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.55918 (QuantReg: 8.44648) QuantErr: 8.44648 batch_time=0.46694
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.45988 (QuantReg: 8.33181) QuantErr: 8.33181 batch_time=0.46166
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.66365 (QuantReg: 8.48539) QuantErr: 8.48539 batch_time=0.47981
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.49429 (QuantReg: 8.29984) QuantErr: 8.29984 batch_time=0.44989
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.44053 (QuantReg: 8.53931) QuantErr: 8.53931 batch_time=0.44684
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.58418 (QuantReg: 8.40242) QuantErr: 8.40242 batch_time=0.47946
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.86108 (QuantReg: 8.24507) QuantErr: 8.24507 batch_time=1.61224
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.37083 (QuantReg: 8.62991) QuantErr: 8.62991 batch_time=0.93745
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.38802 (QuantReg: 8.49283) QuantErr: 8.49283 batch_time=0.47802
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.63221 (QuantReg: 8.57968) QuantErr: 8.57968 batch_time=0.46260
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.47781 (QuantReg: 8.43207) QuantErr: 8.43207 batch_time=0.45182
Train Epoch: 15 codebook_update_time=0.94918
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch15.pth ...
Done in 5.796s
removing stale ckpt [epoch 14] [took 0.04s]
epoch : 15
loss : 1.5671334290504455
quant_reg : 8.398734647750855
quant_err : 8.398734647750855
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.022
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.295817591935005
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.3795
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.76194016526091
mnt_best : 39.353137271010894
not_improved_count: 3
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.76739 (QuantReg: 8.22598) QuantErr: 8.22598 batch_time=35.07568
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.53943 (QuantReg: 8.16828) QuantErr: 8.16828 batch_time=0.43841
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.38525 (QuantReg: 8.34053) QuantErr: 8.34053 batch_time=0.44569
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.15133 (QuantReg: 8.08696) QuantErr: 8.08696 batch_time=0.46303
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.59207 (QuantReg: 8.52809) QuantErr: 8.52809 batch_time=0.44336
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.66303 (QuantReg: 8.66479) QuantErr: 8.66479 batch_time=0.45811
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.22842 (QuantReg: 8.47928) QuantErr: 8.47928 batch_time=0.45491
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.58489 (QuantReg: 8.29422) QuantErr: 8.29422 batch_time=0.45200
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.51965 (QuantReg: 8.33786) QuantErr: 8.33786 batch_time=0.44627
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.47721 (QuantReg: 8.58958) QuantErr: 8.58958 batch_time=0.45142
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.37217 (QuantReg: 8.58202) QuantErr: 8.58202 batch_time=0.48588
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.54916 (QuantReg: 8.48007) QuantErr: 8.48007 batch_time=0.45438
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.47672 (QuantReg: 8.46920) QuantErr: 8.46920 batch_time=1.02836
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.87596 (QuantReg: 8.48649) QuantErr: 8.48649 batch_time=0.56459
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.86514 (QuantReg: 8.59089) QuantErr: 8.59089 batch_time=1.65699
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.68055 (QuantReg: 8.51286) QuantErr: 8.51286 batch_time=0.47520
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.34608 (QuantReg: 8.67287) QuantErr: 8.67287 batch_time=0.45257
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.68178 (QuantReg: 8.43884) QuantErr: 8.43884 batch_time=0.50437
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.58493 (QuantReg: 8.47380) QuantErr: 8.47380 batch_time=0.46524
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.31419 (QuantReg: 8.70526) QuantErr: 8.70526 batch_time=1.84096
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.58917 (QuantReg: 8.44851) QuantErr: 8.44851 batch_time=0.45229
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.72165 (QuantReg: 8.35961) QuantErr: 8.35961 batch_time=0.49584
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.60216 (QuantReg: 8.43330) QuantErr: 8.43330 batch_time=0.48432
Train Epoch: 16 codebook_update_time=0.87928
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch16.pth ...
Done in 6.083s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch16.pth ...
Done in 29.071s
removing stale ckpt [epoch 15] [took 0.43s]
epoch : 16
loss : 1.563151083946228
quant_reg : 8.433514961242675
quant_err : 8.433514961242675
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.3
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.07136490259282
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.9535
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.72262772801427
mnt_best : 41.07136490259282
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.22270 (QuantReg: 8.59016) QuantErr: 8.59016 batch_time=36.09656
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.73567 (QuantReg: 8.21816) QuantErr: 8.21816 batch_time=0.48964
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.42343 (QuantReg: 8.23342) QuantErr: 8.23342 batch_time=0.47617
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.03131 (QuantReg: 8.59433) QuantErr: 8.59433 batch_time=0.48426
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.58270 (QuantReg: 8.17008) QuantErr: 8.17008 batch_time=0.48887
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.53125 (QuantReg: 8.52229) QuantErr: 8.52229 batch_time=0.50386
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.76101 (QuantReg: 8.48045) QuantErr: 8.48045 batch_time=0.46502
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.30225 (QuantReg: 8.42273) QuantErr: 8.42273 batch_time=0.48925
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.45664 (QuantReg: 8.27735) QuantErr: 8.27735 batch_time=0.45566
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.54464 (QuantReg: 8.46544) QuantErr: 8.46544 batch_time=0.50332
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.63795 (QuantReg: 8.53729) QuantErr: 8.53729 batch_time=0.48868
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.58431 (QuantReg: 8.39887) QuantErr: 8.39887 batch_time=0.47408
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.69770 (QuantReg: 8.45008) QuantErr: 8.45008 batch_time=0.52297
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.49879 (QuantReg: 8.55043) QuantErr: 8.55043 batch_time=0.45481
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.90335 (QuantReg: 8.17483) QuantErr: 8.17483 batch_time=0.51404
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.43684 (QuantReg: 8.37842) QuantErr: 8.37842 batch_time=0.50005
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.62599 (QuantReg: 8.64600) QuantErr: 8.64600 batch_time=0.45829
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.31470 (QuantReg: 8.70871) QuantErr: 8.70871 batch_time=0.52651
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.48600 (QuantReg: 8.17061) QuantErr: 8.17061 batch_time=0.48521
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.39058 (QuantReg: 8.60565) QuantErr: 8.60565 batch_time=0.77761
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.42280 (QuantReg: 8.31123) QuantErr: 8.31123 batch_time=0.54150
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.47960 (QuantReg: 8.76960) QuantErr: 8.76960 batch_time=0.46568
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.25352 (QuantReg: 8.41264) QuantErr: 8.41264 batch_time=0.57290
Train Epoch: 17 codebook_update_time=0.97737
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch17.pth ...
Done in 5.046s
removing stale ckpt [epoch 16] [took 0.13s]
epoch : 17
loss : 1.5092860865592956
quant_reg : 8.478766418457031
quant_err : 8.478766418457031
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.8055
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.759436809792675
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.509
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.61157952284934
mnt_best : 41.07136490259282
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.53776 (QuantReg: 8.36449) QuantErr: 8.36449 batch_time=33.40424
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.46705 (QuantReg: 8.40326) QuantErr: 8.40326 batch_time=0.45538
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.34001 (QuantReg: 8.42533) QuantErr: 8.42533 batch_time=0.47553
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.24070 (QuantReg: 8.58337) QuantErr: 8.58337 batch_time=0.48647
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.46683 (QuantReg: 8.33306) QuantErr: 8.33306 batch_time=0.43752
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.31198 (QuantReg: 8.67445) QuantErr: 8.67445 batch_time=0.46369
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.15722 (QuantReg: 8.72648) QuantErr: 8.72648 batch_time=4.38029
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.21215 (QuantReg: 8.62849) QuantErr: 8.62849 batch_time=0.45790
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.66945 (QuantReg: 8.66752) QuantErr: 8.66752 batch_time=1.12719
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.42042 (QuantReg: 8.34951) QuantErr: 8.34951 batch_time=0.49779
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.73854 (QuantReg: 8.44466) QuantErr: 8.44466 batch_time=0.44351
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.54236 (QuantReg: 8.62299) QuantErr: 8.62299 batch_time=0.50596
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.51139 (QuantReg: 8.39960) QuantErr: 8.39960 batch_time=0.46962
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.69906 (QuantReg: 8.47321) QuantErr: 8.47321 batch_time=0.45408
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.65261 (QuantReg: 8.33000) QuantErr: 8.33000 batch_time=0.47049
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.50468 (QuantReg: 8.79205) QuantErr: 8.79205 batch_time=0.45107
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.37267 (QuantReg: 8.38806) QuantErr: 8.38806 batch_time=0.46022
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.27662 (QuantReg: 8.52134) QuantErr: 8.52134 batch_time=0.45931
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.36392 (QuantReg: 8.60386) QuantErr: 8.60386 batch_time=0.46276
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.11397 (QuantReg: 8.63353) QuantErr: 8.63353 batch_time=0.47470
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.89684 (QuantReg: 8.32959) QuantErr: 8.32959 batch_time=0.44431
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.35469 (QuantReg: 8.60189) QuantErr: 8.60189 batch_time=0.44369
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.44475 (QuantReg: 8.52501) QuantErr: 8.52501 batch_time=0.46641
Train Epoch: 18 codebook_update_time=1.02372
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch18.pth ...
Done in 6.639s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 1.4585079727172852
quant_reg : 8.526630207061768
quant_err : 8.526630207061768
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.364
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.39586109283567
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.842
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.52086797358645
mnt_best : 41.07136490259282
not_improved_count: 2
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.31266 (QuantReg: 8.56386) QuantErr: 8.56386 batch_time=33.36660
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.33752 (QuantReg: 8.51363) QuantErr: 8.51363 batch_time=0.44695
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.61108 (QuantReg: 8.32609) QuantErr: 8.32609 batch_time=0.46459
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.72429 (QuantReg: 8.45582) QuantErr: 8.45582 batch_time=0.45015
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.37865 (QuantReg: 8.46324) QuantErr: 8.46324 batch_time=0.48351
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.06125 (QuantReg: 8.65032) QuantErr: 8.65032 batch_time=0.47618
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.08221 (QuantReg: 8.63028) QuantErr: 8.63028 batch_time=0.50107
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.27321 (QuantReg: 8.41506) QuantErr: 8.41506 batch_time=2.73858
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.26417 (QuantReg: 8.50892) QuantErr: 8.50892 batch_time=0.47336
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.73980 (QuantReg: 8.29548) QuantErr: 8.29548 batch_time=0.46661
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.35650 (QuantReg: 8.36241) QuantErr: 8.36241 batch_time=0.44556
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.29602 (QuantReg: 8.71113) QuantErr: 8.71113 batch_time=0.47542
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.45080 (QuantReg: 8.81182) QuantErr: 8.81182 batch_time=0.46711
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.43426 (QuantReg: 8.57738) QuantErr: 8.57738 batch_time=0.46416
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.74848 (QuantReg: 8.39577) QuantErr: 8.39577 batch_time=0.47876
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.04420 (QuantReg: 8.51494) QuantErr: 8.51494 batch_time=0.46614
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.34467 (QuantReg: 8.58177) QuantErr: 8.58177 batch_time=0.46675
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.45644 (QuantReg: 8.48105) QuantErr: 8.48105 batch_time=0.45007
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.53517 (QuantReg: 8.26901) QuantErr: 8.26901 batch_time=0.44980
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.40469 (QuantReg: 8.51148) QuantErr: 8.51148 batch_time=0.99973
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.46412 (QuantReg: 8.79745) QuantErr: 8.79745 batch_time=0.45962
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.77293 (QuantReg: 8.48553) QuantErr: 8.48553 batch_time=0.44839
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.55314 (QuantReg: 8.81946) QuantErr: 8.81946 batch_time=0.44956
Train Epoch: 19 codebook_update_time=0.90018
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch19.pth ...
Done in 7.617s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M16/checkpoint-epoch19.pth ...
Done in 13.366s
removing stale ckpt [epoch 18] [took 0.19s]
epoch : 19
loss : 1.4356058604717254
quant_reg : 8.542489967346192
quant_err : 8.542489967346192
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.7935