-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_bert-large.txt
2591 lines (2591 loc) · 194 KB
/
HCQ_MSRVTT_1kA_bert-large.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large
Preparing the dataloaders ...
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch0.pth ...
Done in 3.689s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch0.pth ...
Done in 7.560s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 6.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 490.001
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3556893304490063
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 467.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 487.826
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.2519842099789747
mnt_best : 0.3556893304490063
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.90157 (QuantReg: 22.62626) QuantErr: 22.62626 batch_time=51.59324
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.12747 (QuantReg: 22.73657) QuantErr: 22.73657 batch_time=0.88629
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.23432 (QuantReg: 22.54213) QuantErr: 22.54213 batch_time=0.65360
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.17346 (QuantReg: 22.48737) QuantErr: 22.48737 batch_time=1.95617
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 5.92079 (QuantReg: 22.58582) QuantErr: 22.58582 batch_time=0.68256
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.53785 (QuantReg: 22.59554) QuantErr: 22.59554 batch_time=0.63550
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.25629 (QuantReg: 22.63478) QuantErr: 22.63478 batch_time=0.77184
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.69528 (QuantReg: 22.61038) QuantErr: 22.61038 batch_time=1.76445
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.40410 (QuantReg: 22.61926) QuantErr: 22.61926 batch_time=0.65325
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 4.85936 (QuantReg: 22.63071) QuantErr: 22.63071 batch_time=0.69039
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.24883 (QuantReg: 22.61330) QuantErr: 22.61330 batch_time=0.66165
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.99816 (QuantReg: 22.62665) QuantErr: 22.62665 batch_time=0.67587
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.93215 (QuantReg: 22.59356) QuantErr: 22.59356 batch_time=0.83818
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.54260 (QuantReg: 22.64045) QuantErr: 22.64045 batch_time=0.68211
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.50378 (QuantReg: 22.61876) QuantErr: 22.61876 batch_time=0.70109
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.07526 (QuantReg: 22.62406) QuantErr: 22.62406 batch_time=0.65333
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.28675 (QuantReg: 22.62665) QuantErr: 22.62665 batch_time=0.65683
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.11598 (QuantReg: 22.60871) QuantErr: 22.60871 batch_time=0.68304
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.38517 (QuantReg: 22.67294) QuantErr: 22.67294 batch_time=0.71154
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 3.86324 (QuantReg: 22.63184) QuantErr: 22.63184 batch_time=0.67266
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.72173 (QuantReg: 22.63331) QuantErr: 22.63331 batch_time=0.72180
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.08901 (QuantReg: 22.65752) QuantErr: 22.65752 batch_time=0.68607
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.87725 (QuantReg: 22.63122) QuantErr: 22.63122 batch_time=0.67786
Train Epoch: 1 codebook_update_time=1.72050
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch1.pth ...
Done in 11.135s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch1.pth ...
Done in 22.381s
epoch : 1
loss : 5.341585871696473
quant_reg : 22.61388256072998
quant_err : 22.61388256072998
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 33.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 47.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 81.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 37.294
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.48801974195142
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 34.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 49.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 80.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 11.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 39.782
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.814694547818867
mnt_best : 26.48801974195142
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.00097 (QuantReg: 12.25031) QuantErr: 12.25031 batch_time=45.60027
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.96727 (QuantReg: 12.46347) QuantErr: 12.46347 batch_time=0.70176
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.84861 (QuantReg: 13.13649) QuantErr: 13.13649 batch_time=0.68861
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.61442 (QuantReg: 13.00640) QuantErr: 13.00640 batch_time=0.72764
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 5.05301 (QuantReg: 13.13036) QuantErr: 13.13036 batch_time=0.67746
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.87369 (QuantReg: 13.46346) QuantErr: 13.46346 batch_time=0.70868
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.79331 (QuantReg: 13.77826) QuantErr: 13.77826 batch_time=0.67038
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.46775 (QuantReg: 12.96553) QuantErr: 12.96553 batch_time=0.68681
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.91949 (QuantReg: 13.77348) QuantErr: 13.77348 batch_time=0.65942
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.45405 (QuantReg: 14.21213) QuantErr: 14.21213 batch_time=0.75893
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.73740 (QuantReg: 14.17816) QuantErr: 14.17816 batch_time=0.67763
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.10927 (QuantReg: 13.52417) QuantErr: 13.52417 batch_time=0.66275
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 4.22875 (QuantReg: 13.97401) QuantErr: 13.97401 batch_time=0.71021
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.89763 (QuantReg: 13.82999) QuantErr: 13.82999 batch_time=0.70909
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.38282 (QuantReg: 13.77207) QuantErr: 13.77207 batch_time=0.65929
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.56216 (QuantReg: 14.05404) QuantErr: 14.05404 batch_time=0.73461
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.58614 (QuantReg: 14.39092) QuantErr: 14.39092 batch_time=0.74633
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.18889 (QuantReg: 14.27547) QuantErr: 14.27547 batch_time=0.66954
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.52012 (QuantReg: 14.32135) QuantErr: 14.32135 batch_time=0.66389
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.45564 (QuantReg: 14.12117) QuantErr: 14.12117 batch_time=0.70689
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.25563 (QuantReg: 14.75181) QuantErr: 14.75181 batch_time=0.64890
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 2.77260 (QuantReg: 14.89141) QuantErr: 14.89141 batch_time=0.72592
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.83829 (QuantReg: 14.88797) QuantErr: 14.88797 batch_time=0.72097
Train Epoch: 2 codebook_update_time=1.80841
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch2.pth ...
Done in 11.007s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch2.pth ...
Done in 22.599s
removing stale ckpt [epoch 1] [took 0.04s]
removing stale ckpt [epoch 0] [took 0.03s]
epoch : 2
loss : 3.7270275983810426
quant_reg : 13.83795336151123
quant_err : 13.83795336151123
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 54.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.498999999999995
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.656395636184513
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.937
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.32939873710492
mnt_best : 30.656395636184513
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.71703 (QuantReg: 11.73929) QuantErr: 11.73929 batch_time=37.32081
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.30070 (QuantReg: 12.36005) QuantErr: 12.36005 batch_time=0.68351
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.76181 (QuantReg: 12.55981) QuantErr: 12.55981 batch_time=0.75750
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.08240 (QuantReg: 12.28752) QuantErr: 12.28752 batch_time=0.68159
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.72419 (QuantReg: 12.04693) QuantErr: 12.04693 batch_time=0.64852
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.86588 (QuantReg: 12.16188) QuantErr: 12.16188 batch_time=0.64134
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.30413 (QuantReg: 12.67384) QuantErr: 12.67384 batch_time=1.15097
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.51742 (QuantReg: 12.96263) QuantErr: 12.96263 batch_time=0.79169
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.20097 (QuantReg: 12.52099) QuantErr: 12.52099 batch_time=0.66961
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.23283 (QuantReg: 12.37053) QuantErr: 12.37053 batch_time=0.66887
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.75188 (QuantReg: 12.90039) QuantErr: 12.90039 batch_time=0.72388
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.44280 (QuantReg: 12.79505) QuantErr: 12.79505 batch_time=0.67242
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.12644 (QuantReg: 13.07330) QuantErr: 13.07330 batch_time=0.90755
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.94075 (QuantReg: 13.07922) QuantErr: 13.07922 batch_time=4.27536
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 2.95158 (QuantReg: 12.98240) QuantErr: 12.98240 batch_time=6.81130
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.05093 (QuantReg: 13.30937) QuantErr: 13.30937 batch_time=0.71829
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.65244 (QuantReg: 13.49017) QuantErr: 13.49017 batch_time=0.66089
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.84400 (QuantReg: 13.23790) QuantErr: 13.23790 batch_time=0.67486
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.24134 (QuantReg: 13.12080) QuantErr: 13.12080 batch_time=1.05770
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.15456 (QuantReg: 13.14541) QuantErr: 13.14541 batch_time=0.71729
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.69442 (QuantReg: 13.34395) QuantErr: 13.34395 batch_time=0.65955
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.83735 (QuantReg: 13.68658) QuantErr: 13.68658 batch_time=0.81369
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.92863 (QuantReg: 13.36873) QuantErr: 13.36873 batch_time=0.70518
Train Epoch: 3 codebook_update_time=1.68283
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch3.pth ...
Done in 11.100s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch3.pth ...
Done in 21.554s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.0887889194488527
quant_reg : 12.848927570343017
quant_err : 12.848927570343017
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.217
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.91108399227773
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.721
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.262926883930355
mnt_best : 33.91108399227773
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.56724 (QuantReg: 11.80882) QuantErr: 11.80882 batch_time=38.39035
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.10112 (QuantReg: 12.21495) QuantErr: 12.21495 batch_time=0.73620
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.34405 (QuantReg: 12.35487) QuantErr: 12.35487 batch_time=0.65456
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.75701 (QuantReg: 12.17541) QuantErr: 12.17541 batch_time=0.69800
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.33252 (QuantReg: 12.77043) QuantErr: 12.77043 batch_time=0.71428
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.89109 (QuantReg: 12.15602) QuantErr: 12.15602 batch_time=0.67807
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.86110 (QuantReg: 12.33327) QuantErr: 12.33327 batch_time=0.63497
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.00007 (QuantReg: 12.41467) QuantErr: 12.41467 batch_time=0.68937
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.89684 (QuantReg: 12.56579) QuantErr: 12.56579 batch_time=0.68036
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.68911 (QuantReg: 12.41777) QuantErr: 12.41777 batch_time=0.68746
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.27930 (QuantReg: 13.14397) QuantErr: 13.14397 batch_time=0.73391
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.79919 (QuantReg: 12.60423) QuantErr: 12.60423 batch_time=0.65747
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.25580 (QuantReg: 12.78897) QuantErr: 12.78897 batch_time=0.66927
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.89205 (QuantReg: 12.55833) QuantErr: 12.55833 batch_time=0.67863
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.83380 (QuantReg: 12.62314) QuantErr: 12.62314 batch_time=0.66605
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.79427 (QuantReg: 12.82724) QuantErr: 12.82724 batch_time=0.71975
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.49351 (QuantReg: 13.03346) QuantErr: 13.03346 batch_time=0.70490
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.89925 (QuantReg: 12.57224) QuantErr: 12.57224 batch_time=0.67730
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.61024 (QuantReg: 13.16607) QuantErr: 13.16607 batch_time=0.68918
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.66471 (QuantReg: 13.18845) QuantErr: 13.18845 batch_time=0.78527
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.34186 (QuantReg: 12.90890) QuantErr: 12.90890 batch_time=0.66713
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.17614 (QuantReg: 13.12488) QuantErr: 13.12488 batch_time=0.66051
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.22345 (QuantReg: 13.08592) QuantErr: 13.08592 batch_time=0.80265
Train Epoch: 4 codebook_update_time=1.80254
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch4.pth ...
Done in 11.281s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch4.pth ...
Done in 22.451s
removing stale ckpt [epoch 3] [took 0.03s]
epoch : 4
loss : 2.724169358253479
quant_reg : 12.724106616973877
quant_err : 12.724106616973877
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.986
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.32993607005325
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.3235
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.59472437321885
mnt_best : 36.32993607005325
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.16937 (QuantReg: 11.96146) QuantErr: 11.96146 batch_time=44.61426
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.34986 (QuantReg: 12.13319) QuantErr: 12.13319 batch_time=0.66822
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.88271 (QuantReg: 12.44464) QuantErr: 12.44464 batch_time=0.71733
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.72054 (QuantReg: 12.51808) QuantErr: 12.51808 batch_time=0.71155
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.19941 (QuantReg: 12.14177) QuantErr: 12.14177 batch_time=0.68075
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.33108 (QuantReg: 12.30169) QuantErr: 12.30169 batch_time=0.77229
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.25907 (QuantReg: 12.70345) QuantErr: 12.70345 batch_time=0.65261
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.36410 (QuantReg: 12.78486) QuantErr: 12.78486 batch_time=0.68515
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.21037 (QuantReg: 12.43190) QuantErr: 12.43190 batch_time=0.71672
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.26417 (QuantReg: 12.48984) QuantErr: 12.48984 batch_time=0.67814
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.88886 (QuantReg: 12.94301) QuantErr: 12.94301 batch_time=0.67905
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.66016 (QuantReg: 12.36267) QuantErr: 12.36267 batch_time=0.68574
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.59169 (QuantReg: 12.58358) QuantErr: 12.58358 batch_time=0.67475
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.70235 (QuantReg: 12.26557) QuantErr: 12.26557 batch_time=9.79028
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.31859 (QuantReg: 12.79196) QuantErr: 12.79196 batch_time=0.70157
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.75836 (QuantReg: 13.01817) QuantErr: 13.01817 batch_time=0.84855
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.45380 (QuantReg: 12.92288) QuantErr: 12.92288 batch_time=0.74181
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.49698 (QuantReg: 12.92245) QuantErr: 12.92245 batch_time=0.67646
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.35395 (QuantReg: 13.26928) QuantErr: 13.26928 batch_time=0.67244
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.71963 (QuantReg: 12.77077) QuantErr: 12.77077 batch_time=1.21969
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.55992 (QuantReg: 12.92205) QuantErr: 12.92205 batch_time=0.68688
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.17863 (QuantReg: 12.90838) QuantErr: 12.90838 batch_time=0.66144
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.26882 (QuantReg: 12.81308) QuantErr: 12.81308 batch_time=0.64627
Train Epoch: 5 codebook_update_time=1.70918
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch5.pth ...
Done in 11.242s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch5.pth ...
Done in 22.054s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.4688736805915834
quant_reg : 12.673792861938477
quant_err : 12.673792861938477
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.764
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.02836201226121
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.5175
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.16909875546781
mnt_best : 39.02836201226121
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.34602 (QuantReg: 12.31955) QuantErr: 12.31955 batch_time=39.79192
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.53791 (QuantReg: 12.47859) QuantErr: 12.47859 batch_time=0.74407
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.68014 (QuantReg: 12.73191) QuantErr: 12.73191 batch_time=0.69066
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.06548 (QuantReg: 12.33042) QuantErr: 12.33042 batch_time=0.80072
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.42596 (QuantReg: 12.55793) QuantErr: 12.55793 batch_time=0.69690
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.05675 (QuantReg: 12.66886) QuantErr: 12.66886 batch_time=0.67710
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.06190 (QuantReg: 12.67822) QuantErr: 12.67822 batch_time=1.18051
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.06354 (QuantReg: 12.66109) QuantErr: 12.66109 batch_time=0.65661
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.46403 (QuantReg: 12.39224) QuantErr: 12.39224 batch_time=0.67279
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.35806 (QuantReg: 13.28012) QuantErr: 13.28012 batch_time=0.67755
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.75112 (QuantReg: 13.07910) QuantErr: 13.07910 batch_time=0.66556
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.30767 (QuantReg: 12.72870) QuantErr: 12.72870 batch_time=0.68626
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.09832 (QuantReg: 12.80555) QuantErr: 12.80555 batch_time=0.68844
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.56312 (QuantReg: 13.01658) QuantErr: 13.01658 batch_time=1.58946
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 1.97964 (QuantReg: 12.87312) QuantErr: 12.87312 batch_time=0.65816
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.36515 (QuantReg: 13.02428) QuantErr: 13.02428 batch_time=0.83648
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.10061 (QuantReg: 12.71294) QuantErr: 12.71294 batch_time=0.65672
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.04125 (QuantReg: 12.74667) QuantErr: 12.74667 batch_time=0.70146
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.26533 (QuantReg: 13.28871) QuantErr: 13.28871 batch_time=0.78303
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.02739 (QuantReg: 12.96090) QuantErr: 12.96090 batch_time=0.68731
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.03240 (QuantReg: 13.14815) QuantErr: 13.14815 batch_time=0.67567
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.63262 (QuantReg: 12.81305) QuantErr: 12.81305 batch_time=0.66241
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.48605 (QuantReg: 13.16372) QuantErr: 13.16372 batch_time=0.72362
Train Epoch: 6 codebook_update_time=1.69888
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch6.pth ...
Done in 10.572s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.325248544692993
quant_reg : 12.806637077331542
quant_err : 12.806637077331542
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.202
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.31263231797979
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.076
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.37758136095879
mnt_best : 39.02836201226121
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 1.89974 (QuantReg: 13.04079) QuantErr: 13.04079 batch_time=50.37739
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.38080 (QuantReg: 12.47096) QuantErr: 12.47096 batch_time=0.66822
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.13624 (QuantReg: 12.40967) QuantErr: 12.40967 batch_time=0.66347
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.16767 (QuantReg: 12.72051) QuantErr: 12.72051 batch_time=0.70058
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.43570 (QuantReg: 12.61013) QuantErr: 12.61013 batch_time=0.69793
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.48013 (QuantReg: 12.68746) QuantErr: 12.68746 batch_time=0.66952
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.23507 (QuantReg: 12.47050) QuantErr: 12.47050 batch_time=0.64963
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.01794 (QuantReg: 12.74138) QuantErr: 12.74138 batch_time=0.70599
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.85782 (QuantReg: 12.69280) QuantErr: 12.69280 batch_time=0.69849
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.10601 (QuantReg: 12.47057) QuantErr: 12.47057 batch_time=0.67436
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.15547 (QuantReg: 12.88379) QuantErr: 12.88379 batch_time=0.68206
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.72685 (QuantReg: 13.19197) QuantErr: 13.19197 batch_time=0.78218
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.43163 (QuantReg: 12.60202) QuantErr: 12.60202 batch_time=0.72378
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 1.99695 (QuantReg: 12.98296) QuantErr: 12.98296 batch_time=0.73088
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.76225 (QuantReg: 12.65994) QuantErr: 12.65994 batch_time=0.67991
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.39965 (QuantReg: 13.07555) QuantErr: 13.07555 batch_time=0.70018
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.91651 (QuantReg: 13.14035) QuantErr: 13.14035 batch_time=0.68016
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.12203 (QuantReg: 12.83861) QuantErr: 12.83861 batch_time=0.66642
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.15899 (QuantReg: 12.98417) QuantErr: 12.98417 batch_time=0.85514
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.78550 (QuantReg: 12.72717) QuantErr: 12.72717 batch_time=0.67995
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.24455 (QuantReg: 13.19365) QuantErr: 13.19365 batch_time=0.66882
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.20508 (QuantReg: 12.77380) QuantErr: 12.77380 batch_time=0.65351
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.56891 (QuantReg: 12.60416) QuantErr: 12.60416 batch_time=0.83092
Train Epoch: 7 codebook_update_time=1.67586
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch7.pth ...
Done in 10.676s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.2067505569458006
quant_reg : 12.78955094909668
quant_err : 12.78955094909668
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.158
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.98322092150765
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.1165
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.297318673127386
mnt_best : 39.02836201226121
not_improved_count: 2
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.92135 (QuantReg: 12.33555) QuantErr: 12.33555 batch_time=38.06044
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 1.84798 (QuantReg: 12.53349) QuantErr: 12.53349 batch_time=0.70142
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 1.91674 (QuantReg: 12.15306) QuantErr: 12.15306 batch_time=0.74813
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.87012 (QuantReg: 12.76036) QuantErr: 12.76036 batch_time=0.66969
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.37068 (QuantReg: 12.19043) QuantErr: 12.19043 batch_time=0.72347
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.99200 (QuantReg: 12.73616) QuantErr: 12.73616 batch_time=0.66047
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.96305 (QuantReg: 13.13259) QuantErr: 13.13259 batch_time=0.68921
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.15674 (QuantReg: 12.62141) QuantErr: 12.62141 batch_time=0.93969
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.08352 (QuantReg: 12.79387) QuantErr: 12.79387 batch_time=0.67224
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.29817 (QuantReg: 12.89079) QuantErr: 12.89079 batch_time=0.70384
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.10334 (QuantReg: 12.80278) QuantErr: 12.80278 batch_time=0.68633
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.81608 (QuantReg: 13.40682) QuantErr: 13.40682 batch_time=0.67644
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.94083 (QuantReg: 12.91917) QuantErr: 12.91917 batch_time=0.68183
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.79277 (QuantReg: 12.83317) QuantErr: 12.83317 batch_time=0.95962
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.03768 (QuantReg: 12.71921) QuantErr: 12.71921 batch_time=0.70701
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.73747 (QuantReg: 13.36383) QuantErr: 13.36383 batch_time=0.67838
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.91070 (QuantReg: 13.11424) QuantErr: 13.11424 batch_time=0.69050
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.03665 (QuantReg: 12.85205) QuantErr: 12.85205 batch_time=0.67977
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.79848 (QuantReg: 13.20968) QuantErr: 13.20968 batch_time=0.66895
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.89337 (QuantReg: 13.01411) QuantErr: 13.01411 batch_time=0.91433
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.15412 (QuantReg: 13.04590) QuantErr: 13.04590 batch_time=0.69362
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.85256 (QuantReg: 13.02162) QuantErr: 13.02162 batch_time=0.67423
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.78732 (QuantReg: 13.33591) QuantErr: 13.33591 batch_time=0.66942
Train Epoch: 8 codebook_update_time=1.64853
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch8.pth ...
Done in 10.808s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch8.pth ...
Done in 21.003s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.064859663963318
quant_reg : 12.879175846099853
quant_err : 12.879175846099853
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.976
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.90615617825036
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.3635
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.37414895277925
mnt_best : 40.90615617825036
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.63067 (QuantReg: 12.75302) QuantErr: 12.75302 batch_time=46.89231
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.18140 (QuantReg: 12.56495) QuantErr: 12.56495 batch_time=0.97986
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.16610 (QuantReg: 12.85226) QuantErr: 12.85226 batch_time=0.67957
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.75529 (QuantReg: 12.86785) QuantErr: 12.86785 batch_time=0.65744
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.98973 (QuantReg: 12.73166) QuantErr: 12.73166 batch_time=0.92948
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.18916 (QuantReg: 12.81564) QuantErr: 12.81564 batch_time=0.65714
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.15209 (QuantReg: 13.21482) QuantErr: 13.21482 batch_time=1.38110
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.65117 (QuantReg: 12.89679) QuantErr: 12.89679 batch_time=0.77713
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.85827 (QuantReg: 12.97694) QuantErr: 12.97694 batch_time=0.67025
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.15676 (QuantReg: 12.59340) QuantErr: 12.59340 batch_time=0.65925
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.29775 (QuantReg: 12.76477) QuantErr: 12.76477 batch_time=0.71823
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.89757 (QuantReg: 12.73691) QuantErr: 12.73691 batch_time=0.70201
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.06680 (QuantReg: 13.04430) QuantErr: 13.04430 batch_time=0.66787
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.99460 (QuantReg: 12.96343) QuantErr: 12.96343 batch_time=0.66676
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.11427 (QuantReg: 12.96784) QuantErr: 12.96784 batch_time=0.68481
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.72216 (QuantReg: 13.30892) QuantErr: 13.30892 batch_time=0.71019
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.95586 (QuantReg: 12.93145) QuantErr: 12.93145 batch_time=0.67853
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.50777 (QuantReg: 12.56325) QuantErr: 12.56325 batch_time=0.70834
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.49784 (QuantReg: 12.66166) QuantErr: 12.66166 batch_time=0.68125
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.63119 (QuantReg: 13.05695) QuantErr: 13.05695 batch_time=0.73773
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.24626 (QuantReg: 13.20031) QuantErr: 13.20031 batch_time=0.68210
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.98714 (QuantReg: 12.95722) QuantErr: 12.95722 batch_time=0.69827
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.73188 (QuantReg: 13.63284) QuantErr: 13.63284 batch_time=0.68266
Train Epoch: 9 codebook_update_time=2.01511
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch9.pth ...
Done in 10.601s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch9.pth ...
Done in 20.446s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.9555841941833496
quant_reg : 12.95854084777832
quant_err : 12.95854084777832
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.068
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.24550067852522
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.3765
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.59242823522422
mnt_best : 41.24550067852522
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.56730 (QuantReg: 13.05787) QuantErr: 13.05787 batch_time=41.41368
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.75459 (QuantReg: 12.79918) QuantErr: 12.79918 batch_time=0.68467
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.85192 (QuantReg: 13.07512) QuantErr: 13.07512 batch_time=6.98280
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.84410 (QuantReg: 12.98655) QuantErr: 12.98655 batch_time=0.68863
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.85532 (QuantReg: 13.16661) QuantErr: 13.16661 batch_time=0.66255
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.03436 (QuantReg: 13.01301) QuantErr: 13.01301 batch_time=0.66131
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.45372 (QuantReg: 13.16368) QuantErr: 13.16368 batch_time=1.05695
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.47569 (QuantReg: 12.89957) QuantErr: 12.89957 batch_time=0.70787
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.79065 (QuantReg: 12.91309) QuantErr: 12.91309 batch_time=0.88069
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.30891 (QuantReg: 13.13867) QuantErr: 13.13867 batch_time=0.66449
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.30504 (QuantReg: 13.15297) QuantErr: 13.15297 batch_time=0.65718
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.26490 (QuantReg: 13.27831) QuantErr: 13.27831 batch_time=0.69616
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.07814 (QuantReg: 13.06802) QuantErr: 13.06802 batch_time=0.67249
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.90023 (QuantReg: 12.68777) QuantErr: 12.68777 batch_time=3.53264
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.90486 (QuantReg: 13.11092) QuantErr: 13.11092 batch_time=0.67626
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.03468 (QuantReg: 13.06591) QuantErr: 13.06591 batch_time=0.71850
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.11661 (QuantReg: 12.97004) QuantErr: 12.97004 batch_time=0.68472
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.16705 (QuantReg: 12.97047) QuantErr: 12.97047 batch_time=0.79297
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.85437 (QuantReg: 13.11772) QuantErr: 13.11772 batch_time=0.70285
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.41050 (QuantReg: 12.94862) QuantErr: 12.94862 batch_time=1.47114
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.73957 (QuantReg: 13.38424) QuantErr: 13.38424 batch_time=0.67250
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.17801 (QuantReg: 13.04232) QuantErr: 13.04232 batch_time=0.66859
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.85158 (QuantReg: 13.13027) QuantErr: 13.13027 batch_time=0.65021
Train Epoch: 10 codebook_update_time=1.75188
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch10.pth ...
Done in 12.247s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.8418220162391663
quant_reg : 13.030643161773682
quant_err : 13.030643161773682
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.031
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.73092888578557
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.41
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.965246774581246
mnt_best : 41.24550067852522
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.63531 (QuantReg: 12.98085) QuantErr: 12.98085 batch_time=32.70881
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.02283 (QuantReg: 13.17005) QuantErr: 13.17005 batch_time=0.69694
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.67630 (QuantReg: 13.27692) QuantErr: 13.27692 batch_time=0.67408
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.38226 (QuantReg: 13.42606) QuantErr: 13.42606 batch_time=0.66047
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.88246 (QuantReg: 12.93668) QuantErr: 12.93668 batch_time=0.66610
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.66848 (QuantReg: 13.01418) QuantErr: 13.01418 batch_time=0.68351
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.43695 (QuantReg: 13.07819) QuantErr: 13.07819 batch_time=1.06355
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.64399 (QuantReg: 12.67999) QuantErr: 12.67999 batch_time=2.29633
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.71769 (QuantReg: 13.13142) QuantErr: 13.13142 batch_time=0.65302
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.08170 (QuantReg: 13.00027) QuantErr: 13.00027 batch_time=0.66931
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.54992 (QuantReg: 13.30259) QuantErr: 13.30259 batch_time=0.73757
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.79663 (QuantReg: 12.97376) QuantErr: 12.97376 batch_time=0.67582
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.60772 (QuantReg: 12.90645) QuantErr: 12.90645 batch_time=0.69969
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.78623 (QuantReg: 13.37091) QuantErr: 13.37091 batch_time=1.09362
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.58040 (QuantReg: 13.31945) QuantErr: 13.31945 batch_time=0.68837
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.83462 (QuantReg: 13.56709) QuantErr: 13.56709 batch_time=0.69988
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.48447 (QuantReg: 13.38585) QuantErr: 13.38585 batch_time=0.83196
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.89874 (QuantReg: 13.47739) QuantErr: 13.47739 batch_time=0.68050
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.79637 (QuantReg: 13.30304) QuantErr: 13.30304 batch_time=0.69441
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.52495 (QuantReg: 13.08452) QuantErr: 13.08452 batch_time=1.93805
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.54371 (QuantReg: 13.24548) QuantErr: 13.24548 batch_time=0.88646
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.45085 (QuantReg: 13.27028) QuantErr: 13.27028 batch_time=0.70198
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.64957 (QuantReg: 13.26912) QuantErr: 13.26912 batch_time=0.64993
Train Epoch: 11 codebook_update_time=2.57269
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch11.pth ...
Done in 10.838s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.732420111656189
quant_reg : 13.092788791656494
quant_err : 13.092788791656494
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.774
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.86875661451874
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.077
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.477494934899674
mnt_best : 41.24550067852522
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.93456 (QuantReg: 12.94293) QuantErr: 12.94293 batch_time=28.90318
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.98956 (QuantReg: 13.01849) QuantErr: 13.01849 batch_time=0.79900
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.60024 (QuantReg: 13.02558) QuantErr: 13.02558 batch_time=0.73684
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.76839 (QuantReg: 12.75871) QuantErr: 12.75871 batch_time=0.67510
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.78005 (QuantReg: 12.95717) QuantErr: 12.95717 batch_time=0.95086
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.66656 (QuantReg: 12.87292) QuantErr: 12.87292 batch_time=0.71744
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.66866 (QuantReg: 13.15747) QuantErr: 13.15747 batch_time=0.70423
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.22817 (QuantReg: 12.65633) QuantErr: 12.65633 batch_time=0.66234
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.69043 (QuantReg: 12.84387) QuantErr: 12.84387 batch_time=0.67176
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.97508 (QuantReg: 12.71885) QuantErr: 12.71885 batch_time=0.69728
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.70008 (QuantReg: 12.97799) QuantErr: 12.97799 batch_time=0.67104
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.75434 (QuantReg: 13.01962) QuantErr: 13.01962 batch_time=0.66832
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.71787 (QuantReg: 13.40906) QuantErr: 13.40906 batch_time=1.16000
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.57455 (QuantReg: 13.35262) QuantErr: 13.35262 batch_time=0.96128
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.43631 (QuantReg: 13.19624) QuantErr: 13.19624 batch_time=0.69954
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.53452 (QuantReg: 12.89735) QuantErr: 12.89735 batch_time=0.67885
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.58105 (QuantReg: 13.39559) QuantErr: 13.39559 batch_time=0.68124
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.39386 (QuantReg: 13.22013) QuantErr: 13.22013 batch_time=0.69030
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.76834 (QuantReg: 13.11258) QuantErr: 13.11258 batch_time=0.64830
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.94220 (QuantReg: 13.17049) QuantErr: 13.17049 batch_time=0.84185
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.37268 (QuantReg: 13.15080) QuantErr: 13.15080 batch_time=0.68251
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.96040 (QuantReg: 13.07991) QuantErr: 13.07991 batch_time=0.67382
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.32240 (QuantReg: 13.21557) QuantErr: 13.21557 batch_time=0.73687
Train Epoch: 12 codebook_update_time=1.71677
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch12.pth ...
Done in 11.474s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch12.pth ...
Done in 22.617s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.662465238571167
quant_reg : 13.08859691619873
quant_err : 13.08859691619873
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.387
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.979924267890894
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.265
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.58206979050199
mnt_best : 41.979924267890894
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.53495 (QuantReg: 13.17280) QuantErr: 13.17280 batch_time=55.87834
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.36079 (QuantReg: 12.74875) QuantErr: 12.74875 batch_time=0.65085
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.34280 (QuantReg: 13.38146) QuantErr: 13.38146 batch_time=0.77318
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.38978 (QuantReg: 13.18074) QuantErr: 13.18074 batch_time=0.77579
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.47547 (QuantReg: 13.03444) QuantErr: 13.03444 batch_time=0.68630
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.81621 (QuantReg: 13.09687) QuantErr: 13.09687 batch_time=0.69333
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.85001 (QuantReg: 13.01064) QuantErr: 13.01064 batch_time=0.71349
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.50828 (QuantReg: 13.07046) QuantErr: 13.07046 batch_time=0.65563
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.34386 (QuantReg: 13.22663) QuantErr: 13.22663 batch_time=0.67244
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.69107 (QuantReg: 13.28848) QuantErr: 13.28848 batch_time=0.66212
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.80889 (QuantReg: 13.06190) QuantErr: 13.06190 batch_time=0.69712
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.41982 (QuantReg: 13.24881) QuantErr: 13.24881 batch_time=0.65216
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.50093 (QuantReg: 13.56364) QuantErr: 13.56364 batch_time=1.50287
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 2.01440 (QuantReg: 13.39202) QuantErr: 13.39202 batch_time=0.68047
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.75475 (QuantReg: 12.96403) QuantErr: 12.96403 batch_time=0.71456
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.59944 (QuantReg: 12.93272) QuantErr: 12.93272 batch_time=0.65646
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.42257 (QuantReg: 13.57654) QuantErr: 13.57654 batch_time=0.82207
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.26970 (QuantReg: 13.65975) QuantErr: 13.65975 batch_time=0.69177
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.06218 (QuantReg: 13.30068) QuantErr: 13.30068 batch_time=0.70138
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.73086 (QuantReg: 13.24579) QuantErr: 13.24579 batch_time=0.66589
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.77847 (QuantReg: 13.38855) QuantErr: 13.38855 batch_time=0.68624
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.21703 (QuantReg: 13.27078) QuantErr: 13.27078 batch_time=0.65536
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.21918 (QuantReg: 13.44851) QuantErr: 13.44851 batch_time=0.69751
Train Epoch: 13 codebook_update_time=1.74880
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch13.pth ...
Done in 11.922s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch13.pth ...
Done in 22.706s
removing stale ckpt [epoch 12] [took 0.04s]
epoch : 13
loss : 1.5876945526599884
quant_reg : 13.18521439743042
quant_err : 13.18521439743042
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.864
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.73913853770691
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.0745
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.46144230747454
mnt_best : 42.73913853770691
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.82980 (QuantReg: 12.86997) QuantErr: 12.86997 batch_time=46.32866
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.64474 (QuantReg: 13.45485) QuantErr: 13.45485 batch_time=0.77969
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.75441 (QuantReg: 13.13675) QuantErr: 13.13675 batch_time=5.06946
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.63841 (QuantReg: 12.88261) QuantErr: 12.88261 batch_time=0.69152
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.80605 (QuantReg: 13.26412) QuantErr: 13.26412 batch_time=0.78104
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.32961 (QuantReg: 13.45867) QuantErr: 13.45867 batch_time=0.66956
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.62956 (QuantReg: 12.71856) QuantErr: 12.71856 batch_time=0.66933
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.70472 (QuantReg: 12.98042) QuantErr: 12.98042 batch_time=9.11781
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.59269 (QuantReg: 13.31730) QuantErr: 13.31730 batch_time=0.72864
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.26199 (QuantReg: 13.01326) QuantErr: 13.01326 batch_time=0.66934
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.53338 (QuantReg: 13.20979) QuantErr: 13.20979 batch_time=0.66973
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.54985 (QuantReg: 13.22293) QuantErr: 13.22293 batch_time=0.69860
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.39698 (QuantReg: 13.18803) QuantErr: 13.18803 batch_time=0.68819
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.64346 (QuantReg: 13.16248) QuantErr: 13.16248 batch_time=0.81374
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.32577 (QuantReg: 13.30675) QuantErr: 13.30675 batch_time=0.71191
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.23464 (QuantReg: 13.32730) QuantErr: 13.32730 batch_time=0.67417
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.48810 (QuantReg: 13.32815) QuantErr: 13.32815 batch_time=0.72107
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.09813 (QuantReg: 13.13550) QuantErr: 13.13550 batch_time=0.82679
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.39766 (QuantReg: 13.42980) QuantErr: 13.42980 batch_time=0.72531
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.39406 (QuantReg: 12.77509) QuantErr: 12.77509 batch_time=0.66238
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.20672 (QuantReg: 13.32870) QuantErr: 13.32870 batch_time=1.45840
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.56298 (QuantReg: 13.31357) QuantErr: 13.31357 batch_time=0.65582
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.55628 (QuantReg: 13.39360) QuantErr: 13.39360 batch_time=0.71237
Train Epoch: 14 codebook_update_time=1.72540
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch14.pth ...
Done in 11.169s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch14.pth ...
Done in 22.405s
removing stale ckpt [epoch 13] [took 0.03s]
epoch : 14
loss : 1.5497506308555602
quant_reg : 13.227337394714356
quant_err : 13.227337394714356
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.657
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.078954991150574
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.348
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.49027130830663
mnt_best : 43.078954991150574
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.60673 (QuantReg: 13.08305) QuantErr: 13.08305 batch_time=43.20338
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.46315 (QuantReg: 13.14190) QuantErr: 13.14190 batch_time=0.72696
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.26776 (QuantReg: 13.29253) QuantErr: 13.29253 batch_time=0.67278
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.60742 (QuantReg: 13.64537) QuantErr: 13.64537 batch_time=0.75791
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.48636 (QuantReg: 13.10649) QuantErr: 13.10649 batch_time=0.73699
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.58639 (QuantReg: 13.27367) QuantErr: 13.27367 batch_time=0.72034
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.15520 (QuantReg: 13.20648) QuantErr: 13.20648 batch_time=5.91605
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.33757 (QuantReg: 13.32266) QuantErr: 13.32266 batch_time=0.67937
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.41709 (QuantReg: 13.63945) QuantErr: 13.63945 batch_time=0.71030
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.40691 (QuantReg: 13.14390) QuantErr: 13.14390 batch_time=0.68305
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.48253 (QuantReg: 12.80666) QuantErr: 12.80666 batch_time=0.75873
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.37509 (QuantReg: 13.02249) QuantErr: 13.02249 batch_time=0.75957
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.32180 (QuantReg: 13.53933) QuantErr: 13.53933 batch_time=0.78472
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.40724 (QuantReg: 13.38583) QuantErr: 13.38583 batch_time=0.75232
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.60707 (QuantReg: 13.24297) QuantErr: 13.24297 batch_time=0.63085
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.17240 (QuantReg: 13.48298) QuantErr: 13.48298 batch_time=0.70572
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.14157 (QuantReg: 13.59701) QuantErr: 13.59701 batch_time=0.68175
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.61488 (QuantReg: 13.44520) QuantErr: 13.44520 batch_time=0.70589
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.47734 (QuantReg: 13.57072) QuantErr: 13.57072 batch_time=0.70274
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.37351 (QuantReg: 13.34203) QuantErr: 13.34203 batch_time=1.13384
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.67944 (QuantReg: 13.38450) QuantErr: 13.38450 batch_time=0.64962
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.71614 (QuantReg: 12.86998) QuantErr: 12.86998 batch_time=0.69376
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.17582 (QuantReg: 13.11736) QuantErr: 13.11736 batch_time=0.73705
Train Epoch: 15 codebook_update_time=1.75183
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch15.pth ...
Done in 28.719s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch15.pth ...
Done in 40.261s
removing stale ckpt [epoch 14] [took 0.04s]
epoch : 15
loss : 1.486278640985489
quant_reg : 13.262633167266845
quant_err : 13.262633167266845
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.741
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.81436946431534
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.2225
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.72852126642668
mnt_best : 43.81436946431534
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.52697 (QuantReg: 12.72495) QuantErr: 12.72495 batch_time=55.05464
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.22866 (QuantReg: 13.53498) QuantErr: 13.53498 batch_time=6.41629
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.50029 (QuantReg: 13.35563) QuantErr: 13.35563 batch_time=0.64986
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.37912 (QuantReg: 13.16278) QuantErr: 13.16278 batch_time=0.68731
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.12874 (QuantReg: 13.51981) QuantErr: 13.51981 batch_time=0.85084
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.42865 (QuantReg: 13.32908) QuantErr: 13.32908 batch_time=0.72809
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.32560 (QuantReg: 13.15163) QuantErr: 13.15163 batch_time=0.63893
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.58961 (QuantReg: 13.02431) QuantErr: 13.02431 batch_time=0.65009
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.65973 (QuantReg: 12.73236) QuantErr: 12.73236 batch_time=0.70835
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.36518 (QuantReg: 13.22608) QuantErr: 13.22608 batch_time=0.68780
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.74784 (QuantReg: 13.36645) QuantErr: 13.36645 batch_time=0.65711
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.34857 (QuantReg: 13.16656) QuantErr: 13.16656 batch_time=0.68249
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.53109 (QuantReg: 13.31872) QuantErr: 13.31872 batch_time=0.68505
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.60845 (QuantReg: 13.33637) QuantErr: 13.33637 batch_time=6.59515
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.48669 (QuantReg: 13.07620) QuantErr: 13.07620 batch_time=0.65278
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.51539 (QuantReg: 13.50965) QuantErr: 13.50965 batch_time=0.65307
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.24529 (QuantReg: 13.18629) QuantErr: 13.18629 batch_time=0.72704
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.08499 (QuantReg: 13.45058) QuantErr: 13.45058 batch_time=0.73998
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.30010 (QuantReg: 13.44621) QuantErr: 13.44621 batch_time=0.70662
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.54961 (QuantReg: 13.44378) QuantErr: 13.44378 batch_time=0.65196
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.65271 (QuantReg: 13.40457) QuantErr: 13.40457 batch_time=0.68197
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.44769 (QuantReg: 12.95922) QuantErr: 12.95922 batch_time=0.87748
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.22554 (QuantReg: 13.39072) QuantErr: 13.39072 batch_time=0.65533
Train Epoch: 16 codebook_update_time=2.00094
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch16.pth ...
Done in 11.370s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch16.pth ...
Done in 22.734s
removing stale ckpt [epoch 15] [took 0.11s]
epoch : 16
loss : 1.440460570335388
quant_reg : 13.311044330596923
quant_err : 13.311044330596923
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.776
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 44.255305012624525
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 69.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.908
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.475705421795766
mnt_best : 44.255305012624525
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.25187 (QuantReg: 13.06203) QuantErr: 13.06203 batch_time=56.16811
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.40524 (QuantReg: 12.89142) QuantErr: 12.89142 batch_time=8.46956
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.23669 (QuantReg: 13.40396) QuantErr: 13.40396 batch_time=0.66225
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.41041 (QuantReg: 12.79893) QuantErr: 12.79893 batch_time=1.71978
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.54115 (QuantReg: 13.24148) QuantErr: 13.24148 batch_time=0.67689
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.66780 (QuantReg: 13.27089) QuantErr: 13.27089 batch_time=0.63995
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.31190 (QuantReg: 13.47788) QuantErr: 13.47788 batch_time=0.65452
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.53979 (QuantReg: 13.20385) QuantErr: 13.20385 batch_time=0.66296
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.11138 (QuantReg: 13.33981) QuantErr: 13.33981 batch_time=0.65423
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.18701 (QuantReg: 13.28192) QuantErr: 13.28192 batch_time=0.67544
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.23455 (QuantReg: 13.13732) QuantErr: 13.13732 batch_time=0.68767
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.19945 (QuantReg: 13.28523) QuantErr: 13.28523 batch_time=0.73363
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.44107 (QuantReg: 13.31021) QuantErr: 13.31021 batch_time=0.69551
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.45560 (QuantReg: 13.20295) QuantErr: 13.20295 batch_time=0.65191
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.37702 (QuantReg: 13.33696) QuantErr: 13.33696 batch_time=0.70397
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.44853 (QuantReg: 13.57499) QuantErr: 13.57499 batch_time=0.65965
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.05190 (QuantReg: 13.80608) QuantErr: 13.80608 batch_time=1.48415
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.72511 (QuantReg: 13.55553) QuantErr: 13.55553 batch_time=0.73085
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.32251 (QuantReg: 13.36535) QuantErr: 13.36535 batch_time=0.92466
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.08047 (QuantReg: 13.91802) QuantErr: 13.91802 batch_time=0.77819
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.24535 (QuantReg: 13.63052) QuantErr: 13.63052 batch_time=0.68178
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.15385 (QuantReg: 13.53104) QuantErr: 13.53104 batch_time=0.67330
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.16621 (QuantReg: 13.25293) QuantErr: 13.25293 batch_time=0.71303
Train Epoch: 17 codebook_update_time=1.68509
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch17.pth ...
Done in 11.578s
removing stale ckpt [epoch 16] [took 0.04s]
epoch : 17
loss : 1.3972294182777405
quant_reg : 13.34130435180664
quant_err : 13.34130435180664
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.148
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.62825091604727
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.462
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.6718528761103
mnt_best : 44.255305012624525
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.50778 (QuantReg: 13.46699) QuantErr: 13.46699 batch_time=55.77780
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.15248 (QuantReg: 13.37426) QuantErr: 13.37426 batch_time=12.21836
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.18552 (QuantReg: 13.30410) QuantErr: 13.30410 batch_time=0.68847
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.38097 (QuantReg: 13.35557) QuantErr: 13.35557 batch_time=0.76838
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.34960 (QuantReg: 13.42922) QuantErr: 13.42922 batch_time=0.69789
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.18938 (QuantReg: 13.29171) QuantErr: 13.29171 batch_time=0.76995
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.21929 (QuantReg: 13.19421) QuantErr: 13.19421 batch_time=0.65289
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.21639 (QuantReg: 13.24880) QuantErr: 13.24880 batch_time=0.65281
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.18658 (QuantReg: 13.61812) QuantErr: 13.61812 batch_time=0.72827
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.30622 (QuantReg: 13.54189) QuantErr: 13.54189 batch_time=0.66120
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.25109 (QuantReg: 13.28003) QuantErr: 13.28003 batch_time=0.68970
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.29088 (QuantReg: 13.60385) QuantErr: 13.60385 batch_time=0.67859
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.16894 (QuantReg: 13.44320) QuantErr: 13.44320 batch_time=0.68981
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.57236 (QuantReg: 13.48923) QuantErr: 13.48923 batch_time=1.04926
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.29400 (QuantReg: 13.37517) QuantErr: 13.37517 batch_time=0.69620
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.63080 (QuantReg: 13.39686) QuantErr: 13.39686 batch_time=0.73111
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.59609 (QuantReg: 13.58225) QuantErr: 13.58225 batch_time=0.69933
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.28146 (QuantReg: 13.46258) QuantErr: 13.46258 batch_time=0.71081
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.66853 (QuantReg: 13.30685) QuantErr: 13.30685 batch_time=0.68053
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.53679 (QuantReg: 13.48382) QuantErr: 13.48382 batch_time=0.85705
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.35644 (QuantReg: 13.34972) QuantErr: 13.34972 batch_time=0.65636
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.49606 (QuantReg: 13.16798) QuantErr: 13.16798 batch_time=0.67414
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.47100 (QuantReg: 13.50450) QuantErr: 13.50450 batch_time=0.71575
Train Epoch: 18 codebook_update_time=1.72827
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch18.pth ...
Done in 11.065s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch18.pth ...
Done in 22.115s
removing stale ckpt [epoch 17] [took 0.00s]
epoch : 18
loss : 1.3647120082378388
quant_reg : 13.393117729187011
quant_err : 13.393117729187011
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 25.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.541
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 45.2138382068709
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 69.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.1085
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.72672372322402
mnt_best : 45.2138382068709
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.15624 (QuantReg: 13.22382) QuantErr: 13.22382 batch_time=105.13889
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.17746 (QuantReg: 13.48020) QuantErr: 13.48020 batch_time=0.69004
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.27976 (QuantReg: 13.42569) QuantErr: 13.42569 batch_time=0.74874
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.37659 (QuantReg: 13.57086) QuantErr: 13.57086 batch_time=0.75934
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.20128 (QuantReg: 13.38230) QuantErr: 13.38230 batch_time=0.66786
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.50558 (QuantReg: 13.56650) QuantErr: 13.56650 batch_time=0.73356
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.41702 (QuantReg: 13.53015) QuantErr: 13.53015 batch_time=3.26972
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.16101 (QuantReg: 13.62296) QuantErr: 13.62296 batch_time=0.66348
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.13243 (QuantReg: 13.52678) QuantErr: 13.52678 batch_time=0.69437
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.38599 (QuantReg: 13.16551) QuantErr: 13.16551 batch_time=0.66393
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.57142 (QuantReg: 13.51797) QuantErr: 13.51797 batch_time=0.67475
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.72902 (QuantReg: 13.36709) QuantErr: 13.36709 batch_time=0.68876
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.32830 (QuantReg: 13.64314) QuantErr: 13.64314 batch_time=0.67822
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.20137 (QuantReg: 13.78611) QuantErr: 13.78611 batch_time=0.70015
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.40604 (QuantReg: 13.45343) QuantErr: 13.45343 batch_time=0.65731
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.38357 (QuantReg: 13.08326) QuantErr: 13.08326 batch_time=0.69744
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.19126 (QuantReg: 13.75674) QuantErr: 13.75674 batch_time=0.66564
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.19638 (QuantReg: 13.81817) QuantErr: 13.81817 batch_time=0.66953
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.40055 (QuantReg: 13.34833) QuantErr: 13.34833 batch_time=0.70706
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.60709 (QuantReg: 13.41084) QuantErr: 13.41084 batch_time=0.67116
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.49625 (QuantReg: 13.67574) QuantErr: 13.67574 batch_time=0.66858
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.14326 (QuantReg: 13.37918) QuantErr: 13.37918 batch_time=0.71746
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.25367 (QuantReg: 13.13838) QuantErr: 13.13838 batch_time=0.67644
Train Epoch: 19 codebook_update_time=1.95948
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bert-large/checkpoint-epoch19.pth ...
Done in 11.245s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 1.3168656113147736
quant_reg : 13.469576217651367
quant_err : 13.469576217651367
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.87
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 45.20046466225533
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0