-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_roberta-large.txt
2662 lines (2662 loc) · 199 KB
/
HCQ_MSRVTT_1kA_roberta-large.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 302.98045659065247 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 39.47932505607605 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 19.7394437789917 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch0.pth ...
Done in 4.442s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch0.pth ...
Done in 8.588s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 499.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 501.354
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 4.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 504.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 506.6715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.3419951893353394
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.76782 (QuantReg: 22.36271) QuantErr: 22.36271 batch_time=30.43627
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.65356 (QuantReg: 22.49701) QuantErr: 22.49701 batch_time=0.65003
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.82615 (QuantReg: 22.55971) QuantErr: 22.55971 batch_time=0.70783
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.81435 (QuantReg: 22.52109) QuantErr: 22.52109 batch_time=0.65173
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.01308 (QuantReg: 22.58470) QuantErr: 22.58470 batch_time=0.65653
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.60404 (QuantReg: 22.64503) QuantErr: 22.64503 batch_time=0.67475
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.06546 (QuantReg: 22.60925) QuantErr: 22.60925 batch_time=0.65836
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.99833 (QuantReg: 22.62748) QuantErr: 22.62748 batch_time=0.64958
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 6.30074 (QuantReg: 22.62539) QuantErr: 22.62539 batch_time=0.65721
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.49864 (QuantReg: 22.58223) QuantErr: 22.58223 batch_time=0.65392
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.02890 (QuantReg: 22.65191) QuantErr: 22.65191 batch_time=0.64539
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.25348 (QuantReg: 22.63950) QuantErr: 22.63950 batch_time=0.67224
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.35946 (QuantReg: 22.59830) QuantErr: 22.59830 batch_time=3.97379
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.34042 (QuantReg: 22.59918) QuantErr: 22.59918 batch_time=0.65304
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.66788 (QuantReg: 22.60771) QuantErr: 22.60771 batch_time=0.67272
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.12520 (QuantReg: 22.62438) QuantErr: 22.62438 batch_time=0.66681
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.27821 (QuantReg: 22.61075) QuantErr: 22.61075 batch_time=0.65281
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.45425 (QuantReg: 22.62991) QuantErr: 22.62991 batch_time=0.64413
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.95940 (QuantReg: 22.62210) QuantErr: 22.62210 batch_time=0.65272
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 404.4636583328247 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 32.85805892944336 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 29.347679615020752 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch0.pth ...
Done in 7.424s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch0.pth ...
Done in 12.499s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 499.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 501.354
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 4.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 504.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 506.6715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.3419951893353394
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.76909 (QuantReg: 22.36266) QuantErr: 22.36266 batch_time=34.37646
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.67585 (QuantReg: 22.48597) QuantErr: 22.48597 batch_time=0.64929
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 9.00726 (QuantReg: 22.52109) QuantErr: 22.52109 batch_time=0.69604
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.84041 (QuantReg: 22.59000) QuantErr: 22.59000 batch_time=0.65098
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.35815 (QuantReg: 22.62373) QuantErr: 22.62373 batch_time=0.66234
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.79918 (QuantReg: 22.66489) QuantErr: 22.66489 batch_time=0.66349
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.99776 (QuantReg: 22.65634) QuantErr: 22.65634 batch_time=0.65743
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.05924 (QuantReg: 22.67007) QuantErr: 22.67007 batch_time=0.64452
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 6.03330 (QuantReg: 22.61390) QuantErr: 22.61390 batch_time=0.68018
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.56706 (QuantReg: 22.62187) QuantErr: 22.62187 batch_time=0.66213
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.02449 (QuantReg: 22.63161) QuantErr: 22.63161 batch_time=0.66693
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.18993 (QuantReg: 22.62838) QuantErr: 22.62838 batch_time=0.65048
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.34551 (QuantReg: 22.64740) QuantErr: 22.64740 batch_time=0.64921
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.15832 (QuantReg: 22.62849) QuantErr: 22.62849 batch_time=0.64720
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.75132 (QuantReg: 22.61525) QuantErr: 22.61525 batch_time=0.66090
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.75677 (QuantReg: 22.63735) QuantErr: 22.63735 batch_time=0.66485
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.97633 (QuantReg: 22.62306) QuantErr: 22.62306 batch_time=0.65338
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.30936 (QuantReg: 22.62717) QuantErr: 22.62717 batch_time=0.65632
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.68645 (QuantReg: 22.62378) QuantErr: 22.62378 batch_time=0.64793
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.37750 (QuantReg: 22.62727) QuantErr: 22.62727 batch_time=0.72017
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.34833 (QuantReg: 22.63185) QuantErr: 22.63185 batch_time=0.66182
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.43315 (QuantReg: 22.62494) QuantErr: 22.62494 batch_time=0.66507
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.72529 (QuantReg: 22.62259) QuantErr: 22.62259 batch_time=0.66306
Train Epoch: 1 codebook_update_time=2.34645
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch1.pth ...
Done in 10.895s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch1.pth ...
Done in 21.653s
epoch : 1
loss : 5.825184028625488
quant_reg : 22.61285701751709
quant_err : 22.61285701751709
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 10.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 31.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 45.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 76.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 13.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 50.525
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.32936212585345
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 47.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 76.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 12.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 48.147
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.259032679867016
mnt_best : 24.32936212585345
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.39809 (QuantReg: 11.25940) QuantErr: 11.25940 batch_time=30.75743
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.53775 (QuantReg: 11.61384) QuantErr: 11.61384 batch_time=0.66346
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.45691 (QuantReg: 12.11482) QuantErr: 12.11482 batch_time=1.81512
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.85940 (QuantReg: 12.18584) QuantErr: 12.18584 batch_time=0.65424
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.62152 (QuantReg: 12.49013) QuantErr: 12.49013 batch_time=0.66365
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.86930 (QuantReg: 12.29909) QuantErr: 12.29909 batch_time=0.65459
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.03175 (QuantReg: 12.72881) QuantErr: 12.72881 batch_time=0.70215
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.43138 (QuantReg: 12.73931) QuantErr: 12.73931 batch_time=0.63795
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.84674 (QuantReg: 13.03551) QuantErr: 13.03551 batch_time=0.65656
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.94362 (QuantReg: 12.68318) QuantErr: 12.68318 batch_time=0.65272
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.79286 (QuantReg: 13.27025) QuantErr: 13.27025 batch_time=0.65236
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.58149 (QuantReg: 12.63090) QuantErr: 12.63090 batch_time=0.65073
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.80233 (QuantReg: 14.01289) QuantErr: 14.01289 batch_time=2.55398
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.51671 (QuantReg: 13.94413) QuantErr: 13.94413 batch_time=0.66227
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.12857 (QuantReg: 13.64366) QuantErr: 13.64366 batch_time=0.65160
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.42785 (QuantReg: 13.75103) QuantErr: 13.75103 batch_time=0.77269
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.50719 (QuantReg: 14.34418) QuantErr: 14.34418 batch_time=0.67364
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.39564 (QuantReg: 14.64650) QuantErr: 14.64650 batch_time=0.65097
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.10119 (QuantReg: 14.72090) QuantErr: 14.72090 batch_time=0.66154
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.64362 (QuantReg: 14.69646) QuantErr: 14.69646 batch_time=0.70136
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.36965 (QuantReg: 14.89790) QuantErr: 14.89790 batch_time=0.90141
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.01854 (QuantReg: 14.82698) QuantErr: 14.82698 batch_time=1.33078
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.05633 (QuantReg: 14.89367) QuantErr: 14.89367 batch_time=0.67118
Train Epoch: 2 codebook_update_time=1.67525
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch2.pth ...
Done in 10.316s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch2.pth ...
Done in 20.242s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 3.7586414680480957
quant_reg : 13.41098484802246
quant_err : 13.41098484802246
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 82.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 38.438
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.815021317246565
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 81.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 35.528
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.131901834666245
mnt_best : 31.815021317246565
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.33793 (QuantReg: 11.74575) QuantErr: 11.74575 batch_time=27.51015
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.08336 (QuantReg: 12.11022) QuantErr: 12.11022 batch_time=0.66114
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.24141 (QuantReg: 12.43608) QuantErr: 12.43608 batch_time=0.64869
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.31931 (QuantReg: 11.82043) QuantErr: 11.82043 batch_time=0.65088
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.01848 (QuantReg: 12.11261) QuantErr: 12.11261 batch_time=0.65386
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.83480 (QuantReg: 12.22652) QuantErr: 12.22652 batch_time=0.65846
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.88271 (QuantReg: 12.73260) QuantErr: 12.73260 batch_time=2.86358
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.35474 (QuantReg: 12.45696) QuantErr: 12.45696 batch_time=0.70522
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.07799 (QuantReg: 12.56150) QuantErr: 12.56150 batch_time=0.64893
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.66532 (QuantReg: 12.55580) QuantErr: 12.55580 batch_time=0.64952
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.17123 (QuantReg: 12.51601) QuantErr: 12.51601 batch_time=0.65804
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.44110 (QuantReg: 12.73552) QuantErr: 12.73552 batch_time=0.64739
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.52477 (QuantReg: 12.71238) QuantErr: 12.71238 batch_time=0.66012
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.81362 (QuantReg: 13.07911) QuantErr: 13.07911 batch_time=0.65064
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.20560 (QuantReg: 13.02739) QuantErr: 13.02739 batch_time=0.65453
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.84851 (QuantReg: 12.90668) QuantErr: 12.90668 batch_time=0.67078
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.43216 (QuantReg: 13.28937) QuantErr: 13.28937 batch_time=0.65091
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.33238 (QuantReg: 12.74035) QuantErr: 12.74035 batch_time=0.66976
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.81500 (QuantReg: 12.91711) QuantErr: 12.91711 batch_time=1.87527
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.51859 (QuantReg: 12.80769) QuantErr: 12.80769 batch_time=1.79450
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.65907 (QuantReg: 13.41781) QuantErr: 13.41781 batch_time=0.65441
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.92113 (QuantReg: 13.42620) QuantErr: 13.42620 batch_time=0.65638
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.06161 (QuantReg: 13.32547) QuantErr: 13.32547 batch_time=0.69242
Train Epoch: 3 codebook_update_time=1.67773
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch3.pth ...
Done in 14.444s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch3.pth ...
Done in 24.796s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.1334662494659424
quant_reg : 12.750779304504395
quant_err : 12.750779304504395
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.884
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.998607786130414
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.3245
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.683438794073155
mnt_best : 33.998607786130414
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.42576 (QuantReg: 11.84745) QuantErr: 11.84745 batch_time=29.44023
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.86820 (QuantReg: 11.87786) QuantErr: 11.87786 batch_time=0.67132
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.79280 (QuantReg: 12.51691) QuantErr: 12.51691 batch_time=1.46879
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.67342 (QuantReg: 12.23338) QuantErr: 12.23338 batch_time=0.67138
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.99271 (QuantReg: 12.15680) QuantErr: 12.15680 batch_time=0.69869
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.81307 (QuantReg: 12.47948) QuantErr: 12.47948 batch_time=0.65211
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.59526 (QuantReg: 12.56120) QuantErr: 12.56120 batch_time=1.11793
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.62357 (QuantReg: 12.37715) QuantErr: 12.37715 batch_time=0.64486
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.69480 (QuantReg: 12.44868) QuantErr: 12.44868 batch_time=0.67382
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.47739 (QuantReg: 12.46146) QuantErr: 12.46146 batch_time=0.67280
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.04576 (QuantReg: 12.78158) QuantErr: 12.78158 batch_time=0.69031
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.09417 (QuantReg: 12.51769) QuantErr: 12.51769 batch_time=0.66075
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.48159 (QuantReg: 12.64845) QuantErr: 12.64845 batch_time=0.67167
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.63324 (QuantReg: 12.80123) QuantErr: 12.80123 batch_time=3.06551
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.67054 (QuantReg: 12.64628) QuantErr: 12.64628 batch_time=0.66870
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.33532 (QuantReg: 12.41361) QuantErr: 12.41361 batch_time=0.69638
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.02293 (QuantReg: 12.58691) QuantErr: 12.58691 batch_time=0.66180
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.75489 (QuantReg: 12.86849) QuantErr: 12.86849 batch_time=0.65957
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.84791 (QuantReg: 13.13184) QuantErr: 13.13184 batch_time=1.06227
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.64554 (QuantReg: 12.73939) QuantErr: 12.73939 batch_time=0.66207
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.57533 (QuantReg: 12.98299) QuantErr: 12.98299 batch_time=0.65166
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.40862 (QuantReg: 12.84515) QuantErr: 12.84515 batch_time=1.33455
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.29517 (QuantReg: 12.86559) QuantErr: 12.86559 batch_time=0.70188
Train Epoch: 4 codebook_update_time=1.79016
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch4.pth ...
Done in 10.926s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch4.pth ...
Done in 21.170s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.7517209067344663
quant_reg : 12.618667369842528
quant_err : 12.618667369842528
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.975
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.01195999426819
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.3445
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.31120327497055
mnt_best : 35.01195999426819
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.08821 (QuantReg: 12.15757) QuantErr: 12.15757 batch_time=37.93470
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.50855 (QuantReg: 12.25694) QuantErr: 12.25694 batch_time=1.23900
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.65024 (QuantReg: 12.64763) QuantErr: 12.64763 batch_time=0.67646
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.43166 (QuantReg: 12.77713) QuantErr: 12.77713 batch_time=0.89755
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.02286 (QuantReg: 12.61595) QuantErr: 12.61595 batch_time=0.66171
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.54093 (QuantReg: 12.33778) QuantErr: 12.33778 batch_time=0.65062
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.60950 (QuantReg: 13.04116) QuantErr: 13.04116 batch_time=0.65872
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.52200 (QuantReg: 12.95210) QuantErr: 12.95210 batch_time=0.66353
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.41811 (QuantReg: 12.56912) QuantErr: 12.56912 batch_time=0.70320
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.18983 (QuantReg: 13.41571) QuantErr: 13.41571 batch_time=0.69828
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.60739 (QuantReg: 12.74454) QuantErr: 12.74454 batch_time=0.71217
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.95997 (QuantReg: 12.64476) QuantErr: 12.64476 batch_time=0.66043
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.11153 (QuantReg: 12.89103) QuantErr: 12.89103 batch_time=0.81865
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.32228 (QuantReg: 12.90726) QuantErr: 12.90726 batch_time=1.29505
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.39300 (QuantReg: 13.28794) QuantErr: 13.28794 batch_time=0.73139
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.33285 (QuantReg: 13.25522) QuantErr: 13.25522 batch_time=0.67174
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.49421 (QuantReg: 12.88857) QuantErr: 12.88857 batch_time=0.66983
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 1.93631 (QuantReg: 13.57724) QuantErr: 13.57724 batch_time=0.67022
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.34465 (QuantReg: 13.36282) QuantErr: 13.36282 batch_time=1.02487
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.13763 (QuantReg: 13.33301) QuantErr: 13.33301 batch_time=1.43334
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.28206 (QuantReg: 13.06593) QuantErr: 13.06593 batch_time=0.68155
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.32021 (QuantReg: 12.94831) QuantErr: 12.94831 batch_time=0.69634
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.16831 (QuantReg: 13.19892) QuantErr: 13.19892 batch_time=0.69630
Train Epoch: 5 codebook_update_time=1.93058
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch5.pth ...
Done in 11.551s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch5.pth ...
Done in 22.706s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.4899971709251405
quant_reg : 12.899170501708983
quant_err : 12.899170501708983
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.9
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.3833538990088
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.468
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.18820491660969
mnt_best : 36.3833538990088
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.48162 (QuantReg: 12.63422) QuantErr: 12.63422 batch_time=49.68875
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.76200 (QuantReg: 12.50474) QuantErr: 12.50474 batch_time=0.65895
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 1.88031 (QuantReg: 12.45931) QuantErr: 12.45931 batch_time=0.69453
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.36475 (QuantReg: 12.96870) QuantErr: 12.96870 batch_time=0.65415
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.26797 (QuantReg: 12.95778) QuantErr: 12.95778 batch_time=0.66079
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.54908 (QuantReg: 12.92240) QuantErr: 12.92240 batch_time=0.65031
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 1.84331 (QuantReg: 12.91935) QuantErr: 12.91935 batch_time=0.67052
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.17165 (QuantReg: 12.91395) QuantErr: 12.91395 batch_time=0.66187
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 1.93122 (QuantReg: 12.90967) QuantErr: 12.90967 batch_time=0.68331
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.35458 (QuantReg: 12.92685) QuantErr: 12.92685 batch_time=0.65878
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.15979 (QuantReg: 12.61000) QuantErr: 12.61000 batch_time=0.65849
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.31996 (QuantReg: 12.97061) QuantErr: 12.97061 batch_time=0.66790
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 1.93840 (QuantReg: 13.10253) QuantErr: 13.10253 batch_time=0.65315
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.48135 (QuantReg: 12.77691) QuantErr: 12.77691 batch_time=0.67519
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.33151 (QuantReg: 13.14875) QuantErr: 13.14875 batch_time=0.68282
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.09545 (QuantReg: 12.95242) QuantErr: 12.95242 batch_time=0.67312
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 1.87533 (QuantReg: 13.32896) QuantErr: 13.32896 batch_time=0.70356
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.09684 (QuantReg: 13.47479) QuantErr: 13.47479 batch_time=0.66091
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.16203 (QuantReg: 13.34404) QuantErr: 13.34404 batch_time=0.66657
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.00670 (QuantReg: 13.37728) QuantErr: 13.37728 batch_time=0.65748
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.35059 (QuantReg: 13.06709) QuantErr: 13.06709 batch_time=0.71792
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.31507 (QuantReg: 13.26053) QuantErr: 13.26053 batch_time=0.67171
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.22287 (QuantReg: 13.35721) QuantErr: 13.35721 batch_time=0.67371
Train Epoch: 6 codebook_update_time=1.79654
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch6.pth ...
Done in 11.247s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch6.pth ...
Done in 22.422s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.251945685863495
quant_reg : 12.992550075531005
quant_err : 12.992550075531005
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.254
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.741001436874384
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.974
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.60082036253023
mnt_best : 37.741001436874384
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.25870 (QuantReg: 13.11625) QuantErr: 13.11625 batch_time=38.40661
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.09904 (QuantReg: 12.92030) QuantErr: 12.92030 batch_time=1.02690
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.01562 (QuantReg: 12.97165) QuantErr: 12.97165 batch_time=0.66032
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.20108 (QuantReg: 13.17317) QuantErr: 13.17317 batch_time=0.66368
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.05260 (QuantReg: 12.73668) QuantErr: 12.73668 batch_time=1.08231
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.11078 (QuantReg: 12.97747) QuantErr: 12.97747 batch_time=0.65882
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 1.79118 (QuantReg: 12.64454) QuantErr: 12.64454 batch_time=0.66067
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 1.66335 (QuantReg: 13.29062) QuantErr: 13.29062 batch_time=0.65885
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.03597 (QuantReg: 13.14596) QuantErr: 13.14596 batch_time=0.66411
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.15312 (QuantReg: 12.35673) QuantErr: 12.35673 batch_time=0.65773
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.89971 (QuantReg: 13.06555) QuantErr: 13.06555 batch_time=0.69420
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.21757 (QuantReg: 13.13260) QuantErr: 13.13260 batch_time=0.69741
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.85571 (QuantReg: 13.24255) QuantErr: 13.24255 batch_time=2.21759
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 1.87171 (QuantReg: 13.22735) QuantErr: 13.22735 batch_time=0.67129
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.92620 (QuantReg: 13.52663) QuantErr: 13.52663 batch_time=0.66912
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.23182 (QuantReg: 13.38952) QuantErr: 13.38952 batch_time=0.65675
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.86195 (QuantReg: 13.31304) QuantErr: 13.31304 batch_time=0.65118
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.15889 (QuantReg: 12.92914) QuantErr: 12.92914 batch_time=0.87245
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.92387 (QuantReg: 13.11683) QuantErr: 13.11683 batch_time=0.70131
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.72478 (QuantReg: 13.40509) QuantErr: 13.40509 batch_time=0.99398
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 1.88393 (QuantReg: 13.28214) QuantErr: 13.28214 batch_time=0.71704
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.53620 (QuantReg: 13.02670) QuantErr: 13.02670 batch_time=0.76068
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.96673 (QuantReg: 13.16356) QuantErr: 13.16356 batch_time=0.66076
Train Epoch: 7 codebook_update_time=1.87672
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch7.pth ...
Done in 11.621s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch7.pth ...
Done in 40.611s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.071570275306702
quant_reg : 13.089737552642822
quant_err : 13.089737552642822
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.026
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.66708336187225
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.169
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.1119778979061
mnt_best : 38.66708336187225
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.87101 (QuantReg: 12.90982) QuantErr: 12.90982 batch_time=31.79505
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.01726 (QuantReg: 12.95795) QuantErr: 12.95795 batch_time=0.66477
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 1.79586 (QuantReg: 12.87651) QuantErr: 12.87651 batch_time=0.68571
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.80046 (QuantReg: 13.08184) QuantErr: 13.08184 batch_time=0.64711
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.12310 (QuantReg: 12.94056) QuantErr: 12.94056 batch_time=0.70445
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.75172 (QuantReg: 12.95403) QuantErr: 12.95403 batch_time=0.66289
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.80211 (QuantReg: 13.45222) QuantErr: 13.45222 batch_time=0.65228
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.30597 (QuantReg: 12.86281) QuantErr: 12.86281 batch_time=0.68847
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.66162 (QuantReg: 13.00886) QuantErr: 13.00886 batch_time=0.65890
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.46371 (QuantReg: 12.94135) QuantErr: 12.94135 batch_time=0.65483
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.78425 (QuantReg: 13.36287) QuantErr: 13.36287 batch_time=0.65397
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.18508 (QuantReg: 13.28462) QuantErr: 13.28462 batch_time=0.68834
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.01732 (QuantReg: 13.27861) QuantErr: 13.27861 batch_time=0.71388
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.24374 (QuantReg: 13.10788) QuantErr: 13.10788 batch_time=0.74535
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.80890 (QuantReg: 13.11076) QuantErr: 13.11076 batch_time=1.06910
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.75062 (QuantReg: 13.15689) QuantErr: 13.15689 batch_time=0.64591
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.70814 (QuantReg: 13.56445) QuantErr: 13.56445 batch_time=0.67647
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.24211 (QuantReg: 13.29390) QuantErr: 13.29390 batch_time=0.66842
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.90585 (QuantReg: 13.36112) QuantErr: 13.36112 batch_time=0.64764
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.14052 (QuantReg: 13.49210) QuantErr: 13.49210 batch_time=0.66403
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.19592 (QuantReg: 13.37045) QuantErr: 13.37045 batch_time=0.65088
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.33659 (QuantReg: 13.41962) QuantErr: 13.41962 batch_time=0.65912
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.80703 (QuantReg: 13.43741) QuantErr: 13.43741 batch_time=0.65696
Train Epoch: 8 codebook_update_time=1.77398
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch8.pth ...
Done in 11.904s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch8.pth ...
Done in 23.107s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 1.9262462091445922
quant_reg : 13.169290523529053
quant_err : 13.169290523529053
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.326
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.54040623049936
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.805
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.27530451068987
mnt_best : 39.54040623049936
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.52556 (QuantReg: 13.13043) QuantErr: 13.13043 batch_time=35.48745
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.63635 (QuantReg: 13.02589) QuantErr: 13.02589 batch_time=1.14259
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.97061 (QuantReg: 13.05903) QuantErr: 13.05903 batch_time=0.69987
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.05895 (QuantReg: 12.99905) QuantErr: 12.99905 batch_time=0.65519
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.11109 (QuantReg: 12.95880) QuantErr: 12.95880 batch_time=0.66315
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.99156 (QuantReg: 13.09224) QuantErr: 13.09224 batch_time=0.66920
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.49172 (QuantReg: 13.41415) QuantErr: 13.41415 batch_time=2.12765
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.76448 (QuantReg: 13.18389) QuantErr: 13.18389 batch_time=2.52358
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.86010 (QuantReg: 13.49360) QuantErr: 13.49360 batch_time=0.66960
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.14415 (QuantReg: 13.09294) QuantErr: 13.09294 batch_time=0.67529
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.36236 (QuantReg: 13.48940) QuantErr: 13.48940 batch_time=0.66865
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.79837 (QuantReg: 13.26653) QuantErr: 13.26653 batch_time=0.78090
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.72693 (QuantReg: 13.35003) QuantErr: 13.35003 batch_time=0.70953
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.45845 (QuantReg: 13.19032) QuantErr: 13.19032 batch_time=0.66276
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.02348 (QuantReg: 13.01085) QuantErr: 13.01085 batch_time=0.67063
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.95590 (QuantReg: 13.52434) QuantErr: 13.52434 batch_time=0.67344
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.08171 (QuantReg: 13.38512) QuantErr: 13.38512 batch_time=0.67679
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.06398 (QuantReg: 13.29409) QuantErr: 13.29409 batch_time=0.68094
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.71411 (QuantReg: 13.40487) QuantErr: 13.40487 batch_time=0.65767
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.52824 (QuantReg: 13.19520) QuantErr: 13.19520 batch_time=0.66242
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.21550 (QuantReg: 13.35225) QuantErr: 13.35225 batch_time=1.94242
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.51370 (QuantReg: 13.49828) QuantErr: 13.49828 batch_time=0.70422
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.56802 (QuantReg: 13.48171) QuantErr: 13.48171 batch_time=0.66221
Train Epoch: 9 codebook_update_time=1.67283
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch9.pth ...
Done in 10.716s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.8014201350212098
quant_reg : 13.280874732971192
quant_err : 13.280874732971192
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.023
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.16954458003912
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.451
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.38152399951815
mnt_best : 39.54040623049936
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.82003 (QuantReg: 13.06542) QuantErr: 13.06542 batch_time=28.88070
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.84594 (QuantReg: 13.01998) QuantErr: 13.01998 batch_time=0.65639
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.94372 (QuantReg: 13.08697) QuantErr: 13.08697 batch_time=0.69936
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.79167 (QuantReg: 13.23608) QuantErr: 13.23608 batch_time=1.75603
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.66141 (QuantReg: 13.35734) QuantErr: 13.35734 batch_time=0.68500
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.92875 (QuantReg: 13.48288) QuantErr: 13.48288 batch_time=0.72218
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.95317 (QuantReg: 12.98320) QuantErr: 12.98320 batch_time=0.65937
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.25038 (QuantReg: 13.36190) QuantErr: 13.36190 batch_time=0.67152
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.81782 (QuantReg: 13.21548) QuantErr: 13.21548 batch_time=0.66244
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.37189 (QuantReg: 13.45844) QuantErr: 13.45844 batch_time=0.67443
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.86108 (QuantReg: 13.25869) QuantErr: 13.25869 batch_time=0.67172
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.64682 (QuantReg: 13.48801) QuantErr: 13.48801 batch_time=0.66265
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.55792 (QuantReg: 13.28144) QuantErr: 13.28144 batch_time=0.66306
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.51274 (QuantReg: 13.28653) QuantErr: 13.28653 batch_time=1.57945
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.36178 (QuantReg: 13.47135) QuantErr: 13.47135 batch_time=0.66622
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.50749 (QuantReg: 13.51989) QuantErr: 13.51989 batch_time=0.67735
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.57188 (QuantReg: 13.44793) QuantErr: 13.44793 batch_time=0.67473
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.39685 (QuantReg: 13.55948) QuantErr: 13.55948 batch_time=0.67105
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.73151 (QuantReg: 13.81938) QuantErr: 13.81938 batch_time=0.69184
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.75970 (QuantReg: 13.26334) QuantErr: 13.26334 batch_time=0.68434
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.23409 (QuantReg: 13.60370) QuantErr: 13.60370 batch_time=0.69840
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.71586 (QuantReg: 13.37436) QuantErr: 13.37436 batch_time=0.69208
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.53361 (QuantReg: 13.52200) QuantErr: 13.52200 batch_time=0.82498
Train Epoch: 10 codebook_update_time=1.80266
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch10.pth ...
Done in 13.420s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch10.pth ...
Done in 24.363s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.7021977248191833
quant_reg : 13.339704524993897
quant_err : 13.339704524993897
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.11
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.98414037930852
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.2715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.79477057159102
mnt_best : 39.98414037930852
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.43648 (QuantReg: 13.52731) QuantErr: 13.52731 batch_time=31.78048
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.39265 (QuantReg: 13.17540) QuantErr: 13.17540 batch_time=0.66872
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.46467 (QuantReg: 13.51298) QuantErr: 13.51298 batch_time=0.76144
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.85120 (QuantReg: 13.06151) QuantErr: 13.06151 batch_time=0.70764
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.59836 (QuantReg: 13.33755) QuantErr: 13.33755 batch_time=0.65381
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.43987 (QuantReg: 13.59542) QuantErr: 13.59542 batch_time=0.68704
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.51306 (QuantReg: 13.25063) QuantErr: 13.25063 batch_time=6.81163
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.14457 (QuantReg: 13.26828) QuantErr: 13.26828 batch_time=0.68202
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.38882 (QuantReg: 13.50664) QuantErr: 13.50664 batch_time=0.71408
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.07076 (QuantReg: 13.11827) QuantErr: 13.11827 batch_time=0.70788
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.78256 (QuantReg: 13.36119) QuantErr: 13.36119 batch_time=0.68516
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.62120 (QuantReg: 13.53200) QuantErr: 13.53200 batch_time=0.72628
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.45586 (QuantReg: 13.17054) QuantErr: 13.17054 batch_time=0.69745
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.74681 (QuantReg: 13.52909) QuantErr: 13.52909 batch_time=1.25635
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.91311 (QuantReg: 13.21407) QuantErr: 13.21407 batch_time=1.33694
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.64999 (QuantReg: 13.22875) QuantErr: 13.22875 batch_time=0.66092
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.54173 (QuantReg: 13.42341) QuantErr: 13.42341 batch_time=0.68833
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.52500 (QuantReg: 13.23454) QuantErr: 13.23454 batch_time=0.65372
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.46293 (QuantReg: 13.92288) QuantErr: 13.92288 batch_time=0.67834
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.47916 (QuantReg: 13.36629) QuantErr: 13.36629 batch_time=0.65899
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.72876 (QuantReg: 13.66555) QuantErr: 13.66555 batch_time=0.74755
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.26856 (QuantReg: 13.73539) QuantErr: 13.73539 batch_time=0.67432
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.71345 (QuantReg: 13.02547) QuantErr: 13.02547 batch_time=0.67846
Train Epoch: 11 codebook_update_time=1.84713
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch11.pth ...
Done in 15.859s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch11.pth ...
Done in 36.155s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.607930206298828
quant_reg : 13.359805320739746
quant_err : 13.359805320739746
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.249
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.308683417867805
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.5555
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.81710091957027
mnt_best : 41.308683417867805
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.48331 (QuantReg: 12.99961) QuantErr: 12.99961 batch_time=29.82831
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.79047 (QuantReg: 13.56208) QuantErr: 13.56208 batch_time=0.67169
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.64703 (QuantReg: 13.57298) QuantErr: 13.57298 batch_time=2.50851
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.42315 (QuantReg: 13.00154) QuantErr: 13.00154 batch_time=0.63661
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.42653 (QuantReg: 13.73501) QuantErr: 13.73501 batch_time=0.69562
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.38647 (QuantReg: 13.56740) QuantErr: 13.56740 batch_time=0.63886
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.55517 (QuantReg: 13.09123) QuantErr: 13.09123 batch_time=3.09897
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.53433 (QuantReg: 13.33111) QuantErr: 13.33111 batch_time=0.65584
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.53096 (QuantReg: 13.47641) QuantErr: 13.47641 batch_time=0.66137
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.75796 (QuantReg: 13.29938) QuantErr: 13.29938 batch_time=0.69696
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.45156 (QuantReg: 13.77416) QuantErr: 13.77416 batch_time=0.69705
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.32104 (QuantReg: 13.38738) QuantErr: 13.38738 batch_time=0.66878
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.59403 (QuantReg: 13.36352) QuantErr: 13.36352 batch_time=0.78428
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.26520 (QuantReg: 13.27565) QuantErr: 13.27565 batch_time=0.65341
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.15062 (QuantReg: 13.82791) QuantErr: 13.82791 batch_time=0.66767
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.27449 (QuantReg: 13.66372) QuantErr: 13.66372 batch_time=0.69894
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.44800 (QuantReg: 13.41560) QuantErr: 13.41560 batch_time=0.70480
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.60233 (QuantReg: 13.35867) QuantErr: 13.35867 batch_time=0.74404
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.88449 (QuantReg: 13.70960) QuantErr: 13.70960 batch_time=4.03146
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.25932 (QuantReg: 13.68826) QuantErr: 13.68826 batch_time=0.69213
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.44955 (QuantReg: 13.51648) QuantErr: 13.51648 batch_time=0.74154
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.32105 (QuantReg: 13.57793) QuantErr: 13.57793 batch_time=0.70318
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.32179 (QuantReg: 13.83147) QuantErr: 13.83147 batch_time=0.67273
Train Epoch: 12 codebook_update_time=1.76220
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch12.pth ...
Done in 11.069s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.5330995869636537
quant_reg : 13.474599723815919
quant_err : 13.474599723815919
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.187
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.276790677274185
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.8455
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.6909618169206
mnt_best : 41.308683417867805
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.43200 (QuantReg: 13.39461) QuantErr: 13.39461 batch_time=31.77802
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.41558 (QuantReg: 13.12468) QuantErr: 13.12468 batch_time=0.67895
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.42784 (QuantReg: 13.41382) QuantErr: 13.41382 batch_time=0.70164
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.30405 (QuantReg: 13.44718) QuantErr: 13.44718 batch_time=0.78948
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.61807 (QuantReg: 13.65821) QuantErr: 13.65821 batch_time=0.66238
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.63970 (QuantReg: 13.46915) QuantErr: 13.46915 batch_time=0.88372
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.71311 (QuantReg: 13.84007) QuantErr: 13.84007 batch_time=1.10794
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.34363 (QuantReg: 13.81513) QuantErr: 13.81513 batch_time=0.69729
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.56127 (QuantReg: 13.48394) QuantErr: 13.48394 batch_time=0.65582
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.44860 (QuantReg: 13.75506) QuantErr: 13.75506 batch_time=0.67776
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.64500 (QuantReg: 13.24815) QuantErr: 13.24815 batch_time=0.65255
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.19502 (QuantReg: 13.90032) QuantErr: 13.90032 batch_time=0.65810
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.44363 (QuantReg: 13.63349) QuantErr: 13.63349 batch_time=2.29187
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.25262 (QuantReg: 13.32389) QuantErr: 13.32389 batch_time=6.59868
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.44916 (QuantReg: 13.74998) QuantErr: 13.74998 batch_time=0.67914
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.42846 (QuantReg: 13.90290) QuantErr: 13.90290 batch_time=0.69650
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.37011 (QuantReg: 13.70504) QuantErr: 13.70504 batch_time=0.66264
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.37756 (QuantReg: 13.35899) QuantErr: 13.35899 batch_time=0.69158
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.50527 (QuantReg: 13.60955) QuantErr: 13.60955 batch_time=0.74513
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.62860 (QuantReg: 13.36018) QuantErr: 13.36018 batch_time=0.65857
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.41984 (QuantReg: 13.71466) QuantErr: 13.71466 batch_time=0.68646
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.09490 (QuantReg: 13.91982) QuantErr: 13.91982 batch_time=0.68681
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.70161 (QuantReg: 13.61750) QuantErr: 13.61750 batch_time=0.66421
Train Epoch: 13 codebook_update_time=1.78183
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch13.pth ...
Done in 11.094s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch13.pth ...
Done in 21.994s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.4802486577033998
quant_reg : 13.522372097015381
quant_err : 13.522372097015381
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.515
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.454747984404605
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.901
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.232171252012876
mnt_best : 41.454747984404605
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.42969 (QuantReg: 13.51307) QuantErr: 13.51307 batch_time=32.60053
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.75685 (QuantReg: 13.39105) QuantErr: 13.39105 batch_time=0.69923
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.57048 (QuantReg: 13.11016) QuantErr: 13.11016 batch_time=0.66647
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.25592 (QuantReg: 13.34387) QuantErr: 13.34387 batch_time=0.65878
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.45099 (QuantReg: 13.34859) QuantErr: 13.34859 batch_time=0.71398
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.40051 (QuantReg: 13.51556) QuantErr: 13.51556 batch_time=0.66526
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.17854 (QuantReg: 13.38540) QuantErr: 13.38540 batch_time=0.65614
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.06778 (QuantReg: 13.57257) QuantErr: 13.57257 batch_time=0.69208
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.46108 (QuantReg: 13.58277) QuantErr: 13.58277 batch_time=1.24897
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.72985 (QuantReg: 13.45620) QuantErr: 13.45620 batch_time=1.58668
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.44647 (QuantReg: 13.73777) QuantErr: 13.73777 batch_time=0.72876
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.67088 (QuantReg: 13.38900) QuantErr: 13.38900 batch_time=0.81513
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.53106 (QuantReg: 13.47215) QuantErr: 13.47215 batch_time=0.64989
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.88212 (QuantReg: 13.80937) QuantErr: 13.80937 batch_time=0.65940
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.75903 (QuantReg: 13.71489) QuantErr: 13.71489 batch_time=0.65008
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.37023 (QuantReg: 13.61047) QuantErr: 13.61047 batch_time=0.71724
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.53656 (QuantReg: 13.51280) QuantErr: 13.51280 batch_time=0.66267
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.22390 (QuantReg: 13.76628) QuantErr: 13.76628 batch_time=0.65905
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.24216 (QuantReg: 13.50550) QuantErr: 13.50550 batch_time=1.30646
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.54194 (QuantReg: 13.59399) QuantErr: 13.59399 batch_time=0.66919
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.33956 (QuantReg: 13.47973) QuantErr: 13.47973 batch_time=0.68005
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.57639 (QuantReg: 13.51877) QuantErr: 13.51877 batch_time=0.66998
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.25486 (QuantReg: 13.75838) QuantErr: 13.75838 batch_time=0.68061
Train Epoch: 14 codebook_update_time=1.75278
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch14.pth ...
Done in 27.174s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch14.pth ...
Done in 37.949s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.419533536672592
quant_reg : 13.572390232086182
quant_err : 13.572390232086182
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.868
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.85832688270351
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.517
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.07862880346705
mnt_best : 41.85832688270351
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 0.97573 (QuantReg: 13.29281) QuantErr: 13.29281 batch_time=33.85801
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.53698 (QuantReg: 13.26563) QuantErr: 13.26563 batch_time=0.66134
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.32427 (QuantReg: 13.52860) QuantErr: 13.52860 batch_time=0.66286
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.08078 (QuantReg: 13.66179) QuantErr: 13.66179 batch_time=0.71694
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.20244 (QuantReg: 13.36855) QuantErr: 13.36855 batch_time=0.68837
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.41947 (QuantReg: 13.73359) QuantErr: 13.73359 batch_time=0.66964
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.99007 (QuantReg: 13.33196) QuantErr: 13.33196 batch_time=1.45970
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.67348 (QuantReg: 13.22276) QuantErr: 13.22276 batch_time=2.17960
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.50045 (QuantReg: 13.20578) QuantErr: 13.20578 batch_time=0.66277
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.31141 (QuantReg: 13.56923) QuantErr: 13.56923 batch_time=0.65046
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.17835 (QuantReg: 13.67442) QuantErr: 13.67442 batch_time=0.67843
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.45126 (QuantReg: 13.76441) QuantErr: 13.76441 batch_time=0.66590
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.40080 (QuantReg: 13.47635) QuantErr: 13.47635 batch_time=0.92174
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.23642 (QuantReg: 13.40327) QuantErr: 13.40327 batch_time=6.45106
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.40819 (QuantReg: 13.37731) QuantErr: 13.37731 batch_time=0.65855
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.45716 (QuantReg: 13.18453) QuantErr: 13.18453 batch_time=0.78194
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.46035 (QuantReg: 13.57556) QuantErr: 13.57556 batch_time=0.79777
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.83335 (QuantReg: 13.56655) QuantErr: 13.56655 batch_time=0.67922
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.11148 (QuantReg: 13.60664) QuantErr: 13.60664 batch_time=0.65456
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.15588 (QuantReg: 14.13127) QuantErr: 14.13127 batch_time=0.64868
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.37990 (QuantReg: 13.74224) QuantErr: 13.74224 batch_time=0.73865
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.06229 (QuantReg: 13.43198) QuantErr: 13.43198 batch_time=0.71929
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.16110 (QuantReg: 13.49090) QuantErr: 13.49090 batch_time=0.69553
Train Epoch: 15 codebook_update_time=1.70952
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch15.pth ...
Done in 11.314s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch15.pth ...
Done in 36.475s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.350037858724594
quant_reg : 13.545472583770753
quant_err : 13.545472583770753
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.17
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.468066253773664
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.0505
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.45478725867523
mnt_best : 42.468066253773664
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.20068 (QuantReg: 13.38278) QuantErr: 13.38278 batch_time=33.87227
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.60406 (QuantReg: 13.36020) QuantErr: 13.36020 batch_time=0.68507
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.13485 (QuantReg: 13.59364) QuantErr: 13.59364 batch_time=0.64839
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.48764 (QuantReg: 13.61015) QuantErr: 13.61015 batch_time=0.64728
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 0.97623 (QuantReg: 13.87675) QuantErr: 13.87675 batch_time=0.65533
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.34442 (QuantReg: 13.44768) QuantErr: 13.44768 batch_time=0.66026
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.18877 (QuantReg: 13.40382) QuantErr: 13.40382 batch_time=0.69071
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.45869 (QuantReg: 13.56211) QuantErr: 13.56211 batch_time=0.65544
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.30936 (QuantReg: 13.66198) QuantErr: 13.66198 batch_time=0.78287
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.24278 (QuantReg: 13.29220) QuantErr: 13.29220 batch_time=0.65704
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.22133 (QuantReg: 13.80861) QuantErr: 13.80861 batch_time=0.68115
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.19612 (QuantReg: 13.60232) QuantErr: 13.60232 batch_time=0.66608
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.11671 (QuantReg: 13.64321) QuantErr: 13.64321 batch_time=0.67553
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.28040 (QuantReg: 13.74859) QuantErr: 13.74859 batch_time=0.66181
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.51969 (QuantReg: 13.83965) QuantErr: 13.83965 batch_time=0.65763
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.43783 (QuantReg: 13.88601) QuantErr: 13.88601 batch_time=0.65116
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.30184 (QuantReg: 13.60016) QuantErr: 13.60016 batch_time=0.67519
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.29702 (QuantReg: 13.67758) QuantErr: 13.67758 batch_time=0.97205
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.18225 (QuantReg: 13.87104) QuantErr: 13.87104 batch_time=0.65880
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.08175 (QuantReg: 13.73026) QuantErr: 13.73026 batch_time=0.68907
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.85575 (QuantReg: 13.42581) QuantErr: 13.42581 batch_time=0.66054
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.23388 (QuantReg: 13.67994) QuantErr: 13.67994 batch_time=0.65285
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.01353 (QuantReg: 13.98554) QuantErr: 13.98554 batch_time=0.67390
Train Epoch: 16 codebook_update_time=2.11549
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch16.pth ...
Done in 11.401s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 1.2746519272327423
quant_reg : 13.647896846771241
quant_err : 13.647896846771241
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.484
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.086948533970855
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.999
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.9104999809316
mnt_best : 42.468066253773664
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.29019 (QuantReg: 13.29553) QuantErr: 13.29553 batch_time=36.11693
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.56983 (QuantReg: 13.00229) QuantErr: 13.00229 batch_time=0.65638
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.18550 (QuantReg: 13.47025) QuantErr: 13.47025 batch_time=0.67856
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 0.99505 (QuantReg: 13.87357) QuantErr: 13.87357 batch_time=0.67105
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 0.91065 (QuantReg: 14.27183) QuantErr: 14.27183 batch_time=0.65342
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.14647 (QuantReg: 13.59805) QuantErr: 13.59805 batch_time=0.68971
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.13913 (QuantReg: 13.84344) QuantErr: 13.84344 batch_time=0.70835
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.29135 (QuantReg: 13.57633) QuantErr: 13.57633 batch_time=0.65416
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.27988 (QuantReg: 13.62901) QuantErr: 13.62901 batch_time=0.64816
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.34959 (QuantReg: 13.47436) QuantErr: 13.47436 batch_time=0.66272
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 0.90952 (QuantReg: 13.74181) QuantErr: 13.74181 batch_time=0.68671
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.39867 (QuantReg: 13.71536) QuantErr: 13.71536 batch_time=0.65236
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.19679 (QuantReg: 13.77260) QuantErr: 13.77260 batch_time=0.65747
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.22952 (QuantReg: 13.83451) QuantErr: 13.83451 batch_time=1.07931
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.15993 (QuantReg: 13.69444) QuantErr: 13.69444 batch_time=0.68268
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.07945 (QuantReg: 14.00143) QuantErr: 14.00143 batch_time=0.68794
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.20569 (QuantReg: 14.01400) QuantErr: 14.01400 batch_time=0.67029
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.37695 (QuantReg: 13.53988) QuantErr: 13.53988 batch_time=0.89016
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.05551 (QuantReg: 13.68895) QuantErr: 13.68895 batch_time=0.66939
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.43975 (QuantReg: 13.42564) QuantErr: 13.42564 batch_time=0.65165
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.34255 (QuantReg: 13.85077) QuantErr: 13.85077 batch_time=0.64678
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.05969 (QuantReg: 13.32149) QuantErr: 13.32149 batch_time=0.66312
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.16113 (QuantReg: 13.97775) QuantErr: 13.97775 batch_time=0.68629
Train Epoch: 17 codebook_update_time=2.18094
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch17.pth ...
Done in 10.614s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 1.247967843055725
quant_reg : 13.675139991760254
quant_err : 13.675139991760254
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.389
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.38823691722827
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.8245
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.09562021692595
mnt_best : 42.468066253773664
not_improved_count: 2
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.19796 (QuantReg: 13.76738) QuantErr: 13.76738 batch_time=33.15896
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 0.98049 (QuantReg: 13.53050) QuantErr: 13.53050 batch_time=1.35904
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.32969 (QuantReg: 13.70643) QuantErr: 13.70643 batch_time=0.64453
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.16818 (QuantReg: 13.51466) QuantErr: 13.51466 batch_time=0.67794
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.32005 (QuantReg: 13.45806) QuantErr: 13.45806 batch_time=0.66910
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.41074 (QuantReg: 13.51583) QuantErr: 13.51583 batch_time=0.65096
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.37208 (QuantReg: 13.64426) QuantErr: 13.64426 batch_time=0.68530
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.04197 (QuantReg: 13.66518) QuantErr: 13.66518 batch_time=1.17320
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.19714 (QuantReg: 13.73302) QuantErr: 13.73302 batch_time=0.68054
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 0.96007 (QuantReg: 13.90767) QuantErr: 13.90767 batch_time=0.69509
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.33743 (QuantReg: 13.36834) QuantErr: 13.36834 batch_time=0.65308
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.34110 (QuantReg: 13.78787) QuantErr: 13.78787 batch_time=0.68477
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.33801 (QuantReg: 13.89207) QuantErr: 13.89207 batch_time=0.68969
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.14135 (QuantReg: 13.91836) QuantErr: 13.91836 batch_time=0.68730
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.22587 (QuantReg: 14.01690) QuantErr: 14.01690 batch_time=0.67252
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.06972 (QuantReg: 13.68459) QuantErr: 13.68459 batch_time=0.65074
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.10694 (QuantReg: 13.83882) QuantErr: 13.83882 batch_time=0.65325
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.23145 (QuantReg: 14.00726) QuantErr: 14.00726 batch_time=0.67490
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.57568 (QuantReg: 13.60139) QuantErr: 13.60139 batch_time=0.69077
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.21628 (QuantReg: 13.64555) QuantErr: 13.64555 batch_time=0.70679
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.22297 (QuantReg: 13.87293) QuantErr: 13.87293 batch_time=0.68913
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.28264 (QuantReg: 13.88229) QuantErr: 13.88229 batch_time=0.68221
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.09777 (QuantReg: 13.93060) QuantErr: 13.93060 batch_time=7.50011
Train Epoch: 18 codebook_update_time=1.75793
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch18.pth ...
Done in 16.249s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-large/checkpoint-epoch18.pth ...
Done in 36.586s
removing stale ckpt [epoch 17] [took 0.00s]
epoch : 18
loss : 1.203884889125824
quant_reg : 13.75698607635498
quant_err : 13.75698607635498
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.8