-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_xlnet-base.txt
2597 lines (2597 loc) · 194 KB
/
HCQ_MSRVTT_1kA_xlnet-base.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 436.2964913845062 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 44.73980689048767 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 24.704132080078125 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch0.pth ...
Done in 1.539s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch0.pth ...
Done in 3.059s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 505.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 498.932
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.2884499140614817
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 502.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 498.875
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3556893304490063
mnt_best : 0.2884499140614817
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.84054 (QuantReg: 22.29313) QuantErr: 22.29313 batch_time=24.29934
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.70549 (QuantReg: 22.34831) QuantErr: 22.34831 batch_time=3.44769
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 9.46065 (QuantReg: 22.68459) QuantErr: 22.68459 batch_time=0.57552
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.30529 (QuantReg: 22.68397) QuantErr: 22.68397 batch_time=0.83857
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.31539 (QuantReg: 22.68751) QuantErr: 22.68751 batch_time=0.57026
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.45569 (QuantReg: 22.75226) QuantErr: 22.75226 batch_time=0.80848
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.32975 (QuantReg: 22.74510) QuantErr: 22.74510 batch_time=0.60332
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.18967 (QuantReg: 22.78173) QuantErr: 22.78173 batch_time=0.60385
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.67447 (QuantReg: 22.67250) QuantErr: 22.67250 batch_time=0.56278
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.52493 (QuantReg: 22.66192) QuantErr: 22.66192 batch_time=1.34023
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.08152 (QuantReg: 22.72320) QuantErr: 22.72320 batch_time=0.57515
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.24552 (QuantReg: 22.65338) QuantErr: 22.65338 batch_time=0.57449
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.93284 (QuantReg: 22.72827) QuantErr: 22.72827 batch_time=0.61503
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.20846 (QuantReg: 22.67856) QuantErr: 22.67856 batch_time=0.55655
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.05965 (QuantReg: 22.75234) QuantErr: 22.75234 batch_time=0.58941
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.89297 (QuantReg: 22.69906) QuantErr: 22.69906 batch_time=0.56487
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.96006 (QuantReg: 22.71171) QuantErr: 22.71171 batch_time=0.58964
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.59777 (QuantReg: 22.72224) QuantErr: 22.72224 batch_time=0.57588
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.90249 (QuantReg: 22.69827) QuantErr: 22.69827 batch_time=0.57139
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.31339 (QuantReg: 22.69985) QuantErr: 22.69985 batch_time=0.58755
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.46399 (QuantReg: 22.73879) QuantErr: 22.73879 batch_time=0.86747
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.53642 (QuantReg: 22.70972) QuantErr: 22.70972 batch_time=0.57005
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.85661 (QuantReg: 22.69503) QuantErr: 22.69503 batch_time=0.56870
Train Epoch: 1 codebook_update_time=1.99167
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch1.pth ...
Done in 4.217s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch1.pth ...
Done in 8.368s
epoch : 1
loss : 5.90193466758728
quant_reg : 22.67569789123535
quant_err : 22.67569789123535
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 10.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 31.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 78.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 43.313
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.88531738703393
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 34.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 47.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 78.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 12.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 42.479
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.05724271578273
mnt_best : 24.88531738703393
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.37589 (QuantReg: 11.50677) QuantErr: 11.50677 batch_time=26.98476
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.96637 (QuantReg: 11.28447) QuantErr: 11.28447 batch_time=0.57492
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.02971 (QuantReg: 12.16400) QuantErr: 12.16400 batch_time=0.58334
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.98573 (QuantReg: 12.10721) QuantErr: 12.10721 batch_time=0.58772
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.06700 (QuantReg: 12.19834) QuantErr: 12.19834 batch_time=0.62069
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.04999 (QuantReg: 11.75603) QuantErr: 11.75603 batch_time=0.58117
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.32400 (QuantReg: 12.56432) QuantErr: 12.56432 batch_time=0.59058
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.75331 (QuantReg: 12.64065) QuantErr: 12.64065 batch_time=0.57105
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.06689 (QuantReg: 12.80527) QuantErr: 12.80527 batch_time=0.56899
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.74627 (QuantReg: 12.67505) QuantErr: 12.67505 batch_time=0.62510
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.70690 (QuantReg: 12.89439) QuantErr: 12.89439 batch_time=0.57098
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.12750 (QuantReg: 13.56123) QuantErr: 13.56123 batch_time=0.60788
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.45504 (QuantReg: 13.19329) QuantErr: 13.19329 batch_time=0.66120
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.47490 (QuantReg: 13.04741) QuantErr: 13.04741 batch_time=0.57791
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.60430 (QuantReg: 13.77944) QuantErr: 13.77944 batch_time=0.56972
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.38320 (QuantReg: 14.05640) QuantErr: 14.05640 batch_time=0.57158
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.59627 (QuantReg: 14.77867) QuantErr: 14.77867 batch_time=0.57673
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.81339 (QuantReg: 13.75703) QuantErr: 13.75703 batch_time=0.59395
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.32929 (QuantReg: 14.39455) QuantErr: 14.39455 batch_time=0.57107
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.08175 (QuantReg: 14.77148) QuantErr: 14.77148 batch_time=4.74426
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.61981 (QuantReg: 14.03590) QuantErr: 14.03590 batch_time=0.57343
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.41266 (QuantReg: 14.76414) QuantErr: 14.76414 batch_time=0.56734
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.97129 (QuantReg: 15.04668) QuantErr: 15.04668 batch_time=0.57204
Train Epoch: 2 codebook_update_time=1.63567
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch2.pth ...
Done in 4.070s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch2.pth ...
Done in 8.388s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 3.807250259399414
quant_reg : 13.250363544464111
quant_err : 13.250363544464111
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 39.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 51.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.342
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.302926842705066
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.55
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.97246981164826
mnt_best : 31.302926842705066
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 4.06451 (QuantReg: 11.40883) QuantErr: 11.40883 batch_time=29.78369
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.61699 (QuantReg: 11.31238) QuantErr: 11.31238 batch_time=0.59373
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.40254 (QuantReg: 11.37488) QuantErr: 11.37488 batch_time=0.67215
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.78288 (QuantReg: 11.77617) QuantErr: 11.77617 batch_time=0.64821
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.42063 (QuantReg: 12.12808) QuantErr: 12.12808 batch_time=0.56857
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.85512 (QuantReg: 11.61159) QuantErr: 11.61159 batch_time=0.57415
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.39296 (QuantReg: 11.80841) QuantErr: 11.80841 batch_time=1.72491
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.06152 (QuantReg: 11.97360) QuantErr: 11.97360 batch_time=0.63614
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.27318 (QuantReg: 12.55697) QuantErr: 12.55697 batch_time=0.58901
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.53747 (QuantReg: 12.12883) QuantErr: 12.12883 batch_time=0.58999
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.87562 (QuantReg: 12.77136) QuantErr: 12.77136 batch_time=0.56998
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.00095 (QuantReg: 12.50315) QuantErr: 12.50315 batch_time=0.59453
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.15885 (QuantReg: 12.16077) QuantErr: 12.16077 batch_time=0.59134
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.28026 (QuantReg: 12.55401) QuantErr: 12.55401 batch_time=0.75386
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 2.85919 (QuantReg: 13.17169) QuantErr: 13.17169 batch_time=0.58035
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.39488 (QuantReg: 12.58823) QuantErr: 12.58823 batch_time=0.59392
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.98163 (QuantReg: 13.17865) QuantErr: 13.17865 batch_time=0.57598
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.28495 (QuantReg: 12.91614) QuantErr: 12.91614 batch_time=0.59750
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.50617 (QuantReg: 12.99412) QuantErr: 12.99412 batch_time=0.60459
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.62685 (QuantReg: 12.86925) QuantErr: 12.86925 batch_time=0.93257
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.27586 (QuantReg: 13.15309) QuantErr: 13.15309 batch_time=0.57328
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.32855 (QuantReg: 13.13085) QuantErr: 13.13085 batch_time=0.56082
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.85898 (QuantReg: 13.13367) QuantErr: 13.13367 batch_time=0.58171
Train Epoch: 3 codebook_update_time=1.66513
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch3.pth ...
Done in 4.049s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch3.pth ...
Done in 8.199s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.26495304107666
quant_reg : 12.386537803649903
quant_err : 12.386537803649903
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.634
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.86942340122338
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.8305
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.27744155563768
mnt_best : 32.86942340122338
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.32005 (QuantReg: 11.81098) QuantErr: 11.81098 batch_time=29.72559
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.09972 (QuantReg: 12.15645) QuantErr: 12.15645 batch_time=0.57137
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.73058 (QuantReg: 11.86332) QuantErr: 11.86332 batch_time=0.55523
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.97440 (QuantReg: 12.18102) QuantErr: 12.18102 batch_time=0.57521
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.04128 (QuantReg: 12.40923) QuantErr: 12.40923 batch_time=0.58487
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.98098 (QuantReg: 11.61537) QuantErr: 11.61537 batch_time=0.57320
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.82425 (QuantReg: 12.01124) QuantErr: 12.01124 batch_time=0.63085
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.93330 (QuantReg: 12.33336) QuantErr: 12.33336 batch_time=0.58853
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.89688 (QuantReg: 12.07626) QuantErr: 12.07626 batch_time=0.58442
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.15210 (QuantReg: 12.28839) QuantErr: 12.28839 batch_time=0.56666
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.30838 (QuantReg: 12.04938) QuantErr: 12.04938 batch_time=0.58474
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.17885 (QuantReg: 12.47266) QuantErr: 12.47266 batch_time=0.56377
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.30652 (QuantReg: 12.22653) QuantErr: 12.22653 batch_time=0.57701
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.20148 (QuantReg: 12.70078) QuantErr: 12.70078 batch_time=0.78401
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.69487 (QuantReg: 12.80402) QuantErr: 12.80402 batch_time=0.59847
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.78739 (QuantReg: 12.81676) QuantErr: 12.81676 batch_time=0.57249
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.02102 (QuantReg: 12.78078) QuantErr: 12.78078 batch_time=0.56205
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.15850 (QuantReg: 12.89877) QuantErr: 12.89877 batch_time=0.54919
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.35127 (QuantReg: 13.06323) QuantErr: 13.06323 batch_time=0.58163
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.43033 (QuantReg: 12.54230) QuantErr: 12.54230 batch_time=0.58508
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.79877 (QuantReg: 12.85304) QuantErr: 12.85304 batch_time=0.57656
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.91148 (QuantReg: 12.50231) QuantErr: 12.50231 batch_time=0.58421
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.82110 (QuantReg: 12.69834) QuantErr: 12.69834 batch_time=0.62339
Train Epoch: 4 codebook_update_time=1.79759
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch4.pth ...
Done in 4.113s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch4.pth ...
Done in 8.318s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.841179857254028
quant_reg : 12.401679286956787
quant_err : 12.401679286956787
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.188
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.78090730554604
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.90878040575703
mnt_best : 34.78090730554604
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.81426 (QuantReg: 11.82159) QuantErr: 11.82159 batch_time=28.84169
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.57024 (QuantReg: 12.14781) QuantErr: 12.14781 batch_time=0.58852
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.87176 (QuantReg: 11.81578) QuantErr: 11.81578 batch_time=0.58058
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.53592 (QuantReg: 11.82706) QuantErr: 11.82706 batch_time=0.56640
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.82722 (QuantReg: 11.79838) QuantErr: 11.79838 batch_time=0.56986
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.68183 (QuantReg: 12.29662) QuantErr: 12.29662 batch_time=0.58011
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.87605 (QuantReg: 12.11109) QuantErr: 12.11109 batch_time=0.57332
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.63637 (QuantReg: 12.27989) QuantErr: 12.27989 batch_time=0.57772
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.06344 (QuantReg: 12.03049) QuantErr: 12.03049 batch_time=1.21919
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.41739 (QuantReg: 12.22333) QuantErr: 12.22333 batch_time=0.56385
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.55519 (QuantReg: 12.08986) QuantErr: 12.08986 batch_time=0.60527
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.16405 (QuantReg: 12.55936) QuantErr: 12.55936 batch_time=0.57888
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.54614 (QuantReg: 12.43358) QuantErr: 12.43358 batch_time=0.58106
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 3.03184 (QuantReg: 12.68514) QuantErr: 12.68514 batch_time=1.37826
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.57235 (QuantReg: 12.17510) QuantErr: 12.17510 batch_time=0.56827
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.43501 (QuantReg: 12.55849) QuantErr: 12.55849 batch_time=0.57343
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.46401 (QuantReg: 12.76454) QuantErr: 12.76454 batch_time=0.60903
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.35875 (QuantReg: 12.68642) QuantErr: 12.68642 batch_time=0.56390
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.44364 (QuantReg: 12.55652) QuantErr: 12.55652 batch_time=0.79061
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.68301 (QuantReg: 12.63165) QuantErr: 12.63165 batch_time=4.76853
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.77011 (QuantReg: 12.57499) QuantErr: 12.57499 batch_time=0.56907
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.08244 (QuantReg: 12.60575) QuantErr: 12.60575 batch_time=0.57184
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.44587 (QuantReg: 12.54423) QuantErr: 12.54423 batch_time=1.28988
Train Epoch: 5 codebook_update_time=1.61687
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch5.pth ...
Done in 4.127s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch5.pth ...
Done in 15.676s
removing stale ckpt [epoch 4] [took 0.02s]
epoch : 5
loss : 2.5538583035469054
quant_reg : 12.386424171447754
quant_err : 12.386424171447754
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.632
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.03079955549883
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.3495
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.96521640252832
mnt_best : 35.03079955549883
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.06718 (QuantReg: 12.05209) QuantErr: 12.05209 batch_time=29.48187
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.60240 (QuantReg: 11.79144) QuantErr: 11.79144 batch_time=0.56740
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.39761 (QuantReg: 12.04591) QuantErr: 12.04591 batch_time=0.61844
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.44757 (QuantReg: 12.53799) QuantErr: 12.53799 batch_time=1.14810
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.22292 (QuantReg: 12.19514) QuantErr: 12.19514 batch_time=0.56889
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.72469 (QuantReg: 12.28425) QuantErr: 12.28425 batch_time=0.62171
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.43155 (QuantReg: 12.32638) QuantErr: 12.32638 batch_time=0.59768
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.36219 (QuantReg: 12.29386) QuantErr: 12.29386 batch_time=0.56473
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.45589 (QuantReg: 12.50930) QuantErr: 12.50930 batch_time=1.02540
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.34382 (QuantReg: 12.58570) QuantErr: 12.58570 batch_time=0.58703
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.53843 (QuantReg: 12.45951) QuantErr: 12.45951 batch_time=0.57122
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.38925 (QuantReg: 12.37466) QuantErr: 12.37466 batch_time=0.59959
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.35088 (QuantReg: 12.60876) QuantErr: 12.60876 batch_time=0.57374
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.30098 (QuantReg: 12.80968) QuantErr: 12.80968 batch_time=1.04682
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.65479 (QuantReg: 12.68843) QuantErr: 12.68843 batch_time=0.60290
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.32803 (QuantReg: 12.31082) QuantErr: 12.31082 batch_time=0.57939
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.99422 (QuantReg: 12.69497) QuantErr: 12.69497 batch_time=0.58358
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.57307 (QuantReg: 12.35843) QuantErr: 12.35843 batch_time=0.57316
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.53196 (QuantReg: 12.48500) QuantErr: 12.48500 batch_time=0.57100
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.59714 (QuantReg: 12.87071) QuantErr: 12.87071 batch_time=0.76024
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 1.79055 (QuantReg: 12.65143) QuantErr: 12.65143 batch_time=0.62555
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.21350 (QuantReg: 12.68088) QuantErr: 12.68088 batch_time=0.56829
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.16487 (QuantReg: 13.16908) QuantErr: 13.16908 batch_time=0.56884
Train Epoch: 6 codebook_update_time=1.61843
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch6.pth ...
Done in 14.243s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch6.pth ...
Done in 18.321s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.36727965593338
quant_reg : 12.516217182159425
quant_err : 12.516217182159425
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.536
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.68744655005674
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.558
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.26237627615433
mnt_best : 36.68744655005674
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.40747 (QuantReg: 12.29937) QuantErr: 12.29937 batch_time=31.00992
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 1.75980 (QuantReg: 12.14509) QuantErr: 12.14509 batch_time=0.59198
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.19466 (QuantReg: 12.72346) QuantErr: 12.72346 batch_time=0.60871
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.39028 (QuantReg: 12.42155) QuantErr: 12.42155 batch_time=0.56529
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.83221 (QuantReg: 12.33153) QuantErr: 12.33153 batch_time=0.57114
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.15598 (QuantReg: 12.11944) QuantErr: 12.11944 batch_time=0.56592
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.01997 (QuantReg: 12.76737) QuantErr: 12.76737 batch_time=0.56084
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.30283 (QuantReg: 12.57315) QuantErr: 12.57315 batch_time=0.55246
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.96081 (QuantReg: 12.83190) QuantErr: 12.83190 batch_time=0.57546
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.12341 (QuantReg: 12.88510) QuantErr: 12.88510 batch_time=0.58302
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.52723 (QuantReg: 12.35839) QuantErr: 12.35839 batch_time=0.57153
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.23038 (QuantReg: 12.47987) QuantErr: 12.47987 batch_time=0.56766
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.11721 (QuantReg: 12.42330) QuantErr: 12.42330 batch_time=0.57711
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 1.84521 (QuantReg: 12.57975) QuantErr: 12.57975 batch_time=4.00287
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.27738 (QuantReg: 12.73722) QuantErr: 12.73722 batch_time=0.60012
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.38262 (QuantReg: 12.79133) QuantErr: 12.79133 batch_time=0.57084
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.05476 (QuantReg: 12.48773) QuantErr: 12.48773 batch_time=0.57434
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.29440 (QuantReg: 12.87815) QuantErr: 12.87815 batch_time=0.57447
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.32886 (QuantReg: 12.47609) QuantErr: 12.47609 batch_time=0.59030
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.23106 (QuantReg: 12.88881) QuantErr: 12.88881 batch_time=0.55744
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.20677 (QuantReg: 12.52693) QuantErr: 12.52693 batch_time=0.55784
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 1.71861 (QuantReg: 13.02498) QuantErr: 13.02498 batch_time=0.56595
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.23589 (QuantReg: 12.94949) QuantErr: 12.94949 batch_time=0.57607
Train Epoch: 7 codebook_update_time=1.67736
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch7.pth ...
Done in 3.932s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch7.pth ...
Done in 25.877s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.2196040849685668
quant_reg : 12.59848087310791
quant_err : 12.59848087310791
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.454
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.20825365545569
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.512
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.50271077781514
mnt_best : 38.20825365545569
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.86397 (QuantReg: 12.69801) QuantErr: 12.69801 batch_time=30.16717
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.03642 (QuantReg: 12.26781) QuantErr: 12.26781 batch_time=0.56414
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.31823 (QuantReg: 12.66993) QuantErr: 12.66993 batch_time=1.39129
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.74363 (QuantReg: 12.72566) QuantErr: 12.72566 batch_time=0.58982
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.48204 (QuantReg: 12.75567) QuantErr: 12.75567 batch_time=0.58201
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.22091 (QuantReg: 12.54618) QuantErr: 12.54618 batch_time=0.56688
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.89564 (QuantReg: 12.55092) QuantErr: 12.55092 batch_time=0.58044
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.06263 (QuantReg: 12.61471) QuantErr: 12.61471 batch_time=0.57745
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.65186 (QuantReg: 12.39071) QuantErr: 12.39071 batch_time=0.63678
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.14240 (QuantReg: 12.39266) QuantErr: 12.39266 batch_time=0.58946
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.12397 (QuantReg: 12.65746) QuantErr: 12.65746 batch_time=0.59057
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.08396 (QuantReg: 12.70522) QuantErr: 12.70522 batch_time=0.69921
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.01957 (QuantReg: 12.98712) QuantErr: 12.98712 batch_time=0.59200
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.88248 (QuantReg: 12.95564) QuantErr: 12.95564 batch_time=0.57177
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.99035 (QuantReg: 12.54844) QuantErr: 12.54844 batch_time=0.59006
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.12869 (QuantReg: 12.77981) QuantErr: 12.77981 batch_time=0.63318
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.37645 (QuantReg: 13.04674) QuantErr: 13.04674 batch_time=0.59596
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.25227 (QuantReg: 12.65421) QuantErr: 12.65421 batch_time=0.58175
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.16460 (QuantReg: 13.00652) QuantErr: 13.00652 batch_time=0.59622
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.73891 (QuantReg: 12.88972) QuantErr: 12.88972 batch_time=0.58534
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.04587 (QuantReg: 12.53616) QuantErr: 12.53616 batch_time=0.58682
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.16949 (QuantReg: 13.07267) QuantErr: 13.07267 batch_time=0.67307
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.98094 (QuantReg: 12.86692) QuantErr: 12.86692 batch_time=0.57881
Train Epoch: 8 codebook_update_time=1.99203
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch8.pth ...
Done in 4.065s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch8.pth ...
Done in 8.298s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.0858988447189333
quant_reg : 12.677543521881104
quant_err : 12.677543521881104
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.285
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.545667964744965
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.8065
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.82183247764618
mnt_best : 38.545667964744965
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.90331 (QuantReg: 12.62944) QuantErr: 12.62944 batch_time=35.31733
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.55412 (QuantReg: 12.64129) QuantErr: 12.64129 batch_time=0.57694
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.17761 (QuantReg: 12.97397) QuantErr: 12.97397 batch_time=0.56594
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.75724 (QuantReg: 12.69793) QuantErr: 12.69793 batch_time=0.56627
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.95249 (QuantReg: 12.78695) QuantErr: 12.78695 batch_time=0.61521
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.79159 (QuantReg: 13.06465) QuantErr: 13.06465 batch_time=0.60504
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.16848 (QuantReg: 12.38758) QuantErr: 12.38758 batch_time=0.61297
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.79632 (QuantReg: 12.70611) QuantErr: 12.70611 batch_time=0.57723
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.32302 (QuantReg: 12.80146) QuantErr: 12.80146 batch_time=0.57295
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.85431 (QuantReg: 12.88535) QuantErr: 12.88535 batch_time=0.57791
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.61770 (QuantReg: 12.93382) QuantErr: 12.93382 batch_time=0.57449
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.90395 (QuantReg: 12.63796) QuantErr: 12.63796 batch_time=0.58893
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.67361 (QuantReg: 12.76215) QuantErr: 12.76215 batch_time=0.63184
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.10571 (QuantReg: 13.00035) QuantErr: 13.00035 batch_time=0.57552
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.03145 (QuantReg: 12.85370) QuantErr: 12.85370 batch_time=0.57877
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.65821 (QuantReg: 12.88234) QuantErr: 12.88234 batch_time=0.56746
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.16395 (QuantReg: 12.93811) QuantErr: 12.93811 batch_time=0.56552
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.61912 (QuantReg: 12.95903) QuantErr: 12.95903 batch_time=0.57408
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.84594 (QuantReg: 12.99314) QuantErr: 12.99314 batch_time=0.57188
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.88658 (QuantReg: 12.93682) QuantErr: 12.93682 batch_time=0.57065
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.87549 (QuantReg: 13.14265) QuantErr: 13.14265 batch_time=0.59775
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.98781 (QuantReg: 12.61800) QuantErr: 12.61800 batch_time=0.56229
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.82578 (QuantReg: 12.64248) QuantErr: 12.64248 batch_time=0.57726
Train Epoch: 9 codebook_update_time=1.71496
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch9.pth ...
Done in 4.196s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch9.pth ...
Done in 8.492s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.9557132234573364
quant_reg : 12.788700916290283
quant_err : 12.788700916290283
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.028
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.68270542845022
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.7915
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.75548778765763
mnt_best : 38.68270542845022
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.94233 (QuantReg: 12.48250) QuantErr: 12.48250 batch_time=38.35687
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.93365 (QuantReg: 12.45852) QuantErr: 12.45852 batch_time=0.56415
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.20859 (QuantReg: 12.69436) QuantErr: 12.69436 batch_time=0.55848
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.38358 (QuantReg: 12.73885) QuantErr: 12.73885 batch_time=0.57273
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.07704 (QuantReg: 12.69167) QuantErr: 12.69167 batch_time=0.56483
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.00889 (QuantReg: 12.86250) QuantErr: 12.86250 batch_time=0.60404
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.53740 (QuantReg: 13.08534) QuantErr: 13.08534 batch_time=0.58107
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.90563 (QuantReg: 12.71487) QuantErr: 12.71487 batch_time=0.56978
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.01394 (QuantReg: 12.58046) QuantErr: 12.58046 batch_time=0.58014
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.02351 (QuantReg: 12.63752) QuantErr: 12.63752 batch_time=0.56562
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.51254 (QuantReg: 12.98125) QuantErr: 12.98125 batch_time=0.56591
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.43361 (QuantReg: 12.57894) QuantErr: 12.57894 batch_time=0.57969
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.93867 (QuantReg: 12.98140) QuantErr: 12.98140 batch_time=0.57060
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.93661 (QuantReg: 12.96955) QuantErr: 12.96955 batch_time=0.56877
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.82954 (QuantReg: 13.06826) QuantErr: 13.06826 batch_time=0.56211
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.77994 (QuantReg: 12.97289) QuantErr: 12.97289 batch_time=0.56949
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.03031 (QuantReg: 12.78348) QuantErr: 12.78348 batch_time=0.62510
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.97924 (QuantReg: 12.69005) QuantErr: 12.69005 batch_time=0.57792
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.90581 (QuantReg: 12.97026) QuantErr: 12.97026 batch_time=0.56795
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.10263 (QuantReg: 12.96808) QuantErr: 12.96808 batch_time=0.58240
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.76344 (QuantReg: 13.14065) QuantErr: 13.14065 batch_time=0.58212
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.91994 (QuantReg: 12.91623) QuantErr: 12.91623 batch_time=0.90111
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.00495 (QuantReg: 13.06770) QuantErr: 13.06770 batch_time=0.59427
Train Epoch: 10 codebook_update_time=1.72336
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch10.pth ...
Done in 4.676s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch10.pth ...
Done in 8.969s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.8637367043495179
quant_reg : 12.861246109008789
quant_err : 12.861246109008789
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.962
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.791632711612586
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.7895
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.26849154408536
mnt_best : 40.791632711612586
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.17437 (QuantReg: 12.69290) QuantErr: 12.69290 batch_time=31.31350
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.64016 (QuantReg: 12.63221) QuantErr: 12.63221 batch_time=0.56969
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.02947 (QuantReg: 12.50705) QuantErr: 12.50705 batch_time=0.57601
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.18074 (QuantReg: 12.85656) QuantErr: 12.85656 batch_time=0.58349
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.79030 (QuantReg: 13.08774) QuantErr: 13.08774 batch_time=0.56405
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.76344 (QuantReg: 12.89193) QuantErr: 12.89193 batch_time=0.56579
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.56451 (QuantReg: 12.87279) QuantErr: 12.87279 batch_time=0.57557
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.59947 (QuantReg: 12.88845) QuantErr: 12.88845 batch_time=0.58468
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.09793 (QuantReg: 12.77839) QuantErr: 12.77839 batch_time=0.57614
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.99823 (QuantReg: 12.58604) QuantErr: 12.58604 batch_time=0.59795
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.81197 (QuantReg: 12.97309) QuantErr: 12.97309 batch_time=0.57499
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.60528 (QuantReg: 13.03268) QuantErr: 13.03268 batch_time=0.58689
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.82493 (QuantReg: 12.92730) QuantErr: 12.92730 batch_time=0.57426
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.94464 (QuantReg: 12.87152) QuantErr: 12.87152 batch_time=4.21834
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.57612 (QuantReg: 12.78520) QuantErr: 12.78520 batch_time=0.61219
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.49648 (QuantReg: 12.97308) QuantErr: 12.97308 batch_time=0.58123
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.19158 (QuantReg: 12.76918) QuantErr: 12.76918 batch_time=0.58466
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.73439 (QuantReg: 13.10242) QuantErr: 13.10242 batch_time=0.60966
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.65945 (QuantReg: 12.81471) QuantErr: 12.81471 batch_time=0.58636
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.62967 (QuantReg: 13.20603) QuantErr: 13.20603 batch_time=0.57464
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.40261 (QuantReg: 13.12998) QuantErr: 13.12998 batch_time=0.58869
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.66692 (QuantReg: 12.79520) QuantErr: 12.79520 batch_time=0.59347
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.59895 (QuantReg: 13.25199) QuantErr: 13.25199 batch_time=0.57117
Train Epoch: 11 codebook_update_time=1.68063
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch11.pth ...
Done in 4.250s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.780605905532837
quant_reg : 12.857238842010497
quant_err : 12.857238842010497
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.521
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.55789210977297
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.873
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.82917234353608
mnt_best : 40.791632711612586
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.05235 (QuantReg: 12.78682) QuantErr: 12.78682 batch_time=32.10757
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.68432 (QuantReg: 13.00819) QuantErr: 13.00819 batch_time=0.59680
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.68892 (QuantReg: 12.77473) QuantErr: 12.77473 batch_time=0.58467
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.70731 (QuantReg: 12.84222) QuantErr: 12.84222 batch_time=0.57215
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.99869 (QuantReg: 12.47147) QuantErr: 12.47147 batch_time=0.58211
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.68444 (QuantReg: 12.81025) QuantErr: 12.81025 batch_time=0.60761
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.88167 (QuantReg: 12.78011) QuantErr: 12.78011 batch_time=0.58222
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.17660 (QuantReg: 12.75966) QuantErr: 12.75966 batch_time=0.64872
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.77653 (QuantReg: 12.65855) QuantErr: 12.65855 batch_time=0.57717
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.90268 (QuantReg: 13.10902) QuantErr: 13.10902 batch_time=0.59584
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.03222 (QuantReg: 13.01243) QuantErr: 13.01243 batch_time=0.57759
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.74615 (QuantReg: 13.00452) QuantErr: 13.00452 batch_time=0.60442
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.74692 (QuantReg: 12.94864) QuantErr: 12.94864 batch_time=0.59934
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.53880 (QuantReg: 12.91275) QuantErr: 12.91275 batch_time=2.07298
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.69865 (QuantReg: 12.74485) QuantErr: 12.74485 batch_time=0.58037
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.61918 (QuantReg: 12.86431) QuantErr: 12.86431 batch_time=0.57797
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.64393 (QuantReg: 12.97138) QuantErr: 12.97138 batch_time=0.56881
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.77411 (QuantReg: 12.72336) QuantErr: 12.72336 batch_time=0.56594
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.53881 (QuantReg: 13.20688) QuantErr: 13.20688 batch_time=1.04212
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.65755 (QuantReg: 13.09354) QuantErr: 13.09354 batch_time=0.56085
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.72912 (QuantReg: 13.11173) QuantErr: 13.11173 batch_time=0.56513
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.49818 (QuantReg: 12.80083) QuantErr: 12.80083 batch_time=0.59594
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.59962 (QuantReg: 13.37977) QuantErr: 13.37977 batch_time=0.84224
Train Epoch: 12 codebook_update_time=1.67371
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch12.pth ...
Done in 6.155s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.7073502130508422
quant_reg : 12.913925254821777
quant_err : 12.913925254821777
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.143
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.66524812155974
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.856
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.84515430592475
mnt_best : 40.791632711612586
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.95550 (QuantReg: 12.97274) QuantErr: 12.97274 batch_time=31.64040
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.74716 (QuantReg: 13.25841) QuantErr: 13.25841 batch_time=0.58184
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.53616 (QuantReg: 12.81459) QuantErr: 12.81459 batch_time=0.59321
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.52243 (QuantReg: 12.67933) QuantErr: 12.67933 batch_time=0.61679
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.53917 (QuantReg: 12.86610) QuantErr: 12.86610 batch_time=1.27728
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.41324 (QuantReg: 12.79351) QuantErr: 12.79351 batch_time=0.57188
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.82453 (QuantReg: 13.00223) QuantErr: 13.00223 batch_time=0.60142
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.92974 (QuantReg: 12.88510) QuantErr: 12.88510 batch_time=0.57010
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.82071 (QuantReg: 13.20412) QuantErr: 13.20412 batch_time=0.56991
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.63968 (QuantReg: 13.06020) QuantErr: 13.06020 batch_time=0.60076
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.14745 (QuantReg: 12.93518) QuantErr: 12.93518 batch_time=0.57767
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.46397 (QuantReg: 12.87394) QuantErr: 12.87394 batch_time=0.57160
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.50733 (QuantReg: 13.02563) QuantErr: 13.02563 batch_time=0.58270
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.78755 (QuantReg: 12.91764) QuantErr: 12.91764 batch_time=2.90574
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.84050 (QuantReg: 13.08976) QuantErr: 13.08976 batch_time=0.57624
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.36787 (QuantReg: 13.06731) QuantErr: 13.06731 batch_time=0.59994
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.65020 (QuantReg: 13.10515) QuantErr: 13.10515 batch_time=1.67628
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.50413 (QuantReg: 13.49333) QuantErr: 13.49333 batch_time=0.59469
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.66695 (QuantReg: 12.86433) QuantErr: 12.86433 batch_time=0.56982
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.51337 (QuantReg: 13.30475) QuantErr: 13.30475 batch_time=0.60745
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.40566 (QuantReg: 12.87484) QuantErr: 12.87484 batch_time=0.74094
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.85927 (QuantReg: 13.02400) QuantErr: 13.02400 batch_time=0.97061
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.50660 (QuantReg: 12.51802) QuantErr: 12.51802 batch_time=0.62981
Train Epoch: 13 codebook_update_time=1.74807
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch13.pth ...
Done in 4.194s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch13.pth ...
Done in 8.651s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.6323177161216735
quant_reg : 12.967290351867677
quant_err : 12.967290351867677
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.087
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.14132663488636
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.132
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.81527390342091
mnt_best : 42.14132663488636
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.11167 (QuantReg: 12.97313) QuantErr: 12.97313 batch_time=28.64071
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.45350 (QuantReg: 12.69119) QuantErr: 12.69119 batch_time=0.58867
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.50546 (QuantReg: 13.01478) QuantErr: 13.01478 batch_time=0.60352
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.75754 (QuantReg: 12.85407) QuantErr: 12.85407 batch_time=0.66376
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.71170 (QuantReg: 13.10893) QuantErr: 13.10893 batch_time=0.58258
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.56905 (QuantReg: 13.03118) QuantErr: 13.03118 batch_time=0.59569
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.77076 (QuantReg: 12.96795) QuantErr: 12.96795 batch_time=1.57794
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.60024 (QuantReg: 12.74356) QuantErr: 12.74356 batch_time=0.59522
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.76517 (QuantReg: 13.01070) QuantErr: 13.01070 batch_time=0.56794
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.34313 (QuantReg: 12.94465) QuantErr: 12.94465 batch_time=0.56629
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.36155 (QuantReg: 12.88978) QuantErr: 12.88978 batch_time=0.57585
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.20823 (QuantReg: 12.94159) QuantErr: 12.94159 batch_time=0.64398
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.73299 (QuantReg: 13.17188) QuantErr: 13.17188 batch_time=0.56859
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.65728 (QuantReg: 12.87136) QuantErr: 12.87136 batch_time=0.58386
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.40226 (QuantReg: 13.44109) QuantErr: 13.44109 batch_time=0.59386
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.40239 (QuantReg: 13.12902) QuantErr: 13.12902 batch_time=0.64239
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.82035 (QuantReg: 12.94117) QuantErr: 12.94117 batch_time=0.60006
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.31840 (QuantReg: 13.06343) QuantErr: 13.06343 batch_time=0.57653
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.40313 (QuantReg: 13.41002) QuantErr: 13.41002 batch_time=0.61088
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.44281 (QuantReg: 13.42015) QuantErr: 13.42015 batch_time=4.12635
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.72433 (QuantReg: 13.08991) QuantErr: 13.08991 batch_time=0.59472
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.54105 (QuantReg: 13.28291) QuantErr: 13.28291 batch_time=0.57354
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.62386 (QuantReg: 13.07798) QuantErr: 13.07798 batch_time=0.59535
Train Epoch: 14 codebook_update_time=1.92336
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch14.pth ...
Done in 15.589s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch14.pth ...
Done in 21.652s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.5592034363746643
quant_reg : 13.059032485961914
quant_err : 13.059032485961914
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.395
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.27721389014404
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.319
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.05283715788236
mnt_best : 42.27721389014404
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.62763 (QuantReg: 13.03843) QuantErr: 13.03843 batch_time=32.29840
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.43186 (QuantReg: 12.92871) QuantErr: 12.92871 batch_time=0.59166
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.22806 (QuantReg: 13.09031) QuantErr: 13.09031 batch_time=0.57348
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.62930 (QuantReg: 12.48484) QuantErr: 12.48484 batch_time=0.64941
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.60554 (QuantReg: 13.29290) QuantErr: 13.29290 batch_time=0.62279
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.38456 (QuantReg: 12.99359) QuantErr: 12.99359 batch_time=0.60657
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.77904 (QuantReg: 12.85214) QuantErr: 12.85214 batch_time=2.56144
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.39964 (QuantReg: 13.19729) QuantErr: 13.19729 batch_time=0.59521
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.45168 (QuantReg: 12.84430) QuantErr: 12.84430 batch_time=0.58038
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.81180 (QuantReg: 13.12374) QuantErr: 13.12374 batch_time=0.56144
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.37802 (QuantReg: 13.05637) QuantErr: 13.05637 batch_time=0.60882
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.45529 (QuantReg: 12.97094) QuantErr: 12.97094 batch_time=0.61206
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.90738 (QuantReg: 13.21046) QuantErr: 13.21046 batch_time=0.59096
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.28083 (QuantReg: 13.21243) QuantErr: 13.21243 batch_time=0.60245
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.55399 (QuantReg: 13.06424) QuantErr: 13.06424 batch_time=0.58251
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.52256 (QuantReg: 13.21403) QuantErr: 13.21403 batch_time=0.60129
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.17063 (QuantReg: 13.05941) QuantErr: 13.05941 batch_time=0.58080
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.50106 (QuantReg: 13.32743) QuantErr: 13.32743 batch_time=0.58906
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.25830 (QuantReg: 13.08810) QuantErr: 13.08810 batch_time=0.58014
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.46400 (QuantReg: 13.37356) QuantErr: 13.37356 batch_time=0.56536
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.53893 (QuantReg: 12.84109) QuantErr: 12.84109 batch_time=0.57149
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.43414 (QuantReg: 13.22621) QuantErr: 13.22621 batch_time=0.60883
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.39145 (QuantReg: 12.75670) QuantErr: 12.75670 batch_time=0.56824
Train Epoch: 15 codebook_update_time=1.69835
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch15.pth ...
Done in 4.401s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.5183355321884155
quant_reg : 13.091108039855957
quant_err : 13.091108039855957
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.274
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.19453850375259
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.57
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.66311456296782
mnt_best : 42.27721389014404
not_improved_count: 1
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.50529 (QuantReg: 12.90795) QuantErr: 12.90795 batch_time=31.35272
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.52515 (QuantReg: 13.03904) QuantErr: 13.03904 batch_time=0.56784
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.47835 (QuantReg: 13.20076) QuantErr: 13.20076 batch_time=0.56151
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.70305 (QuantReg: 12.86343) QuantErr: 12.86343 batch_time=0.59101
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.64153 (QuantReg: 12.97612) QuantErr: 12.97612 batch_time=0.56734
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.51842 (QuantReg: 12.65867) QuantErr: 12.65867 batch_time=0.56312
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.54594 (QuantReg: 13.02434) QuantErr: 13.02434 batch_time=2.37774
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.53369 (QuantReg: 13.22090) QuantErr: 13.22090 batch_time=0.59386
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.24751 (QuantReg: 13.14933) QuantErr: 13.14933 batch_time=0.58415
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.60722 (QuantReg: 13.19777) QuantErr: 13.19777 batch_time=0.59083
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.40576 (QuantReg: 13.18890) QuantErr: 13.18890 batch_time=0.59387
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.12232 (QuantReg: 12.80913) QuantErr: 12.80913 batch_time=0.59936
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.64598 (QuantReg: 13.09051) QuantErr: 13.09051 batch_time=0.57694
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.42409 (QuantReg: 13.40677) QuantErr: 13.40677 batch_time=1.79770
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.52823 (QuantReg: 12.90476) QuantErr: 12.90476 batch_time=0.60828
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.33876 (QuantReg: 13.10507) QuantErr: 13.10507 batch_time=0.60062
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.57759 (QuantReg: 13.25206) QuantErr: 13.25206 batch_time=0.58067
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.31911 (QuantReg: 12.91094) QuantErr: 12.91094 batch_time=0.57229
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.49583 (QuantReg: 13.01416) QuantErr: 13.01416 batch_time=0.56343
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.16803 (QuantReg: 13.28886) QuantErr: 13.28886 batch_time=0.60914
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.47509 (QuantReg: 13.13223) QuantErr: 13.13223 batch_time=1.00607
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.33808 (QuantReg: 12.98450) QuantErr: 12.98450 batch_time=0.66134
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.52064 (QuantReg: 12.97207) QuantErr: 12.97207 batch_time=0.58529
Train Epoch: 16 codebook_update_time=1.86885
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch16.pth ...
Done in 4.238s
removing stale ckpt [epoch 15] [took 0.74s]
epoch : 16
loss : 1.4700185554027558
quant_reg : 13.129893363952636
quant_err : 13.129893363952636
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.919
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.83310755086614
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.007
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.95586800817274
mnt_best : 42.27721389014404
not_improved_count: 2
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.64289 (QuantReg: 12.70227) QuantErr: 12.70227 batch_time=33.38394
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.99549 (QuantReg: 13.03594) QuantErr: 13.03594 batch_time=0.56493
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.35246 (QuantReg: 13.05905) QuantErr: 13.05905 batch_time=0.58209
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.38679 (QuantReg: 12.93351) QuantErr: 12.93351 batch_time=0.59621
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.39312 (QuantReg: 12.83695) QuantErr: 12.83695 batch_time=0.63806
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.53994 (QuantReg: 12.97500) QuantErr: 12.97500 batch_time=0.57001
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.58599 (QuantReg: 12.94350) QuantErr: 12.94350 batch_time=2.26299
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.45660 (QuantReg: 13.23982) QuantErr: 13.23982 batch_time=0.69238
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.02051 (QuantReg: 13.38305) QuantErr: 13.38305 batch_time=0.58237
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.36869 (QuantReg: 13.17441) QuantErr: 13.17441 batch_time=0.58257
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.35516 (QuantReg: 13.25196) QuantErr: 13.25196 batch_time=0.57411
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.12791 (QuantReg: 13.27070) QuantErr: 13.27070 batch_time=0.56584
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.21696 (QuantReg: 13.16163) QuantErr: 13.16163 batch_time=0.78988
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.28217 (QuantReg: 13.22477) QuantErr: 13.22477 batch_time=0.57523
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.23049 (QuantReg: 13.30663) QuantErr: 13.30663 batch_time=0.60236
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.06197 (QuantReg: 13.35555) QuantErr: 13.35555 batch_time=0.56572
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.29765 (QuantReg: 13.40685) QuantErr: 13.40685 batch_time=0.58589
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.40504 (QuantReg: 13.43830) QuantErr: 13.43830 batch_time=0.59649
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.54235 (QuantReg: 12.82153) QuantErr: 12.82153 batch_time=0.58419
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.56793 (QuantReg: 13.19310) QuantErr: 13.19310 batch_time=0.57753
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.15542 (QuantReg: 13.17585) QuantErr: 13.17585 batch_time=0.56814
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.18479 (QuantReg: 13.34953) QuantErr: 13.34953 batch_time=0.63336
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.67354 (QuantReg: 13.02205) QuantErr: 13.02205 batch_time=1.05929
Train Epoch: 17 codebook_update_time=1.65090
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch17.pth ...
Done in 4.523s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch17.pth ...
Done in 8.839s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 1.4011719882488252
quant_reg : 13.146054271697999
quant_err : 13.146054271697999
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.562
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.77088158292808
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 21.712
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.90540127448707
mnt_best : 42.77088158292808
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.39806 (QuantReg: 13.14544) QuantErr: 13.14544 batch_time=30.68097
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.75037 (QuantReg: 12.92440) QuantErr: 12.92440 batch_time=0.62921
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.28660 (QuantReg: 13.13307) QuantErr: 13.13307 batch_time=0.56812
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.45544 (QuantReg: 12.85415) QuantErr: 12.85415 batch_time=0.59890
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.40818 (QuantReg: 13.16295) QuantErr: 13.16295 batch_time=0.58521
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.30922 (QuantReg: 12.93665) QuantErr: 12.93665 batch_time=0.61448
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.42541 (QuantReg: 13.13976) QuantErr: 13.13976 batch_time=0.61910
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.26388 (QuantReg: 13.30151) QuantErr: 13.30151 batch_time=0.57720
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.83370 (QuantReg: 13.26805) QuantErr: 13.26805 batch_time=0.60270
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.28734 (QuantReg: 13.47109) QuantErr: 13.47109 batch_time=0.62660
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.51852 (QuantReg: 13.41301) QuantErr: 13.41301 batch_time=0.61718
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.60018 (QuantReg: 13.06580) QuantErr: 13.06580 batch_time=1.05027
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.43573 (QuantReg: 12.84233) QuantErr: 12.84233 batch_time=0.59533
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.70590 (QuantReg: 12.89532) QuantErr: 12.89532 batch_time=0.59018
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 0.99580 (QuantReg: 13.59170) QuantErr: 13.59170 batch_time=0.59273
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.46686 (QuantReg: 13.07787) QuantErr: 13.07787 batch_time=0.57291
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.25793 (QuantReg: 13.34836) QuantErr: 13.34836 batch_time=0.57443
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.57661 (QuantReg: 12.88616) QuantErr: 12.88616 batch_time=0.64718
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.15755 (QuantReg: 13.19320) QuantErr: 13.19320 batch_time=3.12798
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.72839 (QuantReg: 13.29006) QuantErr: 13.29006 batch_time=1.47599
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.42752 (QuantReg: 13.10995) QuantErr: 13.10995 batch_time=0.57479
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.08481 (QuantReg: 13.53167) QuantErr: 13.53167 batch_time=0.59797
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.38691 (QuantReg: 13.28858) QuantErr: 13.28858 batch_time=0.59176
Train Epoch: 18 codebook_update_time=1.81047
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch18.pth ...
Done in 15.065s
removing stale ckpt [epoch 17] [took 0.00s]
epoch : 18
loss : 1.3946205644607543
quant_reg : 13.165578521728516
quant_err : 13.165578521728516
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.944
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.71633216626605
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.1605
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.13266844763601
mnt_best : 42.77088158292808
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.06063 (QuantReg: 13.49313) QuantErr: 13.49313 batch_time=40.99250
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.07692 (QuantReg: 13.37159) QuantErr: 13.37159 batch_time=0.59366
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.28007 (QuantReg: 13.35286) QuantErr: 13.35286 batch_time=0.57192
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.54703 (QuantReg: 13.03281) QuantErr: 13.03281 batch_time=0.56432
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.40734 (QuantReg: 13.14770) QuantErr: 13.14770 batch_time=0.61765
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.70982 (QuantReg: 13.35975) QuantErr: 13.35975 batch_time=0.60570
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.42498 (QuantReg: 13.32604) QuantErr: 13.32604 batch_time=0.56563
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.36396 (QuantReg: 13.60655) QuantErr: 13.60655 batch_time=0.57746
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 0.91755 (QuantReg: 13.56693) QuantErr: 13.56693 batch_time=0.59399
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.41837 (QuantReg: 13.06335) QuantErr: 13.06335 batch_time=0.61453
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.31618 (QuantReg: 13.21359) QuantErr: 13.21359 batch_time=0.57891
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.26222 (QuantReg: 13.18805) QuantErr: 13.18805 batch_time=0.56767
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.66403 (QuantReg: 13.29283) QuantErr: 13.29283 batch_time=0.55972
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.97942 (QuantReg: 13.39779) QuantErr: 13.39779 batch_time=1.29264
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.35152 (QuantReg: 13.42324) QuantErr: 13.42324 batch_time=0.62132
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.52132 (QuantReg: 12.99799) QuantErr: 12.99799 batch_time=0.57448
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.45791 (QuantReg: 13.89033) QuantErr: 13.89033 batch_time=0.56740
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.14609 (QuantReg: 13.52912) QuantErr: 13.52912 batch_time=0.64886
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.15161 (QuantReg: 13.47418) QuantErr: 13.47418 batch_time=0.59678
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.53409 (QuantReg: 13.28497) QuantErr: 13.28497 batch_time=1.99427
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.34159 (QuantReg: 13.27214) QuantErr: 13.27214 batch_time=0.66402
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.18335 (QuantReg: 13.53432) QuantErr: 13.53432 batch_time=0.66545
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.25854 (QuantReg: 13.28499) QuantErr: 13.28499 batch_time=0.62844
Train Epoch: 19 codebook_update_time=1.75260
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch19.pth ...
Done in 14.919s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_xlnet-base/checkpoint-epoch19.pth ...
Done in 19.018s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 1.342387036561966
quant_reg : 13.251977294921875
quant_err : 13.251977294921875
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8