-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtf-loss-vals.txt
4643 lines (4617 loc) · 462 KB
/
tf-loss-vals.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2018-07-06 06:49:43,003 INFO - Preparing to run TaskInstance <TaskInstance: mrozh/projects/dan-tf/42 (id: q86sk5NfSWC5b62wCnXZ6L)
2018-07-06 06:49:43,026 INFO - Starting attempt 1
2018-07-06 06:49:43,041 INFO - Downloading and setting up data sources
2018-07-06 06:49:43,049 INFO - Downloading and mounting dataset. ETA: 12 seconds
2018-07-06 06:49:49,473 INFO - Pulling Docker image: floydhub/tensorflow:1.7.0-gpu.cuda9cudnn7-py2_aws.30
2018-07-06 06:49:50,743 INFO - Starting container...
2018-07-06 06:49:50,950 INFO -
################################################################################
2018-07-06 06:49:50,950 INFO - Run Output:
2018-07-06 06:49:51,052 INFO - Starting services.
2018-07-06 06:50:09,182 INFO - INFO:tensorflow:Using config: {'_save_checkpoints_secs': 1000000000.0, '_session_config': allow_soft_placement: true
2018-07-06 06:50:09,183 INFO - , '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9dd015e0d0>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/output', '_save_summary_steps': 100}
2018-07-06 06:50:09,183 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dd01c7f50>]
2018-07-06 06:50:09,183 INFO - Starting a training cycle.
2018-07-06 06:50:09,902 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:50:12,684 INFO - WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
2018-07-06 06:50:12,684 INFO - Instructions for updating:
2018-07-06 06:50:12,685 INFO - Use the retry module or similar alternatives.
2018-07-06 06:50:17,769 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:50:17,769 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 06:50:19,089 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:50:19,192 INFO - 2018-07-06 13:50:19.191113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-06 06:50:19,192 INFO - 2018-07-06 13:50:19.191494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
2018-07-06 06:50:19,192 INFO - name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
2018-07-06 06:50:19,192 INFO - pciBusID: 0000:00:1e.0
2018-07-06 06:50:19,193 INFO - totalMemory: 15.78GiB freeMemory: 14.87GiB
2018-07-06 06:50:19,193 INFO - 2018-07-06 13:50:19.191537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:50:19,498 INFO - 2018-07-06 13:50:19.497519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:50:19,498 INFO - 2018-07-06 13:50:19.497614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:50:19,499 INFO - 2018-07-06 13:50:19.497632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:50:19,499 INFO - 2018-07-06 13:50:19.497983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:50:21,170 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:50:21,199 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:50:34,057 INFO - 2018-07-06 13:50:34.056310: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x93ad160
2018-07-06 06:50:41,025 INFO - INFO:tensorflow:Saving checkpoints for 1 into /output/model.ckpt.
2018-07-06 06:50:41,921 INFO - INFO:tensorflow:loss = 0.12174993, step = 0
2018-07-06 06:50:56,774 INFO - INFO:tensorflow:global_step/sec: 6.73231
2018-07-06 06:50:56,775 INFO - INFO:tensorflow:loss = 0.06719494, step = 100 (14.854 sec)
2018-07-06 06:51:11,086 INFO - INFO:tensorflow:global_step/sec: 6.987
2018-07-06 06:51:11,087 INFO - INFO:tensorflow:loss = 0.04767549, step = 200 (14.312 sec)
2018-07-06 06:51:25,390 INFO - INFO:tensorflow:global_step/sec: 6.99129
2018-07-06 06:51:25,390 INFO - INFO:tensorflow:loss = 0.037817203, step = 300 (14.304 sec)
2018-07-06 06:51:39,699 INFO - INFO:tensorflow:global_step/sec: 6.98843
2018-07-06 06:51:39,700 INFO - INFO:tensorflow:loss = 0.0318426, step = 400 (14.309 sec)
2018-07-06 06:51:54,007 INFO - INFO:tensorflow:global_step/sec: 6.98937
2018-07-06 06:51:54,007 INFO - INFO:tensorflow:loss = 0.0318176, step = 500 (14.307 sec)
2018-07-06 06:52:08,319 INFO - INFO:tensorflow:global_step/sec: 6.98711
2018-07-06 06:52:08,319 INFO - INFO:tensorflow:loss = 0.029142972, step = 600 (14.312 sec)
2018-07-06 06:52:09,856 INFO - INFO:tensorflow:Saving checkpoints for 607 into /output/model.ckpt.
2018-07-06 06:52:10,668 INFO - Starting to evaluate.
2018-07-06 06:52:10,668 INFO - INFO:tensorflow:Loss for final step: 0.037562106.
2018-07-06 06:52:11,188 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:52:13,634 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:52:13,657 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-13:52:13
2018-07-06 06:52:14,479 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:52:14,479 INFO - 2018-07-06 13:52:14.478417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:52:14,479 INFO - 2018-07-06 13:52:14.478519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:52:14,479 INFO - 2018-07-06 13:52:14.478552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:52:14,480 INFO - 2018-07-06 13:52:14.478577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:52:14,480 INFO - 2018-07-06 13:52:14.478750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:52:14,480 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-607
2018-07-06 06:52:15,183 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:52:15,215 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:53:21,461 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-13:53:21
2018-07-06 06:53:21,462 INFO - INFO:tensorflow:Saving dict for global step 607: global_step = 607, loss = 0.0603469
2018-07-06 06:53:22,250 INFO - {'loss': 0.0603469, 'global_step': 607}
2018-07-06 06:53:22,251 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc5b9d390>]
2018-07-06 06:53:22,251 INFO - Starting a training cycle.
2018-07-06 06:53:22,809 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:53:25,498 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:53:25,498 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 06:53:26,204 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:53:26,204 INFO - 2018-07-06 13:53:26.203873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:53:26,205 INFO - 2018-07-06 13:53:26.203975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:53:26,205 INFO - 2018-07-06 13:53:26.204001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:53:26,205 INFO - 2018-07-06 13:53:26.204016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:53:26,205 INFO - 2018-07-06 13:53:26.204172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:53:26,205 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-607
2018-07-06 06:53:26,931 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:53:26,960 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:53:31,756 INFO - INFO:tensorflow:Saving checkpoints for 608 into /output/model.ckpt.
2018-07-06 06:53:32,492 INFO - INFO:tensorflow:loss = 0.02997848, step = 607
2018-07-06 06:53:47,362 INFO - INFO:tensorflow:global_step/sec: 6.72481
2018-07-06 06:53:47,363 INFO - INFO:tensorflow:loss = 0.040653348, step = 707 (14.871 sec)
2018-07-06 06:54:01,685 INFO - INFO:tensorflow:global_step/sec: 6.98181
2018-07-06 06:54:01,686 INFO - INFO:tensorflow:loss = 0.02785483, step = 807 (14.323 sec)
2018-07-06 06:54:16,009 INFO - INFO:tensorflow:global_step/sec: 6.9816
2018-07-06 06:54:16,009 INFO - INFO:tensorflow:loss = 0.033256553, step = 907 (14.323 sec)
2018-07-06 06:54:30,333 INFO - INFO:tensorflow:global_step/sec: 6.98123
2018-07-06 06:54:30,333 INFO - INFO:tensorflow:loss = 0.030862913, step = 1007 (14.324 sec)
2018-07-06 06:54:44,658 INFO - INFO:tensorflow:global_step/sec: 6.98049
2018-07-06 06:54:44,659 INFO - INFO:tensorflow:loss = 0.02795117, step = 1107 (14.326 sec)
2018-07-06 06:54:58,980 INFO - INFO:tensorflow:global_step/sec: 6.98256
2018-07-06 06:54:58,980 INFO - INFO:tensorflow:loss = 0.02799232, step = 1207 (14.321 sec)
2018-07-06 06:54:59,808 INFO - INFO:tensorflow:Saving checkpoints for 1214 into /output/model.ckpt.
2018-07-06 06:55:00,482 INFO - INFO:tensorflow:Loss for final step: 0.031708468.
2018-07-06 06:55:00,483 INFO - Starting to evaluate.
2018-07-06 06:55:00,990 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:55:03,671 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:55:03,693 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-13:55:03
2018-07-06 06:55:04,385 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:55:04,692 INFO - 2018-07-06 13:55:04.385222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:55:04,692 INFO - 2018-07-06 13:55:04.385324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:55:04,692 INFO - 2018-07-06 13:55:04.385348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:55:04,692 INFO - 2018-07-06 13:55:04.385363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:55:04,692 INFO - 2018-07-06 13:55:04.385520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:55:04,692 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1214
2018-07-06 06:55:05,116 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:55:05,150 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:56:10,862 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-13:56:10
2018-07-06 06:56:10,863 INFO - INFO:tensorflow:Saving dict for global step 1214: global_step = 1214, loss = 0.033754736
2018-07-06 06:56:10,863 INFO - {'loss': 0.033754736, 'global_step': 1214}
2018-07-06 06:56:10,863 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc668f550>]
2018-07-06 06:56:10,863 INFO - Starting a training cycle.
2018-07-06 06:56:11,478 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:56:14,111 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:56:14,112 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 06:56:14,677 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:56:14,677 INFO - 2018-07-06 13:56:14.676855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:56:14,678 INFO - 2018-07-06 13:56:14.676957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:56:14,678 INFO - 2018-07-06 13:56:14.677007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:56:14,678 INFO - 2018-07-06 13:56:14.677032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:56:14,678 INFO - 2018-07-06 13:56:14.677200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:56:14,678 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1214
2018-07-06 06:56:15,427 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:56:15,456 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:56:20,283 INFO - INFO:tensorflow:Saving checkpoints for 1215 into /output/model.ckpt.
2018-07-06 06:56:21,172 INFO - INFO:tensorflow:loss = 0.025358662, step = 1214
2018-07-06 06:56:36,040 INFO - INFO:tensorflow:global_step/sec: 6.7257
2018-07-06 06:56:36,040 INFO - INFO:tensorflow:loss = 0.034334697, step = 1314 (14.869 sec)
2018-07-06 06:56:50,340 INFO - INFO:tensorflow:global_step/sec: 6.99291
2018-07-06 06:56:50,340 INFO - INFO:tensorflow:loss = 0.026272025, step = 1414 (14.300 sec)
2018-07-06 06:57:04,646 INFO - INFO:tensorflow:global_step/sec: 6.99006
2018-07-06 06:57:04,647 INFO - INFO:tensorflow:loss = 0.030837474, step = 1514 (14.306 sec)
2018-07-06 06:57:18,946 INFO - INFO:tensorflow:global_step/sec: 6.99334
2018-07-06 06:57:18,946 INFO - INFO:tensorflow:loss = 0.025705399, step = 1614 (14.299 sec)
2018-07-06 06:57:33,248 INFO - INFO:tensorflow:global_step/sec: 6.99194
2018-07-06 06:57:33,249 INFO - INFO:tensorflow:loss = 0.02636382, step = 1714 (14.302 sec)
2018-07-06 06:57:47,553 INFO - INFO:tensorflow:global_step/sec: 6.99004
2018-07-06 06:57:47,554 INFO - INFO:tensorflow:loss = 0.02342496, step = 1814 (14.306 sec)
2018-07-06 06:57:48,379 INFO - INFO:tensorflow:Saving checkpoints for 1821 into /output/model.ckpt.
2018-07-06 06:57:49,114 INFO - INFO:tensorflow:Loss for final step: 0.026017826.
2018-07-06 06:57:49,114 INFO - Starting to evaluate.
2018-07-06 06:57:49,623 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:57:52,009 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:57:52,273 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-13:57:52
2018-07-06 06:57:52,852 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:57:52,852 INFO - 2018-07-06 13:57:52.851058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:57:52,853 INFO - 2018-07-06 13:57:52.851155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:57:52,853 INFO - 2018-07-06 13:57:52.851175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:57:52,853 INFO - 2018-07-06 13:57:52.851189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:57:52,853 INFO - 2018-07-06 13:57:52.851355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:57:52,853 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1821
2018-07-06 06:57:53,598 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:57:53,703 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:58:59,545 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-13:58:59
2018-07-06 06:58:59,545 INFO - INFO:tensorflow:Saving dict for global step 1821: global_step = 1821, loss = 0.031983826
2018-07-06 06:58:59,546 INFO - {'loss': 0.031983826, 'global_step': 1821}
2018-07-06 06:58:59,546 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc6688590>]
2018-07-06 06:58:59,546 INFO - Starting a training cycle.
2018-07-06 06:59:00,165 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 06:59:02,760 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 06:59:02,761 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 06:59:03,328 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 06:59:03,328 INFO - 2018-07-06 13:59:03.327273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 06:59:03,328 INFO - 2018-07-06 13:59:03.327370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 06:59:03,328 INFO - 2018-07-06 13:59:03.327406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 06:59:03,329 INFO - 2018-07-06 13:59:03.327448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 06:59:03,329 INFO - 2018-07-06 13:59:03.327598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 06:59:03,329 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1821
2018-07-06 06:59:04,177 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 06:59:04,209 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 06:59:09,051 INFO - INFO:tensorflow:Saving checkpoints for 1822 into /output/model.ckpt.
2018-07-06 06:59:09,850 INFO - INFO:tensorflow:loss = 0.026412271, step = 1821
2018-07-06 06:59:24,707 INFO - INFO:tensorflow:global_step/sec: 6.73063
2018-07-06 06:59:24,708 INFO - INFO:tensorflow:loss = 0.03991484, step = 1921 (14.858 sec)
2018-07-06 06:59:39,020 INFO - INFO:tensorflow:global_step/sec: 6.98645
2018-07-06 06:59:39,021 INFO - INFO:tensorflow:loss = 0.0229844, step = 2021 (14.314 sec)
2018-07-06 06:59:53,326 INFO - INFO:tensorflow:global_step/sec: 6.9903
2018-07-06 06:59:53,326 INFO - INFO:tensorflow:loss = 0.025140185, step = 2121 (14.305 sec)
2018-07-06 07:00:07,636 INFO - INFO:tensorflow:global_step/sec: 6.9882
2018-07-06 07:00:07,636 INFO - INFO:tensorflow:loss = 0.027590739, step = 2221 (14.310 sec)
2018-07-06 07:00:21,941 INFO - INFO:tensorflow:global_step/sec: 6.99053
2018-07-06 07:00:21,941 INFO - INFO:tensorflow:loss = 0.030393828, step = 2321 (14.305 sec)
2018-07-06 07:00:36,257 INFO - INFO:tensorflow:global_step/sec: 6.9853
2018-07-06 07:00:36,257 INFO - INFO:tensorflow:loss = 0.029187968, step = 2421 (14.316 sec)
2018-07-06 07:00:37,083 INFO - INFO:tensorflow:Saving checkpoints for 2428 into /output/model.ckpt.
2018-07-06 07:00:37,820 INFO - Starting to evaluate.
2018-07-06 07:00:37,821 INFO - INFO:tensorflow:Loss for final step: 0.032715242.
2018-07-06 07:00:38,325 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:00:40,984 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:00:41,007 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:00:41
2018-07-06 07:00:41,592 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:00:41,592 INFO - 2018-07-06 14:00:41.591167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:00:41,592 INFO - 2018-07-06 14:00:41.591267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:00:41,593 INFO - 2018-07-06 14:00:41.591299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:00:41,593 INFO - 2018-07-06 14:00:41.591329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:00:41,593 INFO - 2018-07-06 14:00:41.591499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:00:41,593 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-2428
2018-07-06 07:00:42,295 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:00:42,991 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:01:48,461 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:01:48
2018-07-06 07:01:48,461 INFO - INFO:tensorflow:Saving dict for global step 2428: global_step = 2428, loss = 0.025946599
2018-07-06 07:01:48,461 INFO - {'loss': 0.025946599, 'global_step': 2428}
2018-07-06 07:01:48,462 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc66bc110>]
2018-07-06 07:01:48,462 INFO - Starting a training cycle.
2018-07-06 07:01:49,062 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:01:51,616 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:01:51,617 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:01:52,289 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:01:52,291 INFO - 2018-07-06 14:01:52.288339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:01:52,291 INFO - 2018-07-06 14:01:52.288445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:01:52,291 INFO - 2018-07-06 14:01:52.288472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:01:52,291 INFO - 2018-07-06 14:01:52.288488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:01:52,291 INFO - 2018-07-06 14:01:52.288640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:01:52,292 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-2428
2018-07-06 07:01:53,030 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:01:53,064 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:01:57,791 INFO - INFO:tensorflow:Saving checkpoints for 2429 into /output/model.ckpt.
2018-07-06 07:01:58,588 INFO - INFO:tensorflow:loss = 0.023788594, step = 2428
2018-07-06 07:02:13,472 INFO - INFO:tensorflow:global_step/sec: 6.71835
2018-07-06 07:02:13,473 INFO - INFO:tensorflow:loss = 0.03108539, step = 2528 (14.885 sec)
2018-07-06 07:02:27,778 INFO - INFO:tensorflow:global_step/sec: 6.99036
2018-07-06 07:02:27,778 INFO - INFO:tensorflow:loss = 0.03012304, step = 2628 (14.305 sec)
2018-07-06 07:02:42,099 INFO - INFO:tensorflow:global_step/sec: 6.98279
2018-07-06 07:02:42,099 INFO - INFO:tensorflow:loss = 0.02345968, step = 2728 (14.321 sec)
2018-07-06 07:02:56,421 INFO - INFO:tensorflow:global_step/sec: 6.98225
2018-07-06 07:02:56,421 INFO - INFO:tensorflow:loss = 0.025073107, step = 2828 (14.322 sec)
2018-07-06 07:03:10,741 INFO - INFO:tensorflow:global_step/sec: 6.98285
2018-07-06 07:03:10,742 INFO - INFO:tensorflow:loss = 0.021606095, step = 2928 (14.321 sec)
2018-07-06 07:03:25,055 INFO - INFO:tensorflow:global_step/sec: 6.98646
2018-07-06 07:03:25,055 INFO - INFO:tensorflow:loss = 0.025419222, step = 3028 (14.314 sec)
2018-07-06 07:03:25,881 INFO - INFO:tensorflow:Saving checkpoints for 3035 into /output/model.ckpt.
2018-07-06 07:03:26,689 INFO - INFO:tensorflow:Loss for final step: 0.030403417.
2018-07-06 07:03:26,690 INFO - Starting to evaluate.
2018-07-06 07:03:27,193 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:03:29,619 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:03:29,641 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:03:29
2018-07-06 07:03:30,434 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:03:30,434 INFO - 2018-07-06 14:03:30.433014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:03:30,434 INFO - 2018-07-06 14:03:30.433113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:03:30,434 INFO - 2018-07-06 14:03:30.433139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:03:30,434 INFO - 2018-07-06 14:03:30.433154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:03:30,434 INFO - 2018-07-06 14:03:30.433308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:03:30,435 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3035
2018-07-06 07:03:31,162 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:03:31,203 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:04:37,104 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:04:37
2018-07-06 07:04:37,105 INFO - INFO:tensorflow:Saving dict for global step 3035: global_step = 3035, loss = 0.02231731
2018-07-06 07:04:37,105 INFO - {'loss': 0.02231731, 'global_step': 3035}
2018-07-06 07:04:37,105 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9db01d2d90>]
2018-07-06 07:04:37,105 INFO - Starting a training cycle.
2018-07-06 07:04:37,704 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:04:40,228 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:04:40,229 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:04:40,790 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:04:40,791 INFO - 2018-07-06 14:04:40.790074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:04:40,791 INFO - 2018-07-06 14:04:40.790178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:04:40,791 INFO - 2018-07-06 14:04:40.790214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:04:40,791 INFO - 2018-07-06 14:04:40.790230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:04:40,791 INFO - 2018-07-06 14:04:40.790439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:04:40,792 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3035
2018-07-06 07:04:41,510 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:04:41,543 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:04:46,180 INFO - INFO:tensorflow:Saving checkpoints for 3036 into /output/model.ckpt.
2018-07-06 07:04:47,061 INFO - INFO:tensorflow:loss = 0.029393721, step = 3035
2018-07-06 07:05:01,911 INFO - INFO:tensorflow:global_step/sec: 6.73352
2018-07-06 07:05:01,912 INFO - INFO:tensorflow:loss = 0.021206304, step = 3135 (14.851 sec)
2018-07-06 07:05:16,213 INFO - INFO:tensorflow:global_step/sec: 6.99231
2018-07-06 07:05:16,213 INFO - INFO:tensorflow:loss = 0.02336489, step = 3235 (14.301 sec)
2018-07-06 07:05:30,509 INFO - INFO:tensorflow:global_step/sec: 6.99447
2018-07-06 07:05:30,511 INFO - INFO:tensorflow:loss = 0.025131812, step = 3335 (14.298 sec)
2018-07-06 07:05:44,815 INFO - INFO:tensorflow:global_step/sec: 6.99032
2018-07-06 07:05:44,816 INFO - INFO:tensorflow:loss = 0.021225976, step = 3435 (14.305 sec)
2018-07-06 07:05:59,118 INFO - INFO:tensorflow:global_step/sec: 6.99147
2018-07-06 07:05:59,119 INFO - INFO:tensorflow:loss = 0.024251219, step = 3535 (14.303 sec)
2018-07-06 07:06:13,418 INFO - INFO:tensorflow:global_step/sec: 6.99309
2018-07-06 07:06:13,419 INFO - INFO:tensorflow:loss = 0.0385471, step = 3635 (14.300 sec)
2018-07-06 07:06:14,255 INFO - INFO:tensorflow:Saving checkpoints for 3642 into /output/model.ckpt.
2018-07-06 07:06:14,957 INFO - INFO:tensorflow:Loss for final step: 0.030116139.
2018-07-06 07:06:14,957 INFO - Starting to evaluate.
2018-07-06 07:06:15,462 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:06:18,066 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:06:18,090 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:06:18
2018-07-06 07:06:18,668 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:06:18,668 INFO - 2018-07-06 14:06:18.667408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:06:18,668 INFO - 2018-07-06 14:06:18.667504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:06:18,669 INFO - 2018-07-06 14:06:18.667529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:06:18,669 INFO - 2018-07-06 14:06:18.667544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:06:18,669 INFO - 2018-07-06 14:06:18.667727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:06:18,669 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3642
2018-07-06 07:06:19,354 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:06:19,389 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:07:25,268 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:07:25
2018-07-06 07:07:25,269 INFO - INFO:tensorflow:Saving dict for global step 3642: global_step = 3642, loss = 0.022091888
2018-07-06 07:07:25,269 INFO - {'loss': 0.022091888, 'global_step': 3642}
2018-07-06 07:07:25,269 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc6a01b90>]
2018-07-06 07:07:25,269 INFO - Starting a training cycle.
2018-07-06 07:07:25,859 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:07:28,352 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:07:28,352 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:07:29,016 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:07:29,016 INFO - 2018-07-06 14:07:29.015359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:07:29,016 INFO - 2018-07-06 14:07:29.015465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:07:29,016 INFO - 2018-07-06 14:07:29.015493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:07:29,017 INFO - 2018-07-06 14:07:29.015507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:07:29,017 INFO - 2018-07-06 14:07:29.015672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:07:29,017 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3642
2018-07-06 07:07:29,720 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:07:29,748 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:07:34,358 INFO - INFO:tensorflow:Saving checkpoints for 3643 into /output/model.ckpt.
2018-07-06 07:07:35,121 INFO - INFO:tensorflow:loss = 0.02358546, step = 3642
2018-07-06 07:07:49,935 INFO - INFO:tensorflow:global_step/sec: 6.75042
2018-07-06 07:07:49,935 INFO - INFO:tensorflow:loss = 0.024916291, step = 3742 (14.814 sec)
2018-07-06 07:08:04,238 INFO - INFO:tensorflow:global_step/sec: 6.99164
2018-07-06 07:08:04,238 INFO - INFO:tensorflow:loss = 0.021695025, step = 3842 (14.303 sec)
2018-07-06 07:08:18,540 INFO - INFO:tensorflow:global_step/sec: 6.99205
2018-07-06 07:08:18,540 INFO - INFO:tensorflow:loss = 0.030844614, step = 3942 (14.302 sec)
2018-07-06 07:08:32,831 INFO - INFO:tensorflow:global_step/sec: 6.99694
2018-07-06 07:08:32,833 INFO - INFO:tensorflow:loss = 0.024847183, step = 4042 (14.292 sec)
2018-07-06 07:08:47,120 INFO - INFO:tensorflow:global_step/sec: 6.99861
2018-07-06 07:08:47,121 INFO - INFO:tensorflow:loss = 0.025366014, step = 4142 (14.289 sec)
2018-07-06 07:09:01,419 INFO - INFO:tensorflow:global_step/sec: 6.99365
2018-07-06 07:09:01,420 INFO - INFO:tensorflow:loss = 0.020233983, step = 4242 (14.299 sec)
2018-07-06 07:09:02,244 INFO - INFO:tensorflow:Saving checkpoints for 4249 into /output/model.ckpt.
2018-07-06 07:09:02,958 INFO - Starting to evaluate.
2018-07-06 07:09:02,959 INFO - INFO:tensorflow:Loss for final step: 0.036381546.
2018-07-06 07:09:03,480 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:09:06,086 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:09:06,108 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:09:06
2018-07-06 07:09:06,782 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:09:06,783 INFO - 2018-07-06 14:09:06.781956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:09:06,783 INFO - 2018-07-06 14:09:06.782043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:09:06,783 INFO - 2018-07-06 14:09:06.782069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:09:06,783 INFO - 2018-07-06 14:09:06.782084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:09:06,783 INFO - 2018-07-06 14:09:06.782239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:09:06,784 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4249
2018-07-06 07:09:07,479 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:09:07,511 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:10:12,955 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:10:12
2018-07-06 07:10:12,956 INFO - INFO:tensorflow:Saving dict for global step 4249: global_step = 4249, loss = 0.018531874
2018-07-06 07:10:12,956 INFO - {'loss': 0.018531874, 'global_step': 4249}
2018-07-06 07:10:12,956 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9d901d1bd0>]
2018-07-06 07:10:12,956 INFO - Starting a training cycle.
2018-07-06 07:10:13,553 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:10:16,047 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:10:16,048 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:10:16,719 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:10:16,720 INFO - 2018-07-06 14:10:16.718826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:10:16,720 INFO - 2018-07-06 14:10:16.718928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:10:16,720 INFO - 2018-07-06 14:10:16.718955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:10:16,720 INFO - 2018-07-06 14:10:16.718969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:10:16,720 INFO - 2018-07-06 14:10:16.719124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:10:16,720 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4249
2018-07-06 07:10:17,430 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:10:17,463 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:10:22,268 INFO - INFO:tensorflow:Saving checkpoints for 4250 into /output/model.ckpt.
2018-07-06 07:10:23,035 INFO - INFO:tensorflow:loss = 0.021188369, step = 4249
2018-07-06 07:10:37,884 INFO - INFO:tensorflow:global_step/sec: 6.73427
2018-07-06 07:10:37,884 INFO - INFO:tensorflow:loss = 0.022029052, step = 4349 (14.850 sec)
2018-07-06 07:10:52,184 INFO - INFO:tensorflow:global_step/sec: 6.99268
2018-07-06 07:10:52,185 INFO - INFO:tensorflow:loss = 0.020008955, step = 4449 (14.301 sec)
2018-07-06 07:11:06,491 INFO - INFO:tensorflow:global_step/sec: 6.98965
2018-07-06 07:11:06,492 INFO - INFO:tensorflow:loss = 0.020856624, step = 4549 (14.307 sec)
2018-07-06 07:11:20,787 INFO - INFO:tensorflow:global_step/sec: 6.99512
2018-07-06 07:11:20,788 INFO - INFO:tensorflow:loss = 0.027245583, step = 4649 (14.296 sec)
2018-07-06 07:11:35,083 INFO - INFO:tensorflow:global_step/sec: 6.99497
2018-07-06 07:11:35,083 INFO - INFO:tensorflow:loss = 0.026195157, step = 4749 (14.296 sec)
2018-07-06 07:11:49,385 INFO - INFO:tensorflow:global_step/sec: 6.9921
2018-07-06 07:11:49,385 INFO - INFO:tensorflow:loss = 0.029291324, step = 4849 (14.302 sec)
2018-07-06 07:11:50,209 INFO - INFO:tensorflow:Saving checkpoints for 4856 into /output/model.ckpt.
2018-07-06 07:11:51,026 INFO - Starting to evaluate.
2018-07-06 07:11:51,027 INFO - INFO:tensorflow:Loss for final step: 0.020185357.
2018-07-06 07:11:51,544 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:11:53,901 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:11:53,923 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:11:53
2018-07-06 07:11:54,702 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:11:54,703 INFO - 2018-07-06 14:11:54.702068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:11:54,703 INFO - 2018-07-06 14:11:54.702190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:11:54,703 INFO - 2018-07-06 14:11:54.702224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:11:54,704 INFO - 2018-07-06 14:11:54.702251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:11:54,704 INFO - 2018-07-06 14:11:54.702420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:11:54,704 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4856
2018-07-06 07:11:55,413 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:11:55,441 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:13:01,261 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:13:01
2018-07-06 07:13:01,262 INFO - INFO:tensorflow:Saving dict for global step 4856: global_step = 4856, loss = 0.018247942
2018-07-06 07:13:01,262 INFO - {'loss': 0.018247942, 'global_step': 4856}
2018-07-06 07:13:01,262 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9992af80d0>]
2018-07-06 07:13:01,262 INFO - Starting a training cycle.
2018-07-06 07:13:01,843 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:13:04,364 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:13:04,364 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:13:04,911 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:13:04,911 INFO - 2018-07-06 14:13:04.910278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:13:04,911 INFO - 2018-07-06 14:13:04.910383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:13:04,911 INFO - 2018-07-06 14:13:04.910415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:13:04,912 INFO - 2018-07-06 14:13:04.910437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:13:04,912 INFO - 2018-07-06 14:13:04.910603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:13:04,912 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4856
2018-07-06 07:13:05,617 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:13:05,647 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:13:10,281 INFO - INFO:tensorflow:Saving checkpoints for 4857 into /output/model.ckpt.
2018-07-06 07:13:11,132 INFO - INFO:tensorflow:loss = 0.023437228, step = 4856
2018-07-06 07:13:25,971 INFO - INFO:tensorflow:global_step/sec: 6.73869
2018-07-06 07:13:25,971 INFO - INFO:tensorflow:loss = 0.018609205, step = 4956 (14.840 sec)
2018-07-06 07:13:40,272 INFO - INFO:tensorflow:global_step/sec: 6.99212
2018-07-06 07:13:40,273 INFO - INFO:tensorflow:loss = 0.02578427, step = 5056 (14.302 sec)
2018-07-06 07:13:54,574 INFO - INFO:tensorflow:global_step/sec: 6.99215
2018-07-06 07:13:54,575 INFO - INFO:tensorflow:loss = 0.025547411, step = 5156 (14.302 sec)
2018-07-06 07:14:08,882 INFO - INFO:tensorflow:global_step/sec: 6.98919
2018-07-06 07:14:08,883 INFO - INFO:tensorflow:loss = 0.024011675, step = 5256 (14.307 sec)
2018-07-06 07:14:23,181 INFO - INFO:tensorflow:global_step/sec: 6.99358
2018-07-06 07:14:23,181 INFO - INFO:tensorflow:loss = 0.021703092, step = 5356 (14.299 sec)
2018-07-06 07:14:37,478 INFO - INFO:tensorflow:global_step/sec: 6.99448
2018-07-06 07:14:37,479 INFO - INFO:tensorflow:loss = 0.028183136, step = 5456 (14.297 sec)
2018-07-06 07:14:38,305 INFO - INFO:tensorflow:Saving checkpoints for 5463 into /output/model.ckpt.
2018-07-06 07:14:38,996 INFO - INFO:tensorflow:Loss for final step: 0.027514178.
2018-07-06 07:14:38,996 INFO - Starting to evaluate.
2018-07-06 07:14:39,498 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:14:41,986 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:14:42,006 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:14:42
2018-07-06 07:14:42,559 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:14:42,560 INFO - 2018-07-06 14:14:42.558972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:14:42,560 INFO - 2018-07-06 14:14:42.559071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:14:42,560 INFO - 2018-07-06 14:14:42.559097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:14:42,560 INFO - 2018-07-06 14:14:42.559111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:14:42,560 INFO - 2018-07-06 14:14:42.559262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:14:42,560 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-5463
2018-07-06 07:14:43,232 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:14:43,260 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:15:49,377 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:15:49
2018-07-06 07:15:49,377 INFO - INFO:tensorflow:Saving dict for global step 5463: global_step = 5463, loss = 0.017039329
2018-07-06 07:15:49,377 INFO - {'loss': 0.017039329, 'global_step': 5463}
2018-07-06 07:15:49,378 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc64136d0>]
2018-07-06 07:15:49,378 INFO - Starting a training cycle.
2018-07-06 07:15:49,976 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:15:52,450 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:15:52,450 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:15:53,102 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:15:53,102 INFO - 2018-07-06 14:15:53.100837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:15:53,103 INFO - 2018-07-06 14:15:53.100936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:15:53,103 INFO - 2018-07-06 14:15:53.100978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:15:53,103 INFO - 2018-07-06 14:15:53.101005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:15:53,103 INFO - 2018-07-06 14:15:53.101167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:15:53,103 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-5463
2018-07-06 07:15:53,819 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:15:53,851 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:15:58,488 INFO - INFO:tensorflow:Saving checkpoints for 5464 into /output/model.ckpt.
2018-07-06 07:15:59,214 INFO - INFO:tensorflow:loss = 0.022031613, step = 5463
2018-07-06 07:16:14,055 INFO - INFO:tensorflow:global_step/sec: 6.73774
2018-07-06 07:16:14,056 INFO - INFO:tensorflow:loss = 0.021868931, step = 5563 (14.842 sec)
2018-07-06 07:16:28,356 INFO - INFO:tensorflow:global_step/sec: 6.9925
2018-07-06 07:16:28,357 INFO - INFO:tensorflow:loss = 0.024868682, step = 5663 (14.301 sec)
2018-07-06 07:16:42,660 INFO - INFO:tensorflow:global_step/sec: 6.99099
2018-07-06 07:16:42,661 INFO - INFO:tensorflow:loss = 0.02004668, step = 5763 (14.304 sec)
2018-07-06 07:16:56,957 INFO - INFO:tensorflow:global_step/sec: 6.99447
2018-07-06 07:16:56,958 INFO - INFO:tensorflow:loss = 0.020733025, step = 5863 (14.297 sec)
2018-07-06 07:17:11,266 INFO - INFO:tensorflow:global_step/sec: 6.9891
2018-07-06 07:17:11,266 INFO - INFO:tensorflow:loss = 0.03854402, step = 5963 (14.308 sec)
2018-07-06 07:17:25,563 INFO - INFO:tensorflow:global_step/sec: 6.99404
2018-07-06 07:17:25,564 INFO - INFO:tensorflow:loss = 0.021678742, step = 6063 (14.298 sec)
2018-07-06 07:17:26,388 INFO - INFO:tensorflow:Saving checkpoints for 6070 into /output/model.ckpt.
2018-07-06 07:17:27,102 INFO - INFO:tensorflow:Loss for final step: 0.025558887.
2018-07-06 07:17:27,103 INFO - Starting to evaluate.
2018-07-06 07:17:27,604 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:17:30,152 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:17:30,173 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:17:30
2018-07-06 07:17:30,851 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:17:30,851 INFO - 2018-07-06 14:17:30.850205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:17:30,851 INFO - 2018-07-06 14:17:30.850299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:17:30,851 INFO - 2018-07-06 14:17:30.850324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:17:30,851 INFO - 2018-07-06 14:17:30.850340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:17:30,851 INFO - 2018-07-06 14:17:30.850493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:17:30,852 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6070
2018-07-06 07:17:31,558 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:17:31,593 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:18:37,272 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:18:37
2018-07-06 07:18:37,272 INFO - INFO:tensorflow:Saving dict for global step 6070: global_step = 6070, loss = 0.01678104
2018-07-06 07:18:37,272 INFO - {'loss': 0.01678104, 'global_step': 6070}
2018-07-06 07:18:37,272 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc697d9d0>]
2018-07-06 07:18:37,273 INFO - Starting a training cycle.
2018-07-06 07:18:37,844 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:18:40,444 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:18:40,444 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:18:41,010 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:18:41,010 INFO - 2018-07-06 14:18:41.009028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:18:41,010 INFO - 2018-07-06 14:18:41.009133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:18:41,010 INFO - 2018-07-06 14:18:41.009170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:18:41,011 INFO - 2018-07-06 14:18:41.009196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:18:41,011 INFO - 2018-07-06 14:18:41.009374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:18:41,011 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6070
2018-07-06 07:18:41,716 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:18:41,751 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:18:46,444 INFO - INFO:tensorflow:Saving checkpoints for 6071 into /output/model.ckpt.
2018-07-06 07:18:47,229 INFO - INFO:tensorflow:loss = 0.019215744, step = 6070
2018-07-06 07:19:02,057 INFO - INFO:tensorflow:global_step/sec: 6.74352
2018-07-06 07:19:02,058 INFO - INFO:tensorflow:loss = 0.022667719, step = 6170 (14.829 sec)
2018-07-06 07:19:16,349 INFO - INFO:tensorflow:global_step/sec: 6.99699
2018-07-06 07:19:16,350 INFO - INFO:tensorflow:loss = 0.03463141, step = 6270 (14.292 sec)
2018-07-06 07:19:30,640 INFO - INFO:tensorflow:global_step/sec: 6.99762
2018-07-06 07:19:30,640 INFO - INFO:tensorflow:loss = 0.026040215, step = 6370 (14.291 sec)
2018-07-06 07:19:44,939 INFO - INFO:tensorflow:global_step/sec: 6.99347
2018-07-06 07:19:44,939 INFO - INFO:tensorflow:loss = 0.022409841, step = 6470 (14.299 sec)
2018-07-06 07:19:59,232 INFO - INFO:tensorflow:global_step/sec: 6.99616
2018-07-06 07:19:59,233 INFO - INFO:tensorflow:loss = 0.02349041, step = 6570 (14.294 sec)
2018-07-06 07:20:13,532 INFO - INFO:tensorflow:global_step/sec: 6.9931
2018-07-06 07:20:13,533 INFO - INFO:tensorflow:loss = 0.020171147, step = 6670 (14.300 sec)
2018-07-06 07:20:14,357 INFO - INFO:tensorflow:Saving checkpoints for 6677 into /output/model.ckpt.
2018-07-06 07:20:15,159 INFO - Starting to evaluate.
2018-07-06 07:20:15,159 INFO - INFO:tensorflow:Loss for final step: 0.027022036.
2018-07-06 07:20:15,682 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:20:18,008 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:20:18,241 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:20:18
2018-07-06 07:20:18,800 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:20:18,800 INFO - 2018-07-06 14:20:18.799244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:20:18,800 INFO - 2018-07-06 14:20:18.799339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:20:18,800 INFO - 2018-07-06 14:20:18.799363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:20:18,800 INFO - 2018-07-06 14:20:18.799377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:20:18,800 INFO - 2018-07-06 14:20:18.799525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:20:18,801 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6677
2018-07-06 07:20:19,494 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:20:19,526 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:21:24,911 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:21:24
2018-07-06 07:21:24,920 INFO - INFO:tensorflow:Saving dict for global step 6677: global_step = 6677, loss = 0.015808979
2018-07-06 07:21:24,926 INFO - {'loss': 0.015808979, 'global_step': 6677}
2018-07-06 07:21:24,926 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc6960990>]
2018-07-06 07:21:24,926 INFO - Starting a training cycle.
2018-07-06 07:21:25,513 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:21:28,093 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:21:28,093 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:21:28,652 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:21:28,653 INFO - 2018-07-06 14:21:28.651672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:21:28,653 INFO - 2018-07-06 14:21:28.651769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:21:28,653 INFO - 2018-07-06 14:21:28.651794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:21:28,654 INFO - 2018-07-06 14:21:28.651809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:21:28,654 INFO - 2018-07-06 14:21:28.651967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:21:28,654 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6677
2018-07-06 07:21:29,475 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:21:29,512 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:21:34,201 INFO - INFO:tensorflow:Saving checkpoints for 6678 into /output/model.ckpt.
2018-07-06 07:21:34,951 INFO - INFO:tensorflow:loss = 0.018832631, step = 6677
2018-07-06 07:21:49,765 INFO - INFO:tensorflow:global_step/sec: 6.74936
2018-07-06 07:21:50,466 INFO - INFO:tensorflow:loss = 0.022532498, step = 6777 (14.817 sec)
2018-07-06 07:22:04,048 INFO - INFO:tensorflow:global_step/sec: 7.0008
2018-07-06 07:22:04,049 INFO - INFO:tensorflow:loss = 0.028186835, step = 6877 (14.284 sec)
2018-07-06 07:22:18,339 INFO - INFO:tensorflow:global_step/sec: 6.9974
2018-07-06 07:22:18,340 INFO - INFO:tensorflow:loss = 0.023211123, step = 6977 (14.291 sec)
2018-07-06 07:22:32,625 INFO - INFO:tensorflow:global_step/sec: 7.00004
2018-07-06 07:22:32,626 INFO - INFO:tensorflow:loss = 0.022138823, step = 7077 (14.286 sec)
2018-07-06 07:22:46,913 INFO - INFO:tensorflow:global_step/sec: 6.99902
2018-07-06 07:22:46,914 INFO - INFO:tensorflow:loss = 0.018304389, step = 7177 (14.288 sec)
2018-07-06 07:23:01,204 INFO - INFO:tensorflow:global_step/sec: 6.9975
2018-07-06 07:23:01,204 INFO - INFO:tensorflow:loss = 0.019426476, step = 7277 (14.291 sec)
2018-07-06 07:23:02,035 INFO - INFO:tensorflow:Saving checkpoints for 7284 into /output/model.ckpt.
2018-07-06 07:23:02,735 INFO - Starting to evaluate.
2018-07-06 07:23:02,735 INFO - INFO:tensorflow:Loss for final step: 0.027920797.
2018-07-06 07:23:03,245 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:23:05,791 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:23:05,813 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:23:05
2018-07-06 07:23:06,372 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:23:06,517 INFO - 2018-07-06 14:23:06.371170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:23:06,517 INFO - 2018-07-06 14:23:06.371264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:23:06,518 INFO - 2018-07-06 14:23:06.371289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:23:06,518 INFO - 2018-07-06 14:23:06.371305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:23:06,518 INFO - 2018-07-06 14:23:06.371455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:23:06,518 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7284
2018-07-06 07:23:07,056 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:23:07,093 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:24:12,649 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:24:12
2018-07-06 07:24:12,650 INFO - INFO:tensorflow:Saving dict for global step 7284: global_step = 7284, loss = 0.015570666
2018-07-06 07:24:12,650 INFO - {'loss': 0.015570666, 'global_step': 7284}
2018-07-06 07:24:12,650 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9db0124b90>]
2018-07-06 07:24:12,650 INFO - Starting a training cycle.
2018-07-06 07:24:13,238 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:24:15,814 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:24:15,815 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:24:16,494 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:24:16,495 INFO - 2018-07-06 14:24:16.493916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:24:16,495 INFO - 2018-07-06 14:24:16.494011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:24:16,495 INFO - 2018-07-06 14:24:16.494038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:24:16,495 INFO - 2018-07-06 14:24:16.494058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:24:16,495 INFO - 2018-07-06 14:24:16.494231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:24:16,495 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7284
2018-07-06 07:24:17,192 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:24:17,221 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:24:21,920 INFO - INFO:tensorflow:Saving checkpoints for 7285 into /output/model.ckpt.
2018-07-06 07:24:22,674 INFO - INFO:tensorflow:loss = 0.025489815, step = 7284
2018-07-06 07:24:37,486 INFO - INFO:tensorflow:global_step/sec: 6.751
2018-07-06 07:24:37,486 INFO - INFO:tensorflow:loss = 0.020771867, step = 7384 (14.813 sec)
2018-07-06 07:24:51,782 INFO - INFO:tensorflow:global_step/sec: 6.99478
2018-07-06 07:24:51,783 INFO - INFO:tensorflow:loss = 0.023175538, step = 7484 (14.296 sec)
2018-07-06 07:25:06,077 INFO - INFO:tensorflow:global_step/sec: 6.99551
2018-07-06 07:25:06,078 INFO - INFO:tensorflow:loss = 0.025561504, step = 7584 (14.295 sec)
2018-07-06 07:25:20,380 INFO - INFO:tensorflow:global_step/sec: 6.99169
2018-07-06 07:25:20,380 INFO - INFO:tensorflow:loss = 0.02340959, step = 7684 (14.303 sec)
2018-07-06 07:25:34,682 INFO - INFO:tensorflow:global_step/sec: 6.99206
2018-07-06 07:25:34,683 INFO - INFO:tensorflow:loss = 0.025119362, step = 7784 (14.302 sec)
2018-07-06 07:25:48,979 INFO - INFO:tensorflow:global_step/sec: 6.99447
2018-07-06 07:25:48,979 INFO - INFO:tensorflow:loss = 0.01864208, step = 7884 (14.297 sec)
2018-07-06 07:25:49,806 INFO - INFO:tensorflow:Saving checkpoints for 7891 into /output/model.ckpt.
2018-07-06 07:25:50,624 INFO - Starting to evaluate.
2018-07-06 07:25:51,061 INFO - INFO:tensorflow:Loss for final step: 0.033001635.
2018-07-06 07:25:51,141 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:25:53,449 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:25:53,939 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:25:53
2018-07-06 07:25:54,267 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:25:54,267 INFO - 2018-07-06 14:25:54.266114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:25:54,267 INFO - 2018-07-06 14:25:54.266209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:25:54,267 INFO - 2018-07-06 14:25:54.266233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:25:54,267 INFO - 2018-07-06 14:25:54.266249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:25:54,267 INFO - 2018-07-06 14:25:54.266429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:25:54,268 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7891
2018-07-06 07:25:54,946 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:25:54,973 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:27:00,505 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:27:00
2018-07-06 07:27:00,505 INFO - INFO:tensorflow:Saving dict for global step 7891: global_step = 7891, loss = 0.015297853
2018-07-06 07:27:00,505 INFO - {'loss': 0.015297853, 'global_step': 7891}
2018-07-06 07:27:00,505 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dd015ef90>]
2018-07-06 07:27:00,506 INFO - Starting a training cycle.
2018-07-06 07:27:01,092 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:27:03,684 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:27:03,685 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:27:04,248 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:27:04,249 INFO - 2018-07-06 14:27:04.248039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:27:04,249 INFO - 2018-07-06 14:27:04.248140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:27:04,249 INFO - 2018-07-06 14:27:04.248175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:27:04,249 INFO - 2018-07-06 14:27:04.248204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:27:04,249 INFO - 2018-07-06 14:27:04.248370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:27:04,249 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7891
2018-07-06 07:27:04,983 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:27:05,011 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:27:09,718 INFO - INFO:tensorflow:Saving checkpoints for 7892 into /output/model.ckpt.
2018-07-06 07:27:10,682 INFO - INFO:tensorflow:loss = 0.020526953, step = 7891
2018-07-06 07:27:25,519 INFO - INFO:tensorflow:global_step/sec: 6.73934
2018-07-06 07:27:25,520 INFO - INFO:tensorflow:loss = 0.021046769, step = 7991 (14.839 sec)
2018-07-06 07:27:39,819 INFO - INFO:tensorflow:global_step/sec: 6.99326
2018-07-06 07:27:40,221 INFO - INFO:tensorflow:loss = 0.024167553, step = 8091 (14.299 sec)
2018-07-06 07:27:54,112 INFO - INFO:tensorflow:global_step/sec: 6.99643
2018-07-06 07:27:54,112 INFO - INFO:tensorflow:loss = 0.026858985, step = 8191 (14.293 sec)
2018-07-06 07:28:08,404 INFO - INFO:tensorflow:global_step/sec: 6.9969
2018-07-06 07:28:08,405 INFO - INFO:tensorflow:loss = 0.020328082, step = 8291 (14.292 sec)
2018-07-06 07:28:22,691 INFO - INFO:tensorflow:global_step/sec: 6.99938
2018-07-06 07:28:22,692 INFO - INFO:tensorflow:loss = 0.02065707, step = 8391 (14.287 sec)
2018-07-06 07:28:36,981 INFO - INFO:tensorflow:global_step/sec: 6.99792
2018-07-06 07:28:36,982 INFO - INFO:tensorflow:loss = 0.023044137, step = 8491 (14.290 sec)
2018-07-06 07:28:37,805 INFO - INFO:tensorflow:Saving checkpoints for 8498 into /output/model.ckpt.
2018-07-06 07:28:38,497 INFO - INFO:tensorflow:Loss for final step: 0.026111795.
2018-07-06 07:28:38,498 INFO - Starting to evaluate.
2018-07-06 07:28:39,000 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:28:41,537 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:28:41,558 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:28:41
2018-07-06 07:28:42,119 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:28:42,119 INFO - 2018-07-06 14:28:42.118684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:28:42,120 INFO - 2018-07-06 14:28:42.118782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:28:42,120 INFO - 2018-07-06 14:28:42.118813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:28:42,120 INFO - 2018-07-06 14:28:42.118842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:28:42,120 INFO - 2018-07-06 14:28:42.119007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:28:42,120 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-8498
2018-07-06 07:28:42,819 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:28:42,854 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:29:48,262 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:29:48
2018-07-06 07:29:48,262 INFO - INFO:tensorflow:Saving dict for global step 8498: global_step = 8498, loss = 0.014574602
2018-07-06 07:29:48,262 INFO - {'loss': 0.014574602, 'global_step': 8498}
2018-07-06 07:29:48,263 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9d901d10d0>]
2018-07-06 07:29:48,263 INFO - Starting a training cycle.
2018-07-06 07:29:48,836 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:29:51,414 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:29:51,415 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:29:52,096 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:29:52,096 INFO - 2018-07-06 14:29:52.094877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:29:52,096 INFO - 2018-07-06 14:29:52.094978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:29:52,096 INFO - 2018-07-06 14:29:52.095004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:29:52,096 INFO - 2018-07-06 14:29:52.095018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:29:52,097 INFO - 2018-07-06 14:29:52.095169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:29:52,097 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-8498
2018-07-06 07:29:52,788 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:29:52,822 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:29:57,395 INFO - INFO:tensorflow:Saving checkpoints for 8499 into /output/model.ckpt.
2018-07-06 07:29:58,129 INFO - INFO:tensorflow:loss = 0.019918853, step = 8498
2018-07-06 07:30:12,964 INFO - INFO:tensorflow:global_step/sec: 6.74044
2018-07-06 07:30:12,964 INFO - INFO:tensorflow:loss = 0.019486813, step = 8598 (14.836 sec)
2018-07-06 07:30:27,256 INFO - INFO:tensorflow:global_step/sec: 6.9969
2018-07-06 07:30:27,257 INFO - INFO:tensorflow:loss = 0.016216017, step = 8698 (14.292 sec)
2018-07-06 07:30:41,546 INFO - INFO:tensorflow:global_step/sec: 6.99794
2018-07-06 07:30:41,546 INFO - INFO:tensorflow:loss = 0.022534037, step = 8798 (14.290 sec)
2018-07-06 07:30:55,830 INFO - INFO:tensorflow:global_step/sec: 7.00071
2018-07-06 07:30:55,831 INFO - INFO:tensorflow:loss = 0.019833282, step = 8898 (14.284 sec)
2018-07-06 07:31:10,124 INFO - INFO:tensorflow:global_step/sec: 6.99785
2018-07-06 07:31:10,124 INFO - INFO:tensorflow:loss = 0.01734224, step = 8998 (14.290 sec)
2018-07-06 07:31:24,406 INFO - INFO:tensorflow:global_step/sec: 7.00024
2018-07-06 07:31:24,406 INFO - INFO:tensorflow:loss = 0.020113988, step = 9098 (14.285 sec)
2018-07-06 07:31:25,232 INFO - INFO:tensorflow:Saving checkpoints for 9105 into /output/model.ckpt.
2018-07-06 07:31:25,935 INFO - INFO:tensorflow:Loss for final step: 0.019192731.
2018-07-06 07:31:25,935 INFO - Starting to evaluate.
2018-07-06 07:31:26,448 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:31:28,965 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:31:28,986 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:31:28
2018-07-06 07:31:29,647 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:31:29,647 INFO - 2018-07-06 14:31:29.645960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:31:29,647 INFO - 2018-07-06 14:31:29.646060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:31:29,647 INFO - 2018-07-06 14:31:29.646085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:31:29,647 INFO - 2018-07-06 14:31:29.646106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:31:29,647 INFO - 2018-07-06 14:31:29.646269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:31:29,648 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9105
2018-07-06 07:31:30,320 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:31:30,346 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:32:35,680 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:32:35
2018-07-06 07:32:35,681 INFO - INFO:tensorflow:Saving dict for global step 9105: global_step = 9105, loss = 0.0147541445
2018-07-06 07:32:35,681 INFO - {'loss': 0.0147541445, 'global_step': 9105}
2018-07-06 07:32:35,681 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9db0ec2950>]
2018-07-06 07:32:35,681 INFO - Starting a training cycle.
2018-07-06 07:32:36,256 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:32:38,636 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:32:38,637 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:32:39,270 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:32:39,271 INFO - 2018-07-06 14:32:39.269562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:32:39,271 INFO - 2018-07-06 14:32:39.269659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:32:39,271 INFO - 2018-07-06 14:32:39.269684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:32:39,271 INFO - 2018-07-06 14:32:39.269698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:32:39,271 INFO - 2018-07-06 14:32:39.269849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:32:39,271 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9105
2018-07-06 07:32:39,956 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:32:39,984 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:32:44,603 INFO - INFO:tensorflow:Saving checkpoints for 9106 into /output/model.ckpt.
2018-07-06 07:32:45,341 INFO - INFO:tensorflow:loss = 0.017229406, step = 9105
2018-07-06 07:33:00,160 INFO - INFO:tensorflow:global_step/sec: 6.74794
2018-07-06 07:33:00,161 INFO - INFO:tensorflow:loss = 0.027674045, step = 9205 (14.820 sec)
2018-07-06 07:33:14,455 INFO - INFO:tensorflow:global_step/sec: 6.99551
2018-07-06 07:33:14,456 INFO - INFO:tensorflow:loss = 0.016883075, step = 9305 (14.295 sec)
2018-07-06 07:33:28,746 INFO - INFO:tensorflow:global_step/sec: 6.99754
2018-07-06 07:33:28,747 INFO - INFO:tensorflow:loss = 0.016972434, step = 9405 (14.291 sec)
2018-07-06 07:33:43,042 INFO - INFO:tensorflow:global_step/sec: 6.99482
2018-07-06 07:33:43,043 INFO - INFO:tensorflow:loss = 0.023823237, step = 9505 (14.296 sec)
2018-07-06 07:33:57,341 INFO - INFO:tensorflow:global_step/sec: 6.99371
2018-07-06 07:33:57,342 INFO - INFO:tensorflow:loss = 0.02134448, step = 9605 (14.299 sec)
2018-07-06 07:34:11,635 INFO - INFO:tensorflow:global_step/sec: 6.99561
2018-07-06 07:34:11,636 INFO - INFO:tensorflow:loss = 0.02593796, step = 9705 (14.294 sec)
2018-07-06 07:34:12,470 INFO - INFO:tensorflow:Saving checkpoints for 9712 into /output/model.ckpt.
2018-07-06 07:34:13,309 INFO - INFO:tensorflow:Loss for final step: 0.019335682.
2018-07-06 07:34:13,310 INFO - Starting to evaluate.
2018-07-06 07:34:13,812 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:34:16,110 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:34:16,133 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:34:16
2018-07-06 07:34:16,887 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:34:16,887 INFO - 2018-07-06 14:34:16.886519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:34:16,887 INFO - 2018-07-06 14:34:16.886614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:34:16,887 INFO - 2018-07-06 14:34:16.886639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:34:16,888 INFO - 2018-07-06 14:34:16.886655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:34:16,888 INFO - 2018-07-06 14:34:16.886804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:34:16,888 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9712
2018-07-06 07:34:17,599 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:34:17,998 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:35:22,900 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:35:22
2018-07-06 07:35:22,900 INFO - INFO:tensorflow:Saving dict for global step 9712: global_step = 9712, loss = 0.014658521
2018-07-06 07:35:22,901 INFO - {'loss': 0.014658521, 'global_step': 9712}
2018-07-06 07:35:22,901 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dcff12150>]
2018-07-06 07:35:22,901 INFO - Starting a training cycle.
2018-07-06 07:35:23,478 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:35:25,973 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:35:25,974 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:35:26,521 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:35:26,522 INFO - 2018-07-06 14:35:26.520467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:35:26,522 INFO - 2018-07-06 14:35:26.520564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:35:26,522 INFO - 2018-07-06 14:35:26.520595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:35:26,522 INFO - 2018-07-06 14:35:26.520619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:35:26,522 INFO - 2018-07-06 14:35:26.520789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:35:26,523 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9712
2018-07-06 07:35:27,214 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:35:27,249 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:35:31,891 INFO - INFO:tensorflow:Saving checkpoints for 9713 into /output/model.ckpt.
2018-07-06 07:35:32,755 INFO - INFO:tensorflow:loss = 0.020115001, step = 9712
2018-07-06 07:35:47,595 INFO - INFO:tensorflow:global_step/sec: 6.73848
2018-07-06 07:35:47,595 INFO - INFO:tensorflow:loss = 0.018279467, step = 9812 (14.841 sec)
2018-07-06 07:36:01,896 INFO - INFO:tensorflow:global_step/sec: 6.99257
2018-07-06 07:36:01,896 INFO - INFO:tensorflow:loss = 0.01956807, step = 9912 (14.301 sec)
2018-07-06 07:36:16,186 INFO - INFO:tensorflow:global_step/sec: 6.99754
2018-07-06 07:36:16,187 INFO - INFO:tensorflow:loss = 0.019143827, step = 10012 (14.291 sec)
2018-07-06 07:36:30,500 INFO - INFO:tensorflow:global_step/sec: 6.98612
2018-07-06 07:36:30,501 INFO - INFO:tensorflow:loss = 0.020457674, step = 10112 (14.314 sec)
2018-07-06 07:36:44,814 INFO - INFO:tensorflow:global_step/sec: 6.98646
2018-07-06 07:36:44,814 INFO - INFO:tensorflow:loss = 0.017072035, step = 10212 (14.314 sec)
2018-07-06 07:36:59,118 INFO - INFO:tensorflow:global_step/sec: 6.99119
2018-07-06 07:36:59,118 INFO - INFO:tensorflow:loss = 0.018133791, step = 10312 (14.304 sec)
2018-07-06 07:36:59,943 INFO - INFO:tensorflow:Saving checkpoints for 10319 into /output/model.ckpt.
2018-07-06 07:37:00,649 INFO - Starting to evaluate.
2018-07-06 07:37:00,650 INFO - INFO:tensorflow:Loss for final step: 0.018700851.
2018-07-06 07:37:01,157 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:37:03,674 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:37:03,695 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:37:03
2018-07-06 07:37:04,240 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:37:04,241 INFO - 2018-07-06 14:37:04.239931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:37:04,241 INFO - 2018-07-06 14:37:04.240036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:37:04,241 INFO - 2018-07-06 14:37:04.240067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:37:04,241 INFO - 2018-07-06 14:37:04.240093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:37:04,241 INFO - 2018-07-06 14:37:04.240268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:37:04,241 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10319
2018-07-06 07:37:04,909 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:37:04,938 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:38:10,564 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:38:10
2018-07-06 07:38:10,565 INFO - INFO:tensorflow:Saving dict for global step 10319: global_step = 10319, loss = 0.014019106
2018-07-06 07:38:10,565 INFO - {'loss': 0.014019106, 'global_step': 10319}
2018-07-06 07:38:10,565 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc632b990>]
2018-07-06 07:38:10,565 INFO - Starting a training cycle.
2018-07-06 07:38:11,138 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:38:13,599 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:38:13,600 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:38:14,241 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:38:14,242 INFO - 2018-07-06 14:38:14.240702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:38:14,242 INFO - 2018-07-06 14:38:14.240799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:38:14,242 INFO - 2018-07-06 14:38:14.240825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:38:14,242 INFO - 2018-07-06 14:38:14.240846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:38:14,242 INFO - 2018-07-06 14:38:14.241027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:38:14,243 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10319
2018-07-06 07:38:14,919 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:38:14,952 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:38:19,537 INFO - INFO:tensorflow:Saving checkpoints for 10320 into /output/model.ckpt.
2018-07-06 07:38:20,275 INFO - INFO:tensorflow:loss = 0.01974984, step = 10319
2018-07-06 07:38:35,100 INFO - INFO:tensorflow:global_step/sec: 6.74483
2018-07-06 07:38:35,101 INFO - INFO:tensorflow:loss = 0.030908205, step = 10419 (14.827 sec)
2018-07-06 07:38:49,393 INFO - INFO:tensorflow:global_step/sec: 6.99661
2018-07-06 07:38:49,394 INFO - INFO:tensorflow:loss = 0.018937945, step = 10519 (14.293 sec)
2018-07-06 07:39:03,679 INFO - INFO:tensorflow:global_step/sec: 6.99974
2018-07-06 07:39:03,680 INFO - INFO:tensorflow:loss = 0.01768852, step = 10619 (14.286 sec)
2018-07-06 07:39:17,966 INFO - INFO:tensorflow:global_step/sec: 6.99962
2018-07-06 07:39:17,966 INFO - INFO:tensorflow:loss = 0.017551184, step = 10719 (14.287 sec)
2018-07-06 07:39:32,258 INFO - INFO:tensorflow:global_step/sec: 6.99649
2018-07-06 07:39:32,259 INFO - INFO:tensorflow:loss = 0.016564425, step = 10819 (14.293 sec)
2018-07-06 07:39:46,555 INFO - INFO:tensorflow:global_step/sec: 6.99466
2018-07-06 07:39:46,556 INFO - INFO:tensorflow:loss = 0.025033498, step = 10919 (14.297 sec)
2018-07-06 07:39:47,381 INFO - INFO:tensorflow:Saving checkpoints for 10926 into /output/model.ckpt.
2018-07-06 07:39:48,106 INFO - INFO:tensorflow:Loss for final step: 0.017579831.
2018-07-06 07:39:48,106 INFO - Starting to evaluate.
2018-07-06 07:39:48,616 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:39:51,098 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:39:51,119 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:39:51
2018-07-06 07:39:51,778 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:39:52,656 INFO - 2018-07-06 14:39:51.777672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:39:52,656 INFO - 2018-07-06 14:39:51.777759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:39:52,656 INFO - 2018-07-06 14:39:51.777783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:39:52,656 INFO - 2018-07-06 14:39:51.777800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:39:52,656 INFO - 2018-07-06 14:39:51.777957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:39:52,656 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10926
2018-07-06 07:39:52,657 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:39:52,657 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:40:58,143 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:40:58
2018-07-06 07:40:58,143 INFO - INFO:tensorflow:Saving dict for global step 10926: global_step = 10926, loss = 0.013643841
2018-07-06 07:40:58,143 INFO - {'loss': 0.013643841, 'global_step': 10926}
2018-07-06 07:40:58,144 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc5c43ad0>]
2018-07-06 07:40:58,144 INFO - Starting a training cycle.
2018-07-06 07:40:58,718 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:41:01,315 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:41:01,315 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:41:01,880 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:41:01,881 INFO - 2018-07-06 14:41:01.880006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:41:01,881 INFO - 2018-07-06 14:41:01.880108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:41:01,881 INFO - 2018-07-06 14:41:01.880133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:41:01,881 INFO - 2018-07-06 14:41:01.880150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:41:01,881 INFO - 2018-07-06 14:41:01.880306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:41:01,881 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10926
2018-07-06 07:41:02,594 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:41:02,620 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:41:07,252 INFO - INFO:tensorflow:Saving checkpoints for 10927 into /output/model.ckpt.
2018-07-06 07:41:08,014 INFO - INFO:tensorflow:loss = 0.028748998, step = 10926
2018-07-06 07:41:22,864 INFO - INFO:tensorflow:global_step/sec: 6.73395
2018-07-06 07:41:22,864 INFO - INFO:tensorflow:loss = 0.029629385, step = 11026 (14.851 sec)
2018-07-06 07:41:37,159 INFO - INFO:tensorflow:global_step/sec: 6.995
2018-07-06 07:41:37,160 INFO - INFO:tensorflow:loss = 0.018635307, step = 11126 (14.296 sec)
2018-07-06 07:41:51,456 INFO - INFO:tensorflow:global_step/sec: 6.99448
2018-07-06 07:41:51,652 INFO - INFO:tensorflow:loss = 0.017432138, step = 11226 (14.297 sec)
2018-07-06 07:42:05,752 INFO - INFO:tensorflow:global_step/sec: 6.99503
2018-07-06 07:42:05,753 INFO - INFO:tensorflow:loss = 0.029062297, step = 11326 (14.296 sec)
2018-07-06 07:42:20,043 INFO - INFO:tensorflow:global_step/sec: 6.99722
2018-07-06 07:42:20,044 INFO - INFO:tensorflow:loss = 0.029734563, step = 11426 (14.291 sec)
2018-07-06 07:42:34,339 INFO - INFO:tensorflow:global_step/sec: 6.995
2018-07-06 07:42:34,340 INFO - INFO:tensorflow:loss = 0.02338785, step = 11526 (14.296 sec)
2018-07-06 07:42:35,163 INFO - INFO:tensorflow:Saving checkpoints for 11533 into /output/model.ckpt.
2018-07-06 07:42:35,977 INFO - Starting to evaluate.
2018-07-06 07:42:35,977 INFO - INFO:tensorflow:Loss for final step: 0.025124216.
2018-07-06 07:42:36,486 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:42:38,795 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:42:38,817 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:42:38
2018-07-06 07:42:39,570 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:42:39,570 INFO - 2018-07-06 14:42:39.569608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:42:39,571 INFO - 2018-07-06 14:42:39.569704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:42:39,571 INFO - 2018-07-06 14:42:39.569728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:42:39,571 INFO - 2018-07-06 14:42:39.569742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:42:39,571 INFO - 2018-07-06 14:42:39.569895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:42:39,571 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-11533
2018-07-06 07:42:40,266 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:42:40,690 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:43:45,827 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:43:45
2018-07-06 07:43:45,827 INFO - INFO:tensorflow:Saving dict for global step 11533: global_step = 11533, loss = 0.013401381
2018-07-06 07:43:45,827 INFO - {'loss': 0.013401381, 'global_step': 11533}
2018-07-06 07:43:45,828 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9992770850>]
2018-07-06 07:43:45,828 INFO - Starting a training cycle.
2018-07-06 07:43:46,421 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:43:49,002 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:43:49,003 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:43:49,571 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:43:49,572 INFO - 2018-07-06 14:43:49.570799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:43:49,572 INFO - 2018-07-06 14:43:49.570914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:43:49,572 INFO - 2018-07-06 14:43:49.570940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:43:49,572 INFO - 2018-07-06 14:43:49.570958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:43:49,573 INFO - 2018-07-06 14:43:49.571119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:43:49,573 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-11533
2018-07-06 07:43:50,385 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:43:50,419 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:43:54,993 INFO - INFO:tensorflow:Saving checkpoints for 11534 into /output/model.ckpt.
2018-07-06 07:43:55,738 INFO - INFO:tensorflow:loss = 0.016370542, step = 11533
2018-07-06 07:44:10,591 INFO - INFO:tensorflow:global_step/sec: 6.73266
2018-07-06 07:44:10,591 INFO - INFO:tensorflow:loss = 0.01680084, step = 11633 (14.853 sec)
2018-07-06 07:44:24,900 INFO - INFO:tensorflow:global_step/sec: 6.98854
2018-07-06 07:44:24,900 INFO - INFO:tensorflow:loss = 0.021056779, step = 11733 (14.309 sec)
2018-07-06 07:44:39,197 INFO - INFO:tensorflow:global_step/sec: 6.99426
2018-07-06 07:44:39,198 INFO - INFO:tensorflow:loss = 0.018824495, step = 11833 (14.297 sec)
2018-07-06 07:44:53,501 INFO - INFO:tensorflow:global_step/sec: 6.99108
2018-07-06 07:44:53,501 INFO - INFO:tensorflow:loss = 0.018014098, step = 11933 (14.304 sec)
2018-07-06 07:45:07,803 INFO - INFO:tensorflow:global_step/sec: 6.99175
2018-07-06 07:45:07,804 INFO - INFO:tensorflow:loss = 0.017145876, step = 12033 (14.303 sec)
2018-07-06 07:45:22,111 INFO - INFO:tensorflow:global_step/sec: 6.98933
2018-07-06 07:45:22,111 INFO - INFO:tensorflow:loss = 0.020669643, step = 12133 (14.308 sec)
2018-07-06 07:45:22,937 INFO - INFO:tensorflow:Saving checkpoints for 12140 into /output/model.ckpt.
2018-07-06 07:45:23,616 INFO - INFO:tensorflow:Loss for final step: 0.023141582.
2018-07-06 07:45:23,616 INFO - Starting to evaluate.
2018-07-06 07:45:24,121 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:45:26,607 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:45:26,628 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:45:26
2018-07-06 07:45:27,175 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:45:27,176 INFO - 2018-07-06 14:45:27.174997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:45:27,176 INFO - 2018-07-06 14:45:27.175085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:45:27,176 INFO - 2018-07-06 14:45:27.175112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:45:27,176 INFO - 2018-07-06 14:45:27.175128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:45:27,176 INFO - 2018-07-06 14:45:27.175281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:45:27,176 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12140
2018-07-06 07:45:27,880 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:45:27,908 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:46:33,375 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:46:33
2018-07-06 07:46:33,376 INFO - INFO:tensorflow:Saving dict for global step 12140: global_step = 12140, loss = 0.013871309
2018-07-06 07:46:33,376 INFO - {'loss': 0.013871309, 'global_step': 12140}
2018-07-06 07:46:33,376 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9dc69c7b10>]
2018-07-06 07:46:33,376 INFO - Starting a training cycle.
2018-07-06 07:46:33,939 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:46:36,396 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:46:36,396 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:46:37,030 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:46:37,030 INFO - 2018-07-06 14:46:37.029449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:46:37,031 INFO - 2018-07-06 14:46:37.029551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:46:37,031 INFO - 2018-07-06 14:46:37.029575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:46:37,031 INFO - 2018-07-06 14:46:37.029591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:46:37,031 INFO - 2018-07-06 14:46:37.029760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:46:37,031 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12140
2018-07-06 07:46:37,736 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:46:37,764 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:46:42,366 INFO - INFO:tensorflow:Saving checkpoints for 12141 into /output/model.ckpt.
2018-07-06 07:46:43,133 INFO - INFO:tensorflow:loss = 0.029188972, step = 12140
2018-07-06 07:46:57,986 INFO - INFO:tensorflow:global_step/sec: 6.73239
2018-07-06 07:46:57,986 INFO - INFO:tensorflow:loss = 0.016012905, step = 12240 (14.854 sec)
2018-07-06 07:47:12,284 INFO - INFO:tensorflow:global_step/sec: 6.99475
2018-07-06 07:47:12,285 INFO - INFO:tensorflow:loss = 0.020537822, step = 12340 (14.296 sec)
2018-07-06 07:47:26,588 INFO - INFO:tensorflow:global_step/sec: 6.99033
2018-07-06 07:47:26,588 INFO - INFO:tensorflow:loss = 0.027232368, step = 12440 (14.305 sec)
2018-07-06 07:47:40,886 INFO - INFO:tensorflow:global_step/sec: 6.99377
2018-07-06 07:47:40,887 INFO - INFO:tensorflow:loss = 0.018222032, step = 12540 (14.298 sec)
2018-07-06 07:47:55,182 INFO - INFO:tensorflow:global_step/sec: 6.99504
2018-07-06 07:47:55,182 INFO - INFO:tensorflow:loss = 0.024216454, step = 12640 (14.296 sec)
2018-07-06 07:48:09,474 INFO - INFO:tensorflow:global_step/sec: 6.99679
2018-07-06 07:48:09,475 INFO - INFO:tensorflow:loss = 0.025937367, step = 12740 (14.292 sec)
2018-07-06 07:48:10,303 INFO - INFO:tensorflow:Saving checkpoints for 12747 into /output/model.ckpt.
2018-07-06 07:48:11,168 INFO - Starting to evaluate.
2018-07-06 07:48:11,168 INFO - INFO:tensorflow:Loss for final step: 0.02264903.
2018-07-06 07:48:11,673 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:48:13,991 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:48:14,012 INFO - INFO:tensorflow:Starting evaluation at 2018-07-06-14:48:14
2018-07-06 07:48:14,770 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:48:14,771 INFO - 2018-07-06 14:48:14.769761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:48:14,771 INFO - 2018-07-06 14:48:14.769860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:48:14,771 INFO - 2018-07-06 14:48:14.769893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:48:14,771 INFO - 2018-07-06 14:48:14.769920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:48:14,771 INFO - 2018-07-06 14:48:14.770087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:48:14,772 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12747
2018-07-06 07:48:15,475 INFO - INFO:tensorflow:Running local_init_op.
2018-07-06 07:48:16,095 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-06 07:49:20,857 INFO - INFO:tensorflow:Finished evaluation at 2018-07-06-14:49:20
2018-07-06 07:49:20,857 INFO - INFO:tensorflow:Saving dict for global step 12747: global_step = 12747, loss = 0.0131491525
2018-07-06 07:49:20,857 INFO - {'loss': 0.0131491525, 'global_step': 12747}
2018-07-06 07:49:20,857 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f9992bbd2d0>]
2018-07-06 07:49:20,858 INFO - Starting a training cycle.
2018-07-06 07:49:21,436 INFO - INFO:tensorflow:Calling model_fn.
2018-07-06 07:49:23,921 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-06 07:49:23,922 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-06 07:49:24,467 INFO - INFO:tensorflow:Graph was finalized.
2018-07-06 07:49:24,467 INFO - 2018-07-06 14:49:24.466641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-06 07:49:24,467 INFO - 2018-07-06 14:49:24.466738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-06 07:49:24,467 INFO - 2018-07-06 14:49:24.466763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-06 07:49:24,468 INFO - 2018-07-06 14:49:24.466777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-06 07:49:24,468 INFO - 2018-07-06 14:49:24.466928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14400 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2018-07-06 07:49:24,468 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12747
2018-07-06 07:49:25,177 INFO - INFO:tensorflow:Running local_init_op.