-
Notifications
You must be signed in to change notification settings - Fork 0
/
tf-loss-ds2.txt
3848 lines (3797 loc) · 379 KB
/
tf-loss-ds2.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2018-07-17 07:40:51,584 INFO - Preparing to run TaskInstance <TaskInstance: mrozh/projects/dan-tf/59 (id: xMoe76fdZsqZfzDEiWgSG5)
2018-07-17 07:40:51,608 INFO - Starting attempt 1
2018-07-17 07:40:51,629 INFO - Downloading and setting up data sources
2018-07-17 07:40:51,644 INFO - Downloading and mounting dataset. ETA: 12 seconds
2018-07-17 07:40:59,468 INFO - Downloading and mounting dataset. ETA: 62 seconds
2018-07-17 07:41:33,197 INFO - Pulling Docker image: floydhub/tensorflow:1.7.0-gpu.cuda9cudnn7-py2_aws.30
2018-07-17 07:41:34,766 INFO - Starting container...
2018-07-17 07:41:34,965 INFO -
################################################################################
2018-07-17 07:41:34,965 INFO - Run Output:
2018-07-17 07:41:35,129 INFO - Starting services.
2018-07-17 07:42:50,980 INFO - INFO:tensorflow:Using config: {'_save_checkpoints_secs': 1000000000.0, '_session_config': allow_soft_placement: true
2018-07-17 07:42:50,981 INFO - , '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f3e9897d990>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/output', '_save_summary_steps': 100}
2018-07-17 07:42:50,981 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e98968850>]
2018-07-17 07:42:50,982 INFO - Starting a training cycle.
2018-07-17 07:42:52,551 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 07:43:02,359 INFO - WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
2018-07-17 07:43:02,359 INFO - Instructions for updating:
2018-07-17 07:43:02,360 INFO - Use the retry module or similar alternatives.
2018-07-17 07:43:17,111 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 07:43:17,111 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 07:43:19,025 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 07:43:19,100 INFO - 2018-07-17 14:43:19.099033: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-17 07:43:19,101 INFO - 2018-07-17 14:43:19.099420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
2018-07-17 07:43:19,101 INFO - name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8755
2018-07-17 07:43:19,101 INFO - pciBusID: 0000:00:1e.0
2018-07-17 07:43:19,101 INFO - totalMemory: 11.17GiB freeMemory: 10.99GiB
2018-07-17 07:43:19,101 INFO - 2018-07-17 14:43:19.099473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 07:43:19,411 INFO - 2018-07-17 14:43:19.410537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 07:43:19,412 INFO - 2018-07-17 14:43:19.410641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 07:43:19,412 INFO - 2018-07-17 14:43:19.410660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 07:43:19,412 INFO - 2018-07-17 14:43:19.410992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 07:43:19,558 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-74056
2018-07-17 07:43:21,947 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 07:43:21,981 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 07:44:34,252 INFO - 2018-07-17 14:44:34.251062: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x8c4cae0
2018-07-17 07:44:57,940 INFO - INFO:tensorflow:Saving checkpoints for 74057 into /output/model.ckpt.
2018-07-17 07:44:58,877 INFO - INFO:tensorflow:loss = 0.014762228, step = 74056
2018-07-17 07:46:05,716 INFO - INFO:tensorflow:global_step/sec: 1.4961
2018-07-17 07:46:05,718 INFO - INFO:tensorflow:loss = 0.0077043017, step = 74156 (66.842 sec)
2018-07-17 07:47:12,511 INFO - INFO:tensorflow:global_step/sec: 1.49713
2018-07-17 07:47:12,521 INFO - INFO:tensorflow:loss = 0.0077707353, step = 74256 (66.793 sec)
2018-07-17 07:48:19,418 INFO - INFO:tensorflow:global_step/sec: 1.4946
2018-07-17 07:48:19,419 INFO - INFO:tensorflow:loss = 0.008037662, step = 74356 (66.907 sec)
2018-07-17 07:49:26,477 INFO - INFO:tensorflow:global_step/sec: 1.49123
2018-07-17 07:49:26,478 INFO - INFO:tensorflow:loss = 0.007868936, step = 74456 (67.060 sec)
2018-07-17 07:50:33,563 INFO - INFO:tensorflow:global_step/sec: 1.49061
2018-07-17 07:50:33,564 INFO - INFO:tensorflow:loss = 0.0078582205, step = 74556 (67.086 sec)
2018-07-17 07:51:40,665 INFO - INFO:tensorflow:global_step/sec: 1.49028
2018-07-17 07:51:40,666 INFO - INFO:tensorflow:loss = 0.008171882, step = 74656 (67.101 sec)
2018-07-17 07:51:47,162 INFO - INFO:tensorflow:Saving checkpoints for 74663 into /output/model.ckpt.
2018-07-17 07:51:48,150 INFO - Starting to evaluate.
2018-07-17 07:51:48,150 INFO - INFO:tensorflow:Loss for final step: 0.0075098355.
2018-07-17 07:51:48,685 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 07:51:51,096 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 07:51:51,117 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-14:51:51
2018-07-17 07:51:51,961 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 07:51:51,962 INFO - 2018-07-17 14:51:51.960884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 07:51:51,962 INFO - 2018-07-17 14:51:51.961030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 07:51:51,962 INFO - 2018-07-17 14:51:51.961067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 07:51:51,962 INFO - 2018-07-17 14:51:51.961092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 07:51:51,963 INFO - 2018-07-17 14:51:51.961273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 07:51:51,963 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-74663
2018-07-17 07:51:52,728 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 07:51:52,768 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 07:55:50,711 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-14:55:50
2018-07-17 07:55:50,720 INFO - INFO:tensorflow:Saving dict for global step 74663: global_step = 74663, loss = 0.039546475
2018-07-17 07:55:51,580 INFO - {'loss': 0.039546475, 'global_step': 74663}
2018-07-17 07:55:51,581 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8e480690>]
2018-07-17 07:55:51,581 INFO - Starting a training cycle.
2018-07-17 07:55:52,103 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 07:55:54,687 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 07:55:54,688 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 07:55:55,406 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 07:55:55,407 INFO - 2018-07-17 14:55:55.405170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 07:55:55,407 INFO - 2018-07-17 14:55:55.405316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 07:55:55,407 INFO - 2018-07-17 14:55:55.405351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 07:55:55,407 INFO - 2018-07-17 14:55:55.405376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 07:55:55,408 INFO - 2018-07-17 14:55:55.405551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 07:55:55,409 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-74663
2018-07-17 07:55:56,161 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 07:55:56,194 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 07:56:01,499 INFO - INFO:tensorflow:Saving checkpoints for 74664 into /output/model.ckpt.
2018-07-17 07:56:02,346 INFO - INFO:tensorflow:loss = 0.008334314, step = 74663
2018-07-17 07:57:09,944 INFO - INFO:tensorflow:global_step/sec: 1.47931
2018-07-17 07:57:09,946 INFO - INFO:tensorflow:loss = 0.007463559, step = 74763 (67.601 sec)
2018-07-17 07:58:17,122 INFO - INFO:tensorflow:global_step/sec: 1.48859
2018-07-17 07:58:17,122 INFO - INFO:tensorflow:loss = 0.007925411, step = 74863 (67.177 sec)
2018-07-17 07:59:24,268 INFO - INFO:tensorflow:global_step/sec: 1.48928
2018-07-17 07:59:24,269 INFO - INFO:tensorflow:loss = 0.008204818, step = 74963 (67.147 sec)
2018-07-17 08:00:31,380 INFO - INFO:tensorflow:global_step/sec: 1.49006
2018-07-17 08:00:31,380 INFO - INFO:tensorflow:loss = 0.007716546, step = 75063 (67.111 sec)
2018-07-17 08:01:38,430 INFO - INFO:tensorflow:global_step/sec: 1.49142
2018-07-17 08:01:38,431 INFO - INFO:tensorflow:loss = 0.0076737506, step = 75163 (67.051 sec)
2018-07-17 08:02:45,513 INFO - INFO:tensorflow:global_step/sec: 1.49069
2018-07-17 08:02:45,514 INFO - INFO:tensorflow:loss = 0.0077669383, step = 75263 (67.082 sec)
2018-07-17 08:02:49,377 INFO - INFO:tensorflow:Saving checkpoints for 75270 into /output/model.ckpt.
2018-07-17 08:02:50,139 INFO - INFO:tensorflow:Loss for final step: 0.007786456.
2018-07-17 08:02:50,140 INFO - Starting to evaluate.
2018-07-17 08:02:50,641 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:02:53,741 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:02:53,764 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:02:53
2018-07-17 08:02:54,470 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:02:54,470 INFO - 2018-07-17 15:02:54.469122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:02:54,470 INFO - 2018-07-17 15:02:54.469245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:02:54,471 INFO - 2018-07-17 15:02:54.469283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:02:54,471 INFO - 2018-07-17 15:02:54.469310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:02:54,471 INFO - 2018-07-17 15:02:54.469477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:02:54,471 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-75270
2018-07-17 08:02:55,203 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:02:55,234 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:06:53,218 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-15:06:53
2018-07-17 08:06:53,219 INFO - INFO:tensorflow:Saving dict for global step 75270: global_step = 75270, loss = 0.007965996
2018-07-17 08:06:53,219 INFO - {'loss': 0.007965996, 'global_step': 75270}
2018-07-17 08:06:53,219 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e48e24250>]
2018-07-17 08:06:53,219 INFO - Starting a training cycle.
2018-07-17 08:06:53,830 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:06:56,331 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:06:56,331 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 08:06:57,041 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:06:57,042 INFO - 2018-07-17 15:06:57.039883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:06:57,042 INFO - 2018-07-17 15:06:57.040005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:06:57,042 INFO - 2018-07-17 15:06:57.040043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:06:57,042 INFO - 2018-07-17 15:06:57.040067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:06:57,043 INFO - 2018-07-17 15:06:57.040259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:06:57,043 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-75270
2018-07-17 08:06:57,841 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:06:57,879 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:07:03,355 INFO - INFO:tensorflow:Saving checkpoints for 75271 into /output/model.ckpt.
2018-07-17 08:07:04,211 INFO - INFO:tensorflow:loss = 0.007876693, step = 75270
2018-07-17 08:08:11,645 INFO - INFO:tensorflow:global_step/sec: 1.48291
2018-07-17 08:08:11,646 INFO - INFO:tensorflow:loss = 0.007898286, step = 75370 (67.436 sec)
2018-07-17 08:09:18,555 INFO - INFO:tensorflow:global_step/sec: 1.49455
2018-07-17 08:09:18,557 INFO - INFO:tensorflow:loss = 0.007662351, step = 75470 (66.911 sec)
2018-07-17 08:10:25,520 INFO - INFO:tensorflow:global_step/sec: 1.49332
2018-07-17 08:10:25,521 INFO - INFO:tensorflow:loss = 0.0072138226, step = 75570 (66.964 sec)
2018-07-17 08:11:32,539 INFO - INFO:tensorflow:global_step/sec: 1.49212
2018-07-17 08:11:32,540 INFO - INFO:tensorflow:loss = 0.0076395487, step = 75670 (67.019 sec)
2018-07-17 08:12:39,615 INFO - INFO:tensorflow:global_step/sec: 1.49084
2018-07-17 08:12:39,616 INFO - INFO:tensorflow:loss = 0.0076666307, step = 75770 (67.076 sec)
2018-07-17 08:13:46,648 INFO - INFO:tensorflow:global_step/sec: 1.49181
2018-07-17 08:13:46,648 INFO - INFO:tensorflow:loss = 0.0076110903, step = 75870 (67.033 sec)
2018-07-17 08:13:50,522 INFO - INFO:tensorflow:Saving checkpoints for 75877 into /output/model.ckpt.
2018-07-17 08:13:51,511 INFO - Starting to evaluate.
2018-07-17 08:13:51,511 INFO - INFO:tensorflow:Loss for final step: 0.007608682.
2018-07-17 08:13:52,047 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:13:54,468 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:13:54,490 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:13:54
2018-07-17 08:13:55,348 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:13:55,349 INFO - 2018-07-17 15:13:55.347178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:13:55,349 INFO - 2018-07-17 15:13:55.347327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:13:55,349 INFO - 2018-07-17 15:13:55.347359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:13:55,349 INFO - 2018-07-17 15:13:55.347408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:13:55,349 INFO - 2018-07-17 15:13:55.347605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:13:55,350 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-75877
2018-07-17 08:13:56,163 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:13:56,202 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:17:54,505 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-15:17:54
2018-07-17 08:17:54,505 INFO - INFO:tensorflow:Saving dict for global step 75877: global_step = 75877, loss = 0.007696231
2018-07-17 08:17:54,505 INFO - {'loss': 0.007696231, 'global_step': 75877}
2018-07-17 08:17:54,505 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8e43b650>]
2018-07-17 08:17:54,506 INFO - Starting a training cycle.
2018-07-17 08:17:55,112 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:17:57,720 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:17:57,720 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 08:17:58,273 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:17:58,273 INFO - 2018-07-17 15:17:58.272335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:17:58,273 INFO - 2018-07-17 15:17:58.272480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:17:58,274 INFO - 2018-07-17 15:17:58.272525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:17:58,274 INFO - 2018-07-17 15:17:58.272554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:17:58,274 INFO - 2018-07-17 15:17:58.272739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:17:58,274 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-75877
2018-07-17 08:17:59,069 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:17:59,099 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:18:04,520 INFO - INFO:tensorflow:Saving checkpoints for 75878 into /output/model.ckpt.
2018-07-17 08:18:05,534 INFO - INFO:tensorflow:loss = 0.007505774, step = 75877
2018-07-17 08:19:12,973 INFO - INFO:tensorflow:global_step/sec: 1.48279
2018-07-17 08:19:12,974 INFO - INFO:tensorflow:loss = 0.007999809, step = 75977 (67.441 sec)
2018-07-17 08:20:19,942 INFO - INFO:tensorflow:global_step/sec: 1.49322
2018-07-17 08:20:19,943 INFO - INFO:tensorflow:loss = 0.007572571, step = 76077 (66.969 sec)
2018-07-17 08:21:26,935 INFO - INFO:tensorflow:global_step/sec: 1.49271
2018-07-17 08:21:26,936 INFO - INFO:tensorflow:loss = 0.007624653, step = 76177 (66.993 sec)
2018-07-17 08:22:33,895 INFO - INFO:tensorflow:global_step/sec: 1.49343
2018-07-17 08:22:33,896 INFO - INFO:tensorflow:loss = 0.0079095755, step = 76277 (66.960 sec)
2018-07-17 08:23:40,837 INFO - INFO:tensorflow:global_step/sec: 1.49382
2018-07-17 08:23:40,838 INFO - INFO:tensorflow:loss = 0.0077378713, step = 76377 (66.942 sec)
2018-07-17 08:24:47,850 INFO - INFO:tensorflow:global_step/sec: 1.49225
2018-07-17 08:24:47,851 INFO - INFO:tensorflow:loss = 0.0078786425, step = 76477 (67.014 sec)
2018-07-17 08:24:51,705 INFO - INFO:tensorflow:Saving checkpoints for 76484 into /output/model.ckpt.
2018-07-17 08:24:52,495 INFO - Starting to evaluate.
2018-07-17 08:24:52,496 INFO - INFO:tensorflow:Loss for final step: 0.007855113.
2018-07-17 08:24:52,998 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:24:55,405 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:24:55,428 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:24:55
2018-07-17 08:24:56,479 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:24:56,498 INFO - 2018-07-17 15:24:56.474499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:24:56,498 INFO - 2018-07-17 15:24:56.474596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:24:56,499 INFO - 2018-07-17 15:24:56.474616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:24:56,499 INFO - 2018-07-17 15:24:56.474630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:24:56,499 INFO - 2018-07-17 15:24:56.474769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:24:56,499 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-76484
2018-07-17 08:24:57,289 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:24:57,323 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:28:55,647 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-15:28:55
2018-07-17 08:28:55,647 INFO - INFO:tensorflow:Saving dict for global step 76484: global_step = 76484, loss = 0.007620401
2018-07-17 08:28:55,647 INFO - {'loss': 0.007620401, 'global_step': 76484}
2018-07-17 08:28:55,648 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8c25efd0>]
2018-07-17 08:28:55,648 INFO - Starting a training cycle.
2018-07-17 08:28:56,254 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:28:58,866 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:28:58,866 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 08:28:59,419 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:28:59,419 INFO - 2018-07-17 15:28:59.417907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:28:59,420 INFO - 2018-07-17 15:28:59.418015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:28:59,420 INFO - 2018-07-17 15:28:59.418055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:28:59,420 INFO - 2018-07-17 15:28:59.418085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:28:59,420 INFO - 2018-07-17 15:28:59.418233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:28:59,420 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-76484
2018-07-17 08:29:00,195 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:29:00,227 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:29:05,678 INFO - INFO:tensorflow:Saving checkpoints for 76485 into /output/model.ckpt.
2018-07-17 08:29:06,699 INFO - INFO:tensorflow:loss = 0.007928656, step = 76484
2018-07-17 08:30:14,142 INFO - INFO:tensorflow:global_step/sec: 1.48272
2018-07-17 08:30:14,143 INFO - INFO:tensorflow:loss = 0.007609492, step = 76584 (67.445 sec)
2018-07-17 08:31:21,044 INFO - INFO:tensorflow:global_step/sec: 1.49473
2018-07-17 08:31:21,045 INFO - INFO:tensorflow:loss = 0.007531752, step = 76684 (66.902 sec)
2018-07-17 08:32:28,034 INFO - INFO:tensorflow:global_step/sec: 1.49274
2018-07-17 08:32:28,035 INFO - INFO:tensorflow:loss = 0.007413839, step = 76784 (66.990 sec)
2018-07-17 08:33:35,025 INFO - INFO:tensorflow:global_step/sec: 1.49275
2018-07-17 08:33:35,027 INFO - INFO:tensorflow:loss = 0.0076068044, step = 76884 (66.991 sec)
2018-07-17 08:34:42,025 INFO - INFO:tensorflow:global_step/sec: 1.49254
2018-07-17 08:34:42,026 INFO - INFO:tensorflow:loss = 0.0075062816, step = 76984 (67.000 sec)
2018-07-17 08:35:48,978 INFO - INFO:tensorflow:global_step/sec: 1.49357
2018-07-17 08:35:48,980 INFO - INFO:tensorflow:loss = 0.0074743633, step = 77084 (66.954 sec)
2018-07-17 08:35:52,835 INFO - INFO:tensorflow:Saving checkpoints for 77091 into /output/model.ckpt.
2018-07-17 08:35:53,601 INFO - INFO:tensorflow:Loss for final step: 0.0077090627.
2018-07-17 08:35:53,602 INFO - Starting to evaluate.
2018-07-17 08:35:54,118 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:35:57,168 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:35:57,190 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:35:57
2018-07-17 08:35:57,739 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:35:57,739 INFO - 2018-07-17 15:35:57.735571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:35:57,740 INFO - 2018-07-17 15:35:57.735709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:35:57,740 INFO - 2018-07-17 15:35:57.735763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:35:57,740 INFO - 2018-07-17 15:35:57.735803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:35:57,740 INFO - 2018-07-17 15:35:57.735996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:35:57,740 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-77091
2018-07-17 08:35:58,487 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:35:58,518 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:39:56,858 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-15:39:56
2018-07-17 08:39:56,859 INFO - INFO:tensorflow:Saving dict for global step 77091: global_step = 77091, loss = 0.007921719
2018-07-17 08:39:56,859 INFO - {'loss': 0.007921719, 'global_step': 77091}
2018-07-17 08:39:56,859 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8e4f3dd0>]
2018-07-17 08:39:56,859 INFO - Starting a training cycle.
2018-07-17 08:39:57,455 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:40:00,066 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:40:00,067 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 08:40:00,776 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:40:00,777 INFO - 2018-07-17 15:40:00.775803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:40:00,777 INFO - 2018-07-17 15:40:00.775931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:40:00,777 INFO - 2018-07-17 15:40:00.775962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:40:00,778 INFO - 2018-07-17 15:40:00.775991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:40:00,778 INFO - 2018-07-17 15:40:00.776175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:40:00,778 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-77091
2018-07-17 08:40:01,590 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:40:01,632 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:40:07,171 INFO - INFO:tensorflow:Saving checkpoints for 77092 into /output/model.ckpt.
2018-07-17 08:40:08,078 INFO - INFO:tensorflow:loss = 0.007390046, step = 77091
2018-07-17 08:41:15,521 INFO - INFO:tensorflow:global_step/sec: 1.48273
2018-07-17 08:41:15,521 INFO - INFO:tensorflow:loss = 0.0076294774, step = 77191 (67.444 sec)
2018-07-17 08:42:22,490 INFO - INFO:tensorflow:global_step/sec: 1.49321
2018-07-17 08:42:22,491 INFO - INFO:tensorflow:loss = 0.007614246, step = 77291 (66.969 sec)
2018-07-17 08:43:29,461 INFO - INFO:tensorflow:global_step/sec: 1.49319
2018-07-17 08:43:29,462 INFO - INFO:tensorflow:loss = 0.007940156, step = 77391 (66.971 sec)
2018-07-17 08:44:36,476 INFO - INFO:tensorflow:global_step/sec: 1.4922
2018-07-17 08:44:36,476 INFO - INFO:tensorflow:loss = 0.007525078, step = 77491 (67.015 sec)
2018-07-17 08:45:43,465 INFO - INFO:tensorflow:global_step/sec: 1.49278
2018-07-17 08:45:43,467 INFO - INFO:tensorflow:loss = 0.0076255924, step = 77591 (66.990 sec)
2018-07-17 08:46:50,509 INFO - INFO:tensorflow:global_step/sec: 1.49167
2018-07-17 08:46:50,509 INFO - INFO:tensorflow:loss = 0.0071182624, step = 77691 (67.038 sec)
2018-07-17 08:46:54,370 INFO - INFO:tensorflow:Saving checkpoints for 77698 into /output/model.ckpt.
2018-07-17 08:46:55,291 INFO - INFO:tensorflow:Loss for final step: 0.0074961367.
2018-07-17 08:46:55,292 INFO - Starting to evaluate.
2018-07-17 08:46:55,844 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:46:58,627 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:46:58,651 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:46:58
2018-07-17 08:46:59,531 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:46:59,532 INFO - 2018-07-17 15:46:59.530376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:46:59,532 INFO - 2018-07-17 15:46:59.530493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:46:59,532 INFO - 2018-07-17 15:46:59.530534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:46:59,532 INFO - 2018-07-17 15:46:59.530558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:46:59,532 INFO - 2018-07-17 15:46:59.530749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:46:59,533 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-77698
2018-07-17 08:47:00,267 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:47:00,301 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:50:58,632 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-15:50:58
2018-07-17 08:50:58,633 INFO - INFO:tensorflow:Saving dict for global step 77698: global_step = 77698, loss = 0.007892949
2018-07-17 08:50:58,633 INFO - {'loss': 0.007892949, 'global_step': 77698}
2018-07-17 08:50:58,634 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8e1db890>]
2018-07-17 08:50:58,634 INFO - Starting a training cycle.
2018-07-17 08:50:59,230 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:51:01,872 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:51:01,873 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 08:51:02,420 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:51:02,422 INFO - 2018-07-17 15:51:02.419552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:51:02,422 INFO - 2018-07-17 15:51:02.419665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:51:02,422 INFO - 2018-07-17 15:51:02.419691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:51:02,422 INFO - 2018-07-17 15:51:02.419706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:51:02,422 INFO - 2018-07-17 15:51:02.419897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:51:02,423 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-77698
2018-07-17 08:51:03,238 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:51:03,268 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 08:51:08,715 INFO - INFO:tensorflow:Saving checkpoints for 77699 into /output/model.ckpt.
2018-07-17 08:51:09,730 INFO - INFO:tensorflow:loss = 0.0071454877, step = 77698
2018-07-17 08:52:17,221 INFO - INFO:tensorflow:global_step/sec: 1.48167
2018-07-17 08:52:17,222 INFO - INFO:tensorflow:loss = 0.007690554, step = 77798 (67.493 sec)
2018-07-17 08:53:24,223 INFO - INFO:tensorflow:global_step/sec: 1.4925
2018-07-17 08:53:24,223 INFO - INFO:tensorflow:loss = 0.007209904, step = 77898 (67.001 sec)
2018-07-17 08:54:31,230 INFO - INFO:tensorflow:global_step/sec: 1.49235
2018-07-17 08:54:31,231 INFO - INFO:tensorflow:loss = 0.0069917585, step = 77998 (67.008 sec)
2018-07-17 08:55:38,296 INFO - INFO:tensorflow:global_step/sec: 1.49108
2018-07-17 08:55:38,297 INFO - INFO:tensorflow:loss = 0.0071725342, step = 78098 (67.065 sec)
2018-07-17 08:56:45,334 INFO - INFO:tensorflow:global_step/sec: 1.49169
2018-07-17 08:56:45,336 INFO - INFO:tensorflow:loss = 0.00716184, step = 78198 (67.039 sec)
2018-07-17 08:57:52,410 INFO - INFO:tensorflow:global_step/sec: 1.49084
2018-07-17 08:57:52,411 INFO - INFO:tensorflow:loss = 0.0073278155, step = 78298 (67.076 sec)
2018-07-17 08:57:56,279 INFO - INFO:tensorflow:Saving checkpoints for 78305 into /output/model.ckpt.
2018-07-17 08:57:57,075 INFO - Starting to evaluate.
2018-07-17 08:57:57,075 INFO - INFO:tensorflow:Loss for final step: 0.007330706.
2018-07-17 08:57:57,597 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 08:58:00,517 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 08:58:00,540 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-15:58:00
2018-07-17 08:58:01,111 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 08:58:01,111 INFO - 2018-07-17 15:58:01.109714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 08:58:01,111 INFO - 2018-07-17 15:58:01.109831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 08:58:01,112 INFO - 2018-07-17 15:58:01.109867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 08:58:01,112 INFO - 2018-07-17 15:58:01.109894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 08:58:01,112 INFO - 2018-07-17 15:58:01.110073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 08:58:01,112 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-78305
2018-07-17 08:58:01,929 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 08:58:01,968 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:02:00,506 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:02:00
2018-07-17 09:02:00,506 INFO - INFO:tensorflow:Saving dict for global step 78305: global_step = 78305, loss = 0.007375164
2018-07-17 09:02:00,506 INFO - {'loss': 0.007375164, 'global_step': 78305}
2018-07-17 09:02:00,507 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8e1f0b50>]
2018-07-17 09:02:00,507 INFO - Starting a training cycle.
2018-07-17 09:02:01,100 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:02:03,712 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:02:03,713 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:02:04,422 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:02:04,424 INFO - 2018-07-17 16:02:04.422049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:02:04,424 INFO - 2018-07-17 16:02:04.422180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:02:04,425 INFO - 2018-07-17 16:02:04.422218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:02:04,425 INFO - 2018-07-17 16:02:04.422245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:02:04,425 INFO - 2018-07-17 16:02:04.422418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:02:04,425 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-78305
2018-07-17 09:02:05,187 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:02:05,223 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:02:10,635 INFO - INFO:tensorflow:Saving checkpoints for 78306 into /output/model.ckpt.
2018-07-17 09:02:11,495 INFO - INFO:tensorflow:loss = 0.007523109, step = 78305
2018-07-17 09:03:18,955 INFO - INFO:tensorflow:global_step/sec: 1.48234
2018-07-17 09:03:18,956 INFO - INFO:tensorflow:loss = 0.0071060723, step = 78405 (67.461 sec)
2018-07-17 09:04:25,936 INFO - INFO:tensorflow:global_step/sec: 1.49295
2018-07-17 09:04:25,938 INFO - INFO:tensorflow:loss = 0.0070152776, step = 78505 (66.982 sec)
2018-07-17 09:05:32,892 INFO - INFO:tensorflow:global_step/sec: 1.49352
2018-07-17 09:05:32,893 INFO - INFO:tensorflow:loss = 0.0072602364, step = 78605 (66.955 sec)
2018-07-17 09:06:39,925 INFO - INFO:tensorflow:global_step/sec: 1.49182
2018-07-17 09:06:39,926 INFO - INFO:tensorflow:loss = 0.0078029106, step = 78705 (67.032 sec)
2018-07-17 09:07:46,951 INFO - INFO:tensorflow:global_step/sec: 1.49195
2018-07-17 09:07:46,953 INFO - INFO:tensorflow:loss = 0.0071668606, step = 78805 (67.027 sec)
2018-07-17 09:08:54,018 INFO - INFO:tensorflow:global_step/sec: 1.49106
2018-07-17 09:08:54,019 INFO - INFO:tensorflow:loss = 0.007440306, step = 78905 (67.067 sec)
2018-07-17 09:08:57,894 INFO - INFO:tensorflow:Saving checkpoints for 78912 into /output/model.ckpt.
2018-07-17 09:08:58,684 INFO - INFO:tensorflow:Loss for final step: 0.0072862958.
2018-07-17 09:08:58,684 INFO - Starting to evaluate.
2018-07-17 09:08:59,270 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:09:02,414 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:09:02,435 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-16:09:02
2018-07-17 09:09:03,167 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:09:03,167 INFO - 2018-07-17 16:09:03.165731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:09:03,168 INFO - 2018-07-17 16:09:03.165855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:09:03,168 INFO - 2018-07-17 16:09:03.165892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:09:03,168 INFO - 2018-07-17 16:09:03.165923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:09:03,168 INFO - 2018-07-17 16:09:03.166092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:09:03,168 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-78912
2018-07-17 09:09:04,017 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:09:04,076 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:13:02,653 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:13:02
2018-07-17 09:13:02,653 INFO - INFO:tensorflow:Saving dict for global step 78912: global_step = 78912, loss = 0.013383703
2018-07-17 09:13:02,654 INFO - {'loss': 0.013383703, 'global_step': 78912}
2018-07-17 09:13:02,654 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e840d3e90>]
2018-07-17 09:13:02,654 INFO - Starting a training cycle.
2018-07-17 09:13:03,260 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:13:05,807 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:13:05,807 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:13:06,516 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:13:06,516 INFO - 2018-07-17 16:13:06.514922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:13:06,516 INFO - 2018-07-17 16:13:06.515049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:13:06,517 INFO - 2018-07-17 16:13:06.515077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:13:06,517 INFO - 2018-07-17 16:13:06.515092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:13:06,517 INFO - 2018-07-17 16:13:06.515253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:13:06,517 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-78912
2018-07-17 09:13:07,354 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:13:07,393 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:13:12,982 INFO - INFO:tensorflow:Saving checkpoints for 78913 into /output/model.ckpt.
2018-07-17 09:13:13,829 INFO - INFO:tensorflow:loss = 0.007153139, step = 78912
2018-07-17 09:14:21,309 INFO - INFO:tensorflow:global_step/sec: 1.48192
2018-07-17 09:14:21,310 INFO - INFO:tensorflow:loss = 0.0068775555, step = 79012 (67.481 sec)
2018-07-17 09:15:28,256 INFO - INFO:tensorflow:global_step/sec: 1.49371
2018-07-17 09:15:28,258 INFO - INFO:tensorflow:loss = 0.0073156273, step = 79112 (66.948 sec)
2018-07-17 09:16:35,281 INFO - INFO:tensorflow:global_step/sec: 1.49199
2018-07-17 09:16:35,282 INFO - INFO:tensorflow:loss = 0.007385223, step = 79212 (67.024 sec)
2018-07-17 09:17:42,369 INFO - INFO:tensorflow:global_step/sec: 1.49057
2018-07-17 09:17:42,372 INFO - INFO:tensorflow:loss = 0.0070408573, step = 79312 (67.090 sec)
2018-07-17 09:18:49,398 INFO - INFO:tensorflow:global_step/sec: 1.4919
2018-07-17 09:18:49,399 INFO - INFO:tensorflow:loss = 0.007215054, step = 79412 (67.027 sec)
2018-07-17 09:19:56,415 INFO - INFO:tensorflow:global_step/sec: 1.49216
2018-07-17 09:19:56,418 INFO - INFO:tensorflow:loss = 0.0067267735, step = 79512 (67.017 sec)
2018-07-17 09:20:00,267 INFO - INFO:tensorflow:Saving checkpoints for 79519 into /output/model.ckpt.
2018-07-17 09:20:01,217 INFO - Starting to evaluate.
2018-07-17 09:20:01,218 INFO - INFO:tensorflow:Loss for final step: 0.0071282242.
2018-07-17 09:20:01,863 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:20:04,319 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:20:04,340 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-16:20:04
2018-07-17 09:20:05,177 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:20:05,177 INFO - 2018-07-17 16:20:05.176465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:20:05,178 INFO - 2018-07-17 16:20:05.176588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:20:05,178 INFO - 2018-07-17 16:20:05.176624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:20:05,178 INFO - 2018-07-17 16:20:05.176681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:20:05,178 INFO - 2018-07-17 16:20:05.176853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:20:05,178 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-79519
2018-07-17 09:20:05,960 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:20:06,004 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:24:04,530 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:24:04
2018-07-17 09:24:04,530 INFO - INFO:tensorflow:Saving dict for global step 79519: global_step = 79519, loss = 0.0073401798
2018-07-17 09:24:04,531 INFO - {'loss': 0.0073401798, 'global_step': 79519}
2018-07-17 09:24:04,531 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e840dfd10>]
2018-07-17 09:24:04,531 INFO - Starting a training cycle.
2018-07-17 09:24:05,134 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:24:07,733 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:24:07,734 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:24:08,287 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:24:08,287 INFO - 2018-07-17 16:24:08.286044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:24:08,288 INFO - 2018-07-17 16:24:08.286162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:24:08,288 INFO - 2018-07-17 16:24:08.286200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:24:08,288 INFO - 2018-07-17 16:24:08.286225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:24:08,288 INFO - 2018-07-17 16:24:08.286408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:24:08,289 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-79519
2018-07-17 09:24:09,084 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:24:09,119 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:24:14,513 INFO - INFO:tensorflow:Saving checkpoints for 79520 into /output/model.ckpt.
2018-07-17 09:24:15,602 INFO - INFO:tensorflow:loss = 0.006502609, step = 79519
2018-07-17 09:25:23,097 INFO - INFO:tensorflow:global_step/sec: 1.48157
2018-07-17 09:25:23,099 INFO - INFO:tensorflow:loss = 0.006830466, step = 79619 (67.497 sec)
2018-07-17 09:26:30,040 INFO - INFO:tensorflow:global_step/sec: 1.49381
2018-07-17 09:26:30,041 INFO - INFO:tensorflow:loss = 0.0068984334, step = 79719 (66.943 sec)
2018-07-17 09:27:37,051 INFO - INFO:tensorflow:global_step/sec: 1.49229
2018-07-17 09:27:37,052 INFO - INFO:tensorflow:loss = 0.0070277876, step = 79819 (67.011 sec)
2018-07-17 09:28:44,059 INFO - INFO:tensorflow:global_step/sec: 1.49236
2018-07-17 09:28:44,060 INFO - INFO:tensorflow:loss = 0.0068585174, step = 79919 (67.008 sec)
2018-07-17 09:29:51,071 INFO - INFO:tensorflow:global_step/sec: 1.49228
2018-07-17 09:29:51,073 INFO - INFO:tensorflow:loss = 0.007186501, step = 80019 (67.011 sec)
2018-07-17 09:30:58,107 INFO - INFO:tensorflow:global_step/sec: 1.49174
2018-07-17 09:30:58,107 INFO - INFO:tensorflow:loss = 0.006809978, step = 80119 (67.036 sec)
2018-07-17 09:31:01,973 INFO - INFO:tensorflow:Saving checkpoints for 80126 into /output/model.ckpt.
2018-07-17 09:31:02,771 INFO - INFO:tensorflow:Loss for final step: 0.0072093965.
2018-07-17 09:31:02,771 INFO - Starting to evaluate.
2018-07-17 09:31:03,298 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:31:06,480 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:31:06,502 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-16:31:06
2018-07-17 09:31:07,068 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:31:07,068 INFO - 2018-07-17 16:31:07.066979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:31:07,069 INFO - 2018-07-17 16:31:07.067105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:31:07,069 INFO - 2018-07-17 16:31:07.067133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:31:07,069 INFO - 2018-07-17 16:31:07.067147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:31:07,069 INFO - 2018-07-17 16:31:07.067331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:31:07,069 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-80126
2018-07-17 09:31:07,816 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:31:07,852 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:35:06,595 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:35:06
2018-07-17 09:35:06,596 INFO - INFO:tensorflow:Saving dict for global step 80126: global_step = 80126, loss = 0.007245248
2018-07-17 09:35:06,596 INFO - {'loss': 0.007245248, 'global_step': 80126}
2018-07-17 09:35:06,596 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e9884fdd0>]
2018-07-17 09:35:06,596 INFO - Starting a training cycle.
2018-07-17 09:35:07,216 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:35:09,805 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:35:09,806 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:35:10,520 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:35:10,522 INFO - 2018-07-17 16:35:10.518985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:35:10,522 INFO - 2018-07-17 16:35:10.519120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:35:10,522 INFO - 2018-07-17 16:35:10.519158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:35:10,522 INFO - 2018-07-17 16:35:10.519182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:35:10,522 INFO - 2018-07-17 16:35:10.519411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:35:10,523 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-80126
2018-07-17 09:35:11,301 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:35:11,333 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:35:16,776 INFO - INFO:tensorflow:Saving checkpoints for 80127 into /output/model.ckpt.
2018-07-17 09:35:17,639 INFO - INFO:tensorflow:loss = 0.0068462165, step = 80126
2018-07-17 09:36:25,101 INFO - INFO:tensorflow:global_step/sec: 1.48231
2018-07-17 09:36:25,102 INFO - INFO:tensorflow:loss = 0.0070209038, step = 80226 (67.464 sec)
2018-07-17 09:37:32,044 INFO - INFO:tensorflow:global_step/sec: 1.49381
2018-07-17 09:37:32,045 INFO - INFO:tensorflow:loss = 0.0073539214, step = 80326 (66.943 sec)
2018-07-17 09:38:39,087 INFO - INFO:tensorflow:global_step/sec: 1.49156
2018-07-17 09:38:39,089 INFO - INFO:tensorflow:loss = 0.0066508288, step = 80426 (67.044 sec)
2018-07-17 09:39:46,209 INFO - INFO:tensorflow:global_step/sec: 1.48984
2018-07-17 09:39:46,210 INFO - INFO:tensorflow:loss = 0.006493721, step = 80526 (67.121 sec)
2018-07-17 09:40:53,420 INFO - INFO:tensorflow:global_step/sec: 1.48784
2018-07-17 09:40:53,421 INFO - INFO:tensorflow:loss = 0.0065123737, step = 80626 (67.210 sec)
2018-07-17 09:42:00,692 INFO - INFO:tensorflow:global_step/sec: 1.48649
2018-07-17 09:42:00,694 INFO - INFO:tensorflow:loss = 0.006484844, step = 80726 (67.273 sec)
2018-07-17 09:42:04,581 INFO - INFO:tensorflow:Saving checkpoints for 80733 into /output/model.ckpt.
2018-07-17 09:42:05,566 INFO - INFO:tensorflow:Loss for final step: 0.0065495386.
2018-07-17 09:42:05,567 INFO - Starting to evaluate.
2018-07-17 09:42:06,170 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:42:08,902 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:42:08,935 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-16:42:08
2018-07-17 09:42:09,747 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:42:09,749 INFO - 2018-07-17 16:42:09.746326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:42:09,749 INFO - 2018-07-17 16:42:09.746455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:42:09,750 INFO - 2018-07-17 16:42:09.746487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:42:09,751 INFO - 2018-07-17 16:42:09.746521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:42:09,751 INFO - 2018-07-17 16:42:09.746726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:42:09,751 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-80733
2018-07-17 09:42:10,536 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:42:10,590 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:46:10,070 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:46:10
2018-07-17 09:46:10,071 INFO - INFO:tensorflow:Saving dict for global step 80733: global_step = 80733, loss = 0.006296769
2018-07-17 09:46:10,071 INFO - {'loss': 0.006296769, 'global_step': 80733}
2018-07-17 09:46:10,071 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8eb58b10>]
2018-07-17 09:46:10,071 INFO - Starting a training cycle.
2018-07-17 09:46:10,658 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:46:13,291 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:46:13,292 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:46:13,844 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:46:13,844 INFO - 2018-07-17 16:46:13.842639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:46:13,844 INFO - 2018-07-17 16:46:13.842751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:46:13,845 INFO - 2018-07-17 16:46:13.842785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:46:13,845 INFO - 2018-07-17 16:46:13.842811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:46:13,845 INFO - 2018-07-17 16:46:13.842973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:46:13,845 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-80733
2018-07-17 09:46:14,601 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:46:14,631 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:46:20,069 INFO - INFO:tensorflow:Saving checkpoints for 80734 into /output/model.ckpt.
2018-07-17 09:46:20,950 INFO - INFO:tensorflow:loss = 0.0062677613, step = 80733
2018-07-17 09:47:28,597 INFO - INFO:tensorflow:global_step/sec: 1.47827
2018-07-17 09:47:28,597 INFO - INFO:tensorflow:loss = 0.0065974006, step = 80833 (67.647 sec)
2018-07-17 09:48:35,864 INFO - INFO:tensorflow:global_step/sec: 1.4866
2018-07-17 09:48:35,865 INFO - INFO:tensorflow:loss = 0.006233165, step = 80933 (67.268 sec)
2018-07-17 09:49:43,145 INFO - INFO:tensorflow:global_step/sec: 1.4863
2018-07-17 09:49:43,147 INFO - INFO:tensorflow:loss = 0.0063234284, step = 81033 (67.282 sec)
2018-07-17 09:50:50,418 INFO - INFO:tensorflow:global_step/sec: 1.48648
2018-07-17 09:50:50,420 INFO - INFO:tensorflow:loss = 0.006579521, step = 81133 (67.273 sec)
2018-07-17 09:51:57,721 INFO - INFO:tensorflow:global_step/sec: 1.48583
2018-07-17 09:51:57,722 INFO - INFO:tensorflow:loss = 0.0064507863, step = 81233 (67.302 sec)
2018-07-17 09:53:05,009 INFO - INFO:tensorflow:global_step/sec: 1.48614
2018-07-17 09:53:05,010 INFO - INFO:tensorflow:loss = 0.0062477365, step = 81333 (67.287 sec)
2018-07-17 09:53:08,886 INFO - INFO:tensorflow:Saving checkpoints for 81340 into /output/model.ckpt.
2018-07-17 09:53:09,884 INFO - Starting to evaluate.
2018-07-17 09:53:09,885 INFO - INFO:tensorflow:Loss for final step: 0.006130454.
2018-07-17 09:53:10,534 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:53:12,931 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:53:12,952 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-16:53:12
2018-07-17 09:53:13,799 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:53:13,799 INFO - 2018-07-17 16:53:13.798291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:53:13,800 INFO - 2018-07-17 16:53:13.798389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:53:13,800 INFO - 2018-07-17 16:53:13.798416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:53:13,800 INFO - 2018-07-17 16:53:13.798442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:53:13,800 INFO - 2018-07-17 16:53:13.798623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:53:13,800 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-81340
2018-07-17 09:53:14,730 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:53:14,776 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:57:14,459 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-16:57:14
2018-07-17 09:57:14,460 INFO - INFO:tensorflow:Saving dict for global step 81340: global_step = 81340, loss = 0.006022399
2018-07-17 09:57:14,460 INFO - {'loss': 0.006022399, 'global_step': 81340}
2018-07-17 09:57:14,460 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8f0a88d0>]
2018-07-17 09:57:14,460 INFO - Starting a training cycle.
2018-07-17 09:57:15,059 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 09:57:17,699 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 09:57:17,699 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 09:57:18,252 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 09:57:18,253 INFO - 2018-07-17 16:57:18.251931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 09:57:18,253 INFO - 2018-07-17 16:57:18.252045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 09:57:18,253 INFO - 2018-07-17 16:57:18.252068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 09:57:18,254 INFO - 2018-07-17 16:57:18.252085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 09:57:18,254 INFO - 2018-07-17 16:57:18.252228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 09:57:18,254 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-81340
2018-07-17 09:57:19,034 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 09:57:19,064 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 09:57:24,395 INFO - INFO:tensorflow:Saving checkpoints for 81341 into /output/model.ckpt.
2018-07-17 09:57:25,425 INFO - INFO:tensorflow:loss = 0.0059997523, step = 81340
2018-07-17 09:58:33,336 INFO - INFO:tensorflow:global_step/sec: 1.47248
2018-07-17 09:58:33,337 INFO - INFO:tensorflow:loss = 0.0065908637, step = 81440 (67.913 sec)
2018-07-17 09:59:40,675 INFO - INFO:tensorflow:global_step/sec: 1.48502
2018-07-17 09:59:40,676 INFO - INFO:tensorflow:loss = 0.0062334463, step = 81540 (67.339 sec)
2018-07-17 10:00:48,102 INFO - INFO:tensorflow:global_step/sec: 1.4831
2018-07-17 10:00:48,103 INFO - INFO:tensorflow:loss = 0.0061895223, step = 81640 (67.426 sec)
2018-07-17 10:01:55,661 INFO - INFO:tensorflow:global_step/sec: 1.48018
2018-07-17 10:01:55,662 INFO - INFO:tensorflow:loss = 0.0074102487, step = 81740 (67.559 sec)
2018-07-17 10:03:03,168 INFO - INFO:tensorflow:global_step/sec: 1.48133
2018-07-17 10:03:03,169 INFO - INFO:tensorflow:loss = 0.007653161, step = 81840 (67.507 sec)
2018-07-17 10:04:10,599 INFO - INFO:tensorflow:global_step/sec: 1.48299
2018-07-17 10:04:10,601 INFO - INFO:tensorflow:loss = 0.007377509, step = 81940 (67.432 sec)
2018-07-17 10:04:14,487 INFO - INFO:tensorflow:Saving checkpoints for 81947 into /output/model.ckpt.
2018-07-17 10:04:15,252 INFO - Starting to evaluate.
2018-07-17 10:04:15,252 INFO - INFO:tensorflow:Loss for final step: 0.0072456715.
2018-07-17 10:04:15,784 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:04:18,912 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:04:18,934 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:04:18
2018-07-17 10:04:19,497 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:04:19,524 INFO - 2018-07-17 17:04:19.492764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:04:19,524 INFO - 2018-07-17 17:04:19.492877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:04:19,525 INFO - 2018-07-17 17:04:19.492916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:04:19,525 INFO - 2018-07-17 17:04:19.492946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:04:19,525 INFO - 2018-07-17 17:04:19.493126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:04:19,525 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-81947
2018-07-17 10:04:20,244 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:04:20,276 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:08:19,798 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-17:08:19
2018-07-17 10:08:19,798 INFO - INFO:tensorflow:Saving dict for global step 81947: global_step = 81947, loss = 0.010784005
2018-07-17 10:08:19,799 INFO - {'loss': 0.010784005, 'global_step': 81947}
2018-07-17 10:08:19,799 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8ebe6890>]
2018-07-17 10:08:19,799 INFO - Starting a training cycle.
2018-07-17 10:08:20,404 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:08:23,026 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:08:23,026 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 10:08:23,750 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:08:23,750 INFO - 2018-07-17 17:08:23.748286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:08:23,750 INFO - 2018-07-17 17:08:23.748401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:08:23,751 INFO - 2018-07-17 17:08:23.748455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:08:23,751 INFO - 2018-07-17 17:08:23.748494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:08:23,751 INFO - 2018-07-17 17:08:23.748710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:08:23,752 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-81947
2018-07-17 10:08:24,541 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:08:24,579 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:08:30,093 INFO - INFO:tensorflow:Saving checkpoints for 81948 into /output/model.ckpt.
2018-07-17 10:08:31,000 INFO - INFO:tensorflow:loss = 0.0072006565, step = 81947
2018-07-17 10:09:38,705 INFO - INFO:tensorflow:global_step/sec: 1.477
2018-07-17 10:09:38,706 INFO - INFO:tensorflow:loss = 0.0069951853, step = 82047 (67.707 sec)
2018-07-17 10:10:45,938 INFO - INFO:tensorflow:global_step/sec: 1.48737
2018-07-17 10:10:45,939 INFO - INFO:tensorflow:loss = 0.007001099, step = 82147 (67.233 sec)
2018-07-17 10:11:53,176 INFO - INFO:tensorflow:global_step/sec: 1.48724
2018-07-17 10:11:53,178 INFO - INFO:tensorflow:loss = 0.0065255105, step = 82247 (67.239 sec)
2018-07-17 10:13:00,425 INFO - INFO:tensorflow:global_step/sec: 1.48701
2018-07-17 10:13:00,426 INFO - INFO:tensorflow:loss = 0.006365511, step = 82347 (67.248 sec)
2018-07-17 10:14:07,717 INFO - INFO:tensorflow:global_step/sec: 1.48606
2018-07-17 10:14:07,718 INFO - INFO:tensorflow:loss = 0.006390277, step = 82447 (67.292 sec)
2018-07-17 10:15:15,002 INFO - INFO:tensorflow:global_step/sec: 1.48621
2018-07-17 10:15:15,003 INFO - INFO:tensorflow:loss = 0.0066206255, step = 82547 (67.286 sec)
2018-07-17 10:15:18,870 INFO - INFO:tensorflow:Saving checkpoints for 82554 into /output/model.ckpt.
2018-07-17 10:15:19,631 INFO - Starting to evaluate.
2018-07-17 10:15:19,631 INFO - INFO:tensorflow:Loss for final step: 0.0065129786.
2018-07-17 10:15:20,129 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:15:22,916 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:15:22,948 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:15:22
2018-07-17 10:15:23,697 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:15:23,697 INFO - 2018-07-17 17:15:23.695518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:15:23,697 INFO - 2018-07-17 17:15:23.695676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:15:23,698 INFO - 2018-07-17 17:15:23.695739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:15:23,698 INFO - 2018-07-17 17:15:23.695802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:15:23,698 INFO - 2018-07-17 17:15:23.696025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:15:23,698 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-82554
2018-07-17 10:15:24,425 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:15:24,457 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:19:23,772 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-17:19:23
2018-07-17 10:19:23,772 INFO - INFO:tensorflow:Saving dict for global step 82554: global_step = 82554, loss = 0.0060554976
2018-07-17 10:19:23,772 INFO - {'loss': 0.0060554976, 'global_step': 82554}
2018-07-17 10:19:23,772 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e49389850>]
2018-07-17 10:19:23,773 INFO - Starting a training cycle.
2018-07-17 10:19:24,350 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:19:26,949 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:19:26,950 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 10:19:27,492 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:19:27,493 INFO - 2018-07-17 17:19:27.491668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:19:27,493 INFO - 2018-07-17 17:19:27.491800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:19:27,493 INFO - 2018-07-17 17:19:27.491857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:19:27,494 INFO - 2018-07-17 17:19:27.491896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:19:27,494 INFO - 2018-07-17 17:19:27.492073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:19:27,494 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-82554
2018-07-17 10:19:28,269 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:19:28,300 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:19:33,676 INFO - INFO:tensorflow:Saving checkpoints for 82555 into /output/model.ckpt.
2018-07-17 10:19:34,682 INFO - INFO:tensorflow:loss = 0.0062216595, step = 82554
2018-07-17 10:20:42,353 INFO - INFO:tensorflow:global_step/sec: 1.47771
2018-07-17 10:20:42,355 INFO - INFO:tensorflow:loss = 0.0059930123, step = 82654 (67.673 sec)
2018-07-17 10:21:49,554 INFO - INFO:tensorflow:global_step/sec: 1.48808
2018-07-17 10:21:49,556 INFO - INFO:tensorflow:loss = 0.00628374, step = 82754 (67.202 sec)
2018-07-17 10:22:56,761 INFO - INFO:tensorflow:global_step/sec: 1.48795
2018-07-17 10:22:56,762 INFO - INFO:tensorflow:loss = 0.0063495636, step = 82854 (67.205 sec)
2018-07-17 10:24:04,037 INFO - INFO:tensorflow:global_step/sec: 1.4864
2018-07-17 10:24:04,038 INFO - INFO:tensorflow:loss = 0.006214886, step = 82954 (67.277 sec)
2018-07-17 10:25:11,285 INFO - INFO:tensorflow:global_step/sec: 1.48705
2018-07-17 10:25:11,286 INFO - INFO:tensorflow:loss = 0.006104918, step = 83054 (67.248 sec)
2018-07-17 10:26:18,541 INFO - INFO:tensorflow:global_step/sec: 1.48686
2018-07-17 10:26:18,541 INFO - INFO:tensorflow:loss = 0.006080169, step = 83154 (67.256 sec)
2018-07-17 10:26:22,434 INFO - INFO:tensorflow:Saving checkpoints for 83161 into /output/model.ckpt.
2018-07-17 10:26:23,203 INFO - INFO:tensorflow:Loss for final step: 0.0061585503.
2018-07-17 10:26:23,203 INFO - Starting to evaluate.
2018-07-17 10:26:23,906 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:26:26,711 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:26:26,732 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:26:26
2018-07-17 10:26:27,282 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:26:27,282 INFO - 2018-07-17 17:26:27.281194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:26:27,283 INFO - 2018-07-17 17:26:27.281322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:26:27,283 INFO - 2018-07-17 17:26:27.281361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:26:27,283 INFO - 2018-07-17 17:26:27.281392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:26:27,283 INFO - 2018-07-17 17:26:27.281547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:26:27,283 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-83161
2018-07-17 10:26:28,022 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:26:28,083 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:30:27,767 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-17:30:27
2018-07-17 10:30:27,767 INFO - INFO:tensorflow:Saving dict for global step 83161: global_step = 83161, loss = 0.0056072534
2018-07-17 10:30:27,767 INFO - {'loss': 0.0056072534, 'global_step': 83161}
2018-07-17 10:30:27,768 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e49f5cb10>]
2018-07-17 10:30:27,768 INFO - Starting a training cycle.
2018-07-17 10:30:28,359 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:30:30,988 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:30:30,989 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 10:30:31,530 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:30:31,532 INFO - 2018-07-17 17:30:31.529346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:30:31,532 INFO - 2018-07-17 17:30:31.529465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:30:31,533 INFO - 2018-07-17 17:30:31.529503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:30:31,533 INFO - 2018-07-17 17:30:31.529561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:30:31,533 INFO - 2018-07-17 17:30:31.529763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:30:31,533 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-83161
2018-07-17 10:30:32,296 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:30:32,329 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:30:37,927 INFO - INFO:tensorflow:Saving checkpoints for 83162 into /output/model.ckpt.
2018-07-17 10:30:38,789 INFO - INFO:tensorflow:loss = 0.0057883337, step = 83161
2018-07-17 10:31:46,374 INFO - INFO:tensorflow:global_step/sec: 1.47962
2018-07-17 10:31:46,376 INFO - INFO:tensorflow:loss = 0.0056669293, step = 83261 (67.587 sec)
2018-07-17 10:32:53,639 INFO - INFO:tensorflow:global_step/sec: 1.48666
2018-07-17 10:32:53,643 INFO - INFO:tensorflow:loss = 0.0060666003, step = 83361 (67.265 sec)
2018-07-17 10:34:00,914 INFO - INFO:tensorflow:global_step/sec: 1.48646
2018-07-17 10:34:00,914 INFO - INFO:tensorflow:loss = 0.006022655, step = 83461 (67.273 sec)
2018-07-17 10:35:08,198 INFO - INFO:tensorflow:global_step/sec: 1.48619
2018-07-17 10:35:08,199 INFO - INFO:tensorflow:loss = 0.0059570177, step = 83561 (67.286 sec)
2018-07-17 10:36:15,524 INFO - INFO:tensorflow:global_step/sec: 1.48533
2018-07-17 10:36:15,525 INFO - INFO:tensorflow:loss = 0.005850059, step = 83661 (67.326 sec)
2018-07-17 10:37:22,857 INFO - INFO:tensorflow:global_step/sec: 1.48515
2018-07-17 10:37:22,858 INFO - INFO:tensorflow:loss = 0.005998697, step = 83761 (67.333 sec)
2018-07-17 10:37:26,737 INFO - INFO:tensorflow:Saving checkpoints for 83768 into /output/model.ckpt.
2018-07-17 10:37:27,493 INFO - INFO:tensorflow:Loss for final step: 0.0061423182.
2018-07-17 10:37:27,493 INFO - Starting to evaluate.
2018-07-17 10:37:27,997 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:37:30,821 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:37:30,856 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:37:30
2018-07-17 10:37:31,587 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:37:31,588 INFO - 2018-07-17 17:37:31.586637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:37:31,588 INFO - 2018-07-17 17:37:31.586749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:37:31,588 INFO - 2018-07-17 17:37:31.586774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:37:31,588 INFO - 2018-07-17 17:37:31.586789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:37:31,589 INFO - 2018-07-17 17:37:31.586955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:37:31,589 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-83768
2018-07-17 10:37:32,418 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:37:32,453 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:41:32,149 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-17:41:32
2018-07-17 10:41:32,149 INFO - INFO:tensorflow:Saving dict for global step 83768: global_step = 83768, loss = 0.0051460746
2018-07-17 10:41:32,150 INFO - {'loss': 0.0051460746, 'global_step': 83768}
2018-07-17 10:41:32,150 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e48c75610>]
2018-07-17 10:41:32,150 INFO - Starting a training cycle.
2018-07-17 10:41:32,764 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:41:35,384 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:41:35,384 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 10:41:36,097 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:41:36,097 INFO - 2018-07-17 17:41:36.095845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:41:36,097 INFO - 2018-07-17 17:41:36.095957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:41:36,098 INFO - 2018-07-17 17:41:36.095984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:41:36,098 INFO - 2018-07-17 17:41:36.095999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:41:36,098 INFO - 2018-07-17 17:41:36.096170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:41:36,098 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-83768
2018-07-17 10:41:36,904 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:41:36,947 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:41:42,373 INFO - INFO:tensorflow:Saving checkpoints for 83769 into /output/model.ckpt.
2018-07-17 10:41:43,365 INFO - INFO:tensorflow:loss = 0.00584599, step = 83768
2018-07-17 10:42:51,191 INFO - INFO:tensorflow:global_step/sec: 1.47437
2018-07-17 10:42:51,192 INFO - INFO:tensorflow:loss = 0.005824735, step = 83868 (67.826 sec)
2018-07-17 10:43:58,499 INFO - INFO:tensorflow:global_step/sec: 1.48569
2018-07-17 10:43:58,500 INFO - INFO:tensorflow:loss = 0.0060633263, step = 83968 (67.309 sec)
2018-07-17 10:45:05,845 INFO - INFO:tensorflow:global_step/sec: 1.48488
2018-07-17 10:45:05,846 INFO - INFO:tensorflow:loss = 0.006159378, step = 84068 (67.346 sec)
2018-07-17 10:46:13,188 INFO - INFO:tensorflow:global_step/sec: 1.48492
2018-07-17 10:46:13,190 INFO - INFO:tensorflow:loss = 0.005888385, step = 84168 (67.344 sec)
2018-07-17 10:47:20,524 INFO - INFO:tensorflow:global_step/sec: 1.48508
2018-07-17 10:47:20,525 INFO - INFO:tensorflow:loss = 0.0058895806, step = 84268 (67.335 sec)
2018-07-17 10:48:27,849 INFO - INFO:tensorflow:global_step/sec: 1.48534
2018-07-17 10:48:27,851 INFO - INFO:tensorflow:loss = 0.0057843686, step = 84368 (67.326 sec)
2018-07-17 10:48:31,745 INFO - INFO:tensorflow:Saving checkpoints for 84375 into /output/model.ckpt.
2018-07-17 10:48:32,726 INFO - Starting to evaluate.
2018-07-17 10:48:32,727 INFO - INFO:tensorflow:Loss for final step: 0.0059565576.
2018-07-17 10:48:33,391 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:48:35,949 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:48:35,970 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:48:35
2018-07-17 10:48:36,844 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:48:36,845 INFO - 2018-07-17 17:48:36.842673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:48:36,845 INFO - 2018-07-17 17:48:36.842869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:48:36,846 INFO - 2018-07-17 17:48:36.842913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:48:36,846 INFO - 2018-07-17 17:48:36.842980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:48:36,846 INFO - 2018-07-17 17:48:36.843285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:48:36,846 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-84375
2018-07-17 10:48:37,750 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:48:37,793 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:52:37,365 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-17:52:37
2018-07-17 10:52:37,365 INFO - INFO:tensorflow:Saving dict for global step 84375: global_step = 84375, loss = 0.0051627597
2018-07-17 10:52:37,365 INFO - {'loss': 0.0051627597, 'global_step': 84375}
2018-07-17 10:52:37,365 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8eebb310>]
2018-07-17 10:52:37,366 INFO - Starting a training cycle.
2018-07-17 10:52:37,954 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:52:40,540 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:52:40,541 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 10:52:41,085 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:52:41,085 INFO - 2018-07-17 17:52:41.084119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:52:41,086 INFO - 2018-07-17 17:52:41.084239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:52:41,086 INFO - 2018-07-17 17:52:41.084277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:52:41,086 INFO - 2018-07-17 17:52:41.084326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:52:41,086 INFO - 2018-07-17 17:52:41.084520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:52:41,086 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-84375
2018-07-17 10:52:41,860 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:52:41,888 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 10:52:47,240 INFO - INFO:tensorflow:Saving checkpoints for 84376 into /output/model.ckpt.
2018-07-17 10:52:48,262 INFO - INFO:tensorflow:loss = 0.005558675, step = 84375
2018-07-17 10:53:56,043 INFO - INFO:tensorflow:global_step/sec: 1.47533
2018-07-17 10:53:56,043 INFO - INFO:tensorflow:loss = 0.0059022466, step = 84475 (67.781 sec)
2018-07-17 10:55:03,272 INFO - INFO:tensorflow:global_step/sec: 1.48745
2018-07-17 10:55:03,275 INFO - INFO:tensorflow:loss = 0.0060315756, step = 84575 (67.230 sec)
2018-07-17 10:56:10,616 INFO - INFO:tensorflow:global_step/sec: 1.48489
2018-07-17 10:56:10,618 INFO - INFO:tensorflow:loss = 0.0058761006, step = 84675 (67.345 sec)
2018-07-17 10:57:17,952 INFO - INFO:tensorflow:global_step/sec: 1.4851
2018-07-17 10:57:17,953 INFO - INFO:tensorflow:loss = 0.0061236224, step = 84775 (67.335 sec)
2018-07-17 10:58:25,307 INFO - INFO:tensorflow:global_step/sec: 1.48467
2018-07-17 10:58:25,308 INFO - INFO:tensorflow:loss = 0.0058026845, step = 84875 (67.354 sec)
2018-07-17 10:59:32,673 INFO - INFO:tensorflow:global_step/sec: 1.48443
2018-07-17 10:59:32,675 INFO - INFO:tensorflow:loss = 0.00605274, step = 84975 (67.367 sec)
2018-07-17 10:59:36,547 INFO - INFO:tensorflow:Saving checkpoints for 84982 into /output/model.ckpt.
2018-07-17 10:59:37,330 INFO - Starting to evaluate.
2018-07-17 10:59:37,332 INFO - INFO:tensorflow:Loss for final step: 0.0060409005.
2018-07-17 10:59:37,828 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 10:59:40,539 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 10:59:40,562 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-17:59:40
2018-07-17 10:59:41,171 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 10:59:41,172 INFO - 2018-07-17 17:59:41.170542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 10:59:41,172 INFO - 2018-07-17 17:59:41.170652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 10:59:41,172 INFO - 2018-07-17 17:59:41.170672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 10:59:41,172 INFO - 2018-07-17 17:59:41.170686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 10:59:41,173 INFO - 2018-07-17 17:59:41.170850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 10:59:41,173 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-84982
2018-07-17 10:59:42,075 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 10:59:42,109 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:03:41,663 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-18:03:41
2018-07-17 11:03:41,664 INFO - INFO:tensorflow:Saving dict for global step 84982: global_step = 84982, loss = 0.0057707145
2018-07-17 11:03:41,664 INFO - {'loss': 0.0057707145, 'global_step': 84982}
2018-07-17 11:03:41,664 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e49389990>]
2018-07-17 11:03:41,665 INFO - Starting a training cycle.
2018-07-17 11:03:42,274 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:03:44,851 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:03:44,852 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 11:03:45,554 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:03:45,555 INFO - 2018-07-17 18:03:45.554011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:03:45,555 INFO - 2018-07-17 18:03:45.554123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:03:45,555 INFO - 2018-07-17 18:03:45.554150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:03:45,556 INFO - 2018-07-17 18:03:45.554175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:03:45,556 INFO - 2018-07-17 18:03:45.554352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:03:45,556 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-84982
2018-07-17 11:03:46,319 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:03:46,350 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:03:51,754 INFO - INFO:tensorflow:Saving checkpoints for 84983 into /output/model.ckpt.
2018-07-17 11:03:52,589 INFO - INFO:tensorflow:loss = 0.0058397213, step = 84982
2018-07-17 11:05:00,295 INFO - INFO:tensorflow:global_step/sec: 1.47696
2018-07-17 11:05:00,297 INFO - INFO:tensorflow:loss = 0.0055904565, step = 85082 (67.708 sec)
2018-07-17 11:06:07,512 INFO - INFO:tensorflow:global_step/sec: 1.48772
2018-07-17 11:06:07,513 INFO - INFO:tensorflow:loss = 0.0054691285, step = 85182 (67.216 sec)
2018-07-17 11:07:14,822 INFO - INFO:tensorflow:global_step/sec: 1.48566
2018-07-17 11:07:14,823 INFO - INFO:tensorflow:loss = 0.0056434986, step = 85282 (67.310 sec)
2018-07-17 11:08:22,128 INFO - INFO:tensorflow:global_step/sec: 1.48576
2018-07-17 11:08:22,129 INFO - INFO:tensorflow:loss = 0.0060298406, step = 85382 (67.306 sec)
2018-07-17 11:09:29,496 INFO - INFO:tensorflow:global_step/sec: 1.48438
2018-07-17 11:09:29,497 INFO - INFO:tensorflow:loss = 0.005841311, step = 85482 (67.368 sec)
2018-07-17 11:10:36,813 INFO - INFO:tensorflow:global_step/sec: 1.4855
2018-07-17 11:10:36,815 INFO - INFO:tensorflow:loss = 0.0055320943, step = 85582 (67.318 sec)
2018-07-17 11:10:40,703 INFO - INFO:tensorflow:Saving checkpoints for 85589 into /output/model.ckpt.
2018-07-17 11:10:41,493 INFO - INFO:tensorflow:Loss for final step: 0.005730892.
2018-07-17 11:10:41,494 INFO - Starting to evaluate.
2018-07-17 11:10:42,119 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:10:45,242 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:10:45,264 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-18:10:45
2018-07-17 11:10:45,842 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:10:45,842 INFO - 2018-07-17 18:10:45.841072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:10:45,842 INFO - 2018-07-17 18:10:45.841206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:10:45,842 INFO - 2018-07-17 18:10:45.841245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:10:45,843 INFO - 2018-07-17 18:10:45.841273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:10:45,843 INFO - 2018-07-17 18:10:45.841468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:10:45,843 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-85589
2018-07-17 11:10:46,644 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:10:46,686 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:14:46,359 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-18:14:46
2018-07-17 11:14:46,360 INFO - INFO:tensorflow:Saving dict for global step 85589: global_step = 85589, loss = 0.004698152
2018-07-17 11:14:46,360 INFO - {'loss': 0.004698152, 'global_step': 85589}
2018-07-17 11:14:46,360 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8eab68d0>]
2018-07-17 11:14:46,361 INFO - Starting a training cycle.
2018-07-17 11:14:46,951 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:14:49,492 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:14:49,493 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 11:14:50,188 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:14:50,188 INFO - 2018-07-17 18:14:50.187738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:14:50,189 INFO - 2018-07-17 18:14:50.187849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:14:50,189 INFO - 2018-07-17 18:14:50.187885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:14:50,189 INFO - 2018-07-17 18:14:50.187930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:14:50,189 INFO - 2018-07-17 18:14:50.188140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:14:50,190 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-85589
2018-07-17 11:14:50,950 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:14:50,981 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:14:56,479 INFO - INFO:tensorflow:Saving checkpoints for 85590 into /output/model.ckpt.
2018-07-17 11:14:57,443 INFO - INFO:tensorflow:loss = 0.0052493243, step = 85589
2018-07-17 11:16:05,058 INFO - INFO:tensorflow:global_step/sec: 1.47894
2018-07-17 11:16:05,059 INFO - INFO:tensorflow:loss = 0.0052481014, step = 85689 (67.617 sec)
2018-07-17 11:17:12,247 INFO - INFO:tensorflow:global_step/sec: 1.48834
2018-07-17 11:17:12,247 INFO - INFO:tensorflow:loss = 0.0053552394, step = 85789 (67.189 sec)
2018-07-17 11:18:19,502 INFO - INFO:tensorflow:global_step/sec: 1.48687
2018-07-17 11:18:19,503 INFO - INFO:tensorflow:loss = 0.005682001, step = 85889 (67.255 sec)
2018-07-17 11:19:26,818 INFO - INFO:tensorflow:global_step/sec: 1.48553
2018-07-17 11:19:26,819 INFO - INFO:tensorflow:loss = 0.005572074, step = 85989 (67.317 sec)
2018-07-17 11:20:34,097 INFO - INFO:tensorflow:global_step/sec: 1.48635
2018-07-17 11:20:34,098 INFO - INFO:tensorflow:loss = 0.005556782, step = 86089 (67.278 sec)
2018-07-17 11:21:41,430 INFO - INFO:tensorflow:global_step/sec: 1.48517
2018-07-17 11:21:41,431 INFO - INFO:tensorflow:loss = 0.005485625, step = 86189 (67.334 sec)
2018-07-17 11:21:45,308 INFO - INFO:tensorflow:Saving checkpoints for 86196 into /output/model.ckpt.
2018-07-17 11:21:46,247 INFO - INFO:tensorflow:Loss for final step: 0.005687912.
2018-07-17 11:21:46,247 INFO - Starting to evaluate.
2018-07-17 11:21:46,796 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:21:49,302 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:21:49,323 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-18:21:49
2018-07-17 11:21:50,163 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:21:50,163 INFO - 2018-07-17 18:21:50.161789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:21:50,163 INFO - 2018-07-17 18:21:50.161901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:21:50,163 INFO - 2018-07-17 18:21:50.161938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:21:50,164 INFO - 2018-07-17 18:21:50.161970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:21:50,164 INFO - 2018-07-17 18:21:50.162152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:21:50,164 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-86196
2018-07-17 11:21:51,061 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:21:51,120 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:25:50,550 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-18:25:50
2018-07-17 11:25:50,551 INFO - INFO:tensorflow:Saving dict for global step 86196: global_step = 86196, loss = 0.004532863
2018-07-17 11:25:50,551 INFO - {'loss': 0.004532863, 'global_step': 86196}
2018-07-17 11:25:50,551 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e8ef00cd0>]
2018-07-17 11:25:50,552 INFO - Starting a training cycle.
2018-07-17 11:25:51,149 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:25:53,754 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:25:53,755 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 11:25:54,299 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:25:54,300 INFO - 2018-07-17 18:25:54.298277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:25:54,300 INFO - 2018-07-17 18:25:54.298404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:25:54,300 INFO - 2018-07-17 18:25:54.298437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:25:54,300 INFO - 2018-07-17 18:25:54.298469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:25:54,301 INFO - 2018-07-17 18:25:54.298629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:25:54,301 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-86196
2018-07-17 11:25:55,057 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:25:55,087 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:26:00,393 INFO - INFO:tensorflow:Saving checkpoints for 86197 into /output/model.ckpt.
2018-07-17 11:26:01,472 INFO - INFO:tensorflow:loss = 0.0051090564, step = 86196
2018-07-17 11:27:09,114 INFO - INFO:tensorflow:global_step/sec: 1.47835
2018-07-17 11:27:09,115 INFO - INFO:tensorflow:loss = 0.0051454473, step = 86296 (67.644 sec)
2018-07-17 11:28:16,374 INFO - INFO:tensorflow:global_step/sec: 1.48675
2018-07-17 11:28:16,377 INFO - INFO:tensorflow:loss = 0.005344742, step = 86396 (67.261 sec)
2018-07-17 11:29:23,635 INFO - INFO:tensorflow:global_step/sec: 1.48677
2018-07-17 11:29:23,635 INFO - INFO:tensorflow:loss = 0.0054508774, step = 86496 (67.259 sec)
2018-07-17 11:30:30,962 INFO - INFO:tensorflow:global_step/sec: 1.48528
2018-07-17 11:30:30,964 INFO - INFO:tensorflow:loss = 0.0053538396, step = 86596 (67.328 sec)
2018-07-17 11:31:38,321 INFO - INFO:tensorflow:global_step/sec: 1.48458
2018-07-17 11:31:38,323 INFO - INFO:tensorflow:loss = 0.0052725454, step = 86696 (67.359 sec)
2018-07-17 11:32:45,665 INFO - INFO:tensorflow:global_step/sec: 1.48492
2018-07-17 11:32:45,666 INFO - INFO:tensorflow:loss = 0.0052362094, step = 86796 (67.343 sec)
2018-07-17 11:32:49,557 INFO - INFO:tensorflow:Saving checkpoints for 86803 into /output/model.ckpt.
2018-07-17 11:32:50,329 INFO - INFO:tensorflow:Loss for final step: 0.0054322165.
2018-07-17 11:32:50,330 INFO - Starting to evaluate.
2018-07-17 11:32:50,827 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:32:53,855 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:32:53,881 INFO - INFO:tensorflow:Starting evaluation at 2018-07-17-18:32:53
2018-07-17 11:32:54,514 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:32:54,514 INFO - 2018-07-17 18:32:54.513008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:32:54,514 INFO - 2018-07-17 18:32:54.513111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:32:54,514 INFO - 2018-07-17 18:32:54.513130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:32:54,515 INFO - 2018-07-17 18:32:54.513145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:32:54,515 INFO - 2018-07-17 18:32:54.513328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-17 11:32:54,515 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-86803
2018-07-17 11:32:55,440 INFO - INFO:tensorflow:Running local_init_op.
2018-07-17 11:32:55,481 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-17 11:36:55,162 INFO - INFO:tensorflow:Finished evaluation at 2018-07-17-18:36:55
2018-07-17 11:36:55,163 INFO - INFO:tensorflow:Saving dict for global step 86803: global_step = 86803, loss = 0.0044263764
2018-07-17 11:36:55,163 INFO - {'loss': 0.0044263764, 'global_step': 86803}
2018-07-17 11:36:55,163 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f3e48e26c50>]
2018-07-17 11:36:55,163 INFO - Starting a training cycle.
2018-07-17 11:36:55,763 INFO - INFO:tensorflow:Calling model_fn.
2018-07-17 11:36:58,339 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-17 11:36:58,340 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-17 11:36:59,043 INFO - INFO:tensorflow:Graph was finalized.
2018-07-17 11:36:59,043 INFO - 2018-07-17 18:36:59.042574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-17 11:36:59,044 INFO - 2018-07-17 18:36:59.042693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-17 11:36:59,044 INFO - 2018-07-17 18:36:59.042723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-17 11:36:59,044 INFO - 2018-07-17 18:36:59.042750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-17 11:36:59,044 INFO - 2018-07-17 18:36:59.042918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)