-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtf-loss-ds1.txt
5665 lines (5609 loc) · 558 KB
/
tf-loss-ds1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2018-07-16 02:58:39,552 INFO - Preparing to run TaskInstance <TaskInstance: mrozh/projects/dan-tf/56 (id: X3eukSbiaMfEHtYfygCtvg)
2018-07-16 02:58:39,572 INFO - Starting attempt 1
2018-07-16 02:58:39,586 INFO - Downloading and setting up data sources
2018-07-16 02:58:39,594 INFO - Downloading and mounting dataset. ETA: 12 seconds
2018-07-16 02:58:47,570 INFO - Pulling Docker image: floydhub/tensorflow:1.7.0-gpu.cuda9cudnn7-py2_aws.30
2018-07-16 02:58:48,819 INFO - Starting container...
2018-07-16 02:58:49,061 INFO -
################################################################################
2018-07-16 02:58:49,061 INFO - Run Output:
2018-07-16 02:58:49,194 INFO - Starting services.
2018-07-16 02:59:14,166 INFO - INFO:tensorflow:Using config: {'_save_checkpoints_secs': 1000000000.0, '_session_config': allow_soft_placement: true
2018-07-16 02:59:14,166 INFO - , '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f0631d33710>, '_evaluation_master': '', '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/output', '_save_summary_steps': 100}
2018-07-16 02:59:14,167 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0631d215d0>]
2018-07-16 02:59:14,167 INFO - Starting a training cycle.
2018-07-16 02:59:15,094 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 02:59:19,085 INFO - WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
2018-07-16 02:59:19,086 INFO - Instructions for updating:
2018-07-16 02:59:19,086 INFO - Use the retry module or similar alternatives.
2018-07-16 02:59:25,281 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 02:59:25,281 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 02:59:26,773 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 02:59:26,834 INFO - 2018-07-16 09:59:26.833189: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-16 02:59:26,835 INFO - 2018-07-16 09:59:26.833578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
2018-07-16 02:59:26,835 INFO - name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8755
2018-07-16 02:59:26,835 INFO - pciBusID: 0000:00:1e.0
2018-07-16 02:59:26,836 INFO - totalMemory: 11.17GiB freeMemory: 10.99GiB
2018-07-16 02:59:26,836 INFO - 2018-07-16 09:59:26.833622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 02:59:27,123 INFO - 2018-07-16 09:59:27.121265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 02:59:27,123 INFO - 2018-07-16 09:59:27.121388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 02:59:27,124 INFO - 2018-07-16 09:59:27.121418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 02:59:27,124 INFO - 2018-07-16 09:59:27.121763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 02:59:29,393 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 02:59:29,430 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 02:59:38,854 INFO - 2018-07-16 09:59:38.852929: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x79f62c0
2018-07-16 02:59:47,468 INFO - INFO:tensorflow:Saving checkpoints for 1 into /output/model.ckpt.
2018-07-16 02:59:48,563 INFO - INFO:tensorflow:loss = 0.12410064, step = 0
2018-07-16 03:00:27,083 INFO - INFO:tensorflow:global_step/sec: 2.59602
2018-07-16 03:00:27,084 INFO - INFO:tensorflow:loss = 0.047968358, step = 100 (38.521 sec)
2018-07-16 03:01:05,393 INFO - INFO:tensorflow:global_step/sec: 2.61025
2018-07-16 03:01:05,394 INFO - INFO:tensorflow:loss = 0.047405638, step = 200 (38.311 sec)
2018-07-16 03:01:43,928 INFO - INFO:tensorflow:global_step/sec: 2.59508
2018-07-16 03:01:43,929 INFO - INFO:tensorflow:loss = 0.0348837, step = 300 (38.535 sec)
2018-07-16 03:02:22,522 INFO - INFO:tensorflow:global_step/sec: 2.59108
2018-07-16 03:02:22,524 INFO - INFO:tensorflow:loss = 0.032921463, step = 400 (38.594 sec)
2018-07-16 03:03:01,169 INFO - INFO:tensorflow:global_step/sec: 2.58751
2018-07-16 03:03:01,169 INFO - INFO:tensorflow:loss = 0.028639156, step = 500 (38.647 sec)
2018-07-16 03:03:39,810 INFO - INFO:tensorflow:global_step/sec: 2.58793
2018-07-16 03:03:39,810 INFO - INFO:tensorflow:loss = 0.040835045, step = 600 (38.641 sec)
2018-07-16 03:03:44,320 INFO - INFO:tensorflow:Saving checkpoints for 607 into /output/model.ckpt.
2018-07-16 03:03:45,205 INFO - Starting to evaluate.
2018-07-16 03:03:45,206 INFO - INFO:tensorflow:Loss for final step: 0.03732633.
2018-07-16 03:03:45,756 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:03:48,565 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:03:48,588 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:03:48
2018-07-16 03:03:49,488 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:03:49,489 INFO - 2018-07-16 10:03:49.487626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:03:49,490 INFO - 2018-07-16 10:03:49.487811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:03:49,490 INFO - 2018-07-16 10:03:49.487857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:03:49,490 INFO - 2018-07-16 10:03:49.487884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:03:49,490 INFO - 2018-07-16 10:03:49.488133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:03:49,490 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-607
2018-07-16 03:03:50,306 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:03:50,340 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:05:07,562 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:05:07
2018-07-16 03:05:07,562 INFO - INFO:tensorflow:Saving dict for global step 607: global_step = 607, loss = 0.07803877
2018-07-16 03:05:08,404 INFO - {'loss': 0.07803877, 'global_step': 607}
2018-07-16 03:05:08,404 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f05dc248c90>]
2018-07-16 03:05:08,404 INFO - Starting a training cycle.
2018-07-16 03:05:08,956 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:05:11,750 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:05:11,751 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:05:12,494 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:05:12,495 INFO - 2018-07-16 10:05:12.493404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:05:12,495 INFO - 2018-07-16 10:05:12.493532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:05:12,495 INFO - 2018-07-16 10:05:12.493573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:05:12,496 INFO - 2018-07-16 10:05:12.493605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:05:12,496 INFO - 2018-07-16 10:05:12.493796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:05:12,496 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-607
2018-07-16 03:05:13,284 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:05:13,320 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:05:18,480 INFO - INFO:tensorflow:Saving checkpoints for 608 into /output/model.ckpt.
2018-07-16 03:05:19,290 INFO - INFO:tensorflow:loss = 0.03504315, step = 607
2018-07-16 03:05:58,433 INFO - INFO:tensorflow:global_step/sec: 2.55466
2018-07-16 03:05:58,434 INFO - INFO:tensorflow:loss = 0.030349553, step = 707 (39.146 sec)
2018-07-16 03:06:37,060 INFO - INFO:tensorflow:global_step/sec: 2.58883
2018-07-16 03:06:37,062 INFO - INFO:tensorflow:loss = 0.03338877, step = 807 (38.627 sec)
2018-07-16 03:07:15,669 INFO - INFO:tensorflow:global_step/sec: 2.59011
2018-07-16 03:07:15,670 INFO - INFO:tensorflow:loss = 0.030412097, step = 907 (38.608 sec)
2018-07-16 03:07:54,289 INFO - INFO:tensorflow:global_step/sec: 2.5893
2018-07-16 03:07:54,291 INFO - INFO:tensorflow:loss = 0.028493458, step = 1007 (38.621 sec)
2018-07-16 03:08:33,000 INFO - INFO:tensorflow:global_step/sec: 2.58327
2018-07-16 03:08:33,001 INFO - INFO:tensorflow:loss = 0.027930427, step = 1107 (38.710 sec)
2018-07-16 03:09:11,621 INFO - INFO:tensorflow:global_step/sec: 2.58923
2018-07-16 03:09:11,622 INFO - INFO:tensorflow:loss = 0.036594614, step = 1207 (38.621 sec)
2018-07-16 03:09:13,834 INFO - INFO:tensorflow:Saving checkpoints for 1214 into /output/model.ckpt.
2018-07-16 03:09:14,549 INFO - Starting to evaluate.
2018-07-16 03:09:14,550 INFO - INFO:tensorflow:Loss for final step: 0.03580283.
2018-07-16 03:09:15,056 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:09:18,404 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:09:18,427 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:09:18
2018-07-16 03:09:19,190 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:09:19,191 INFO - 2018-07-16 10:09:19.189989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:09:19,192 INFO - 2018-07-16 10:09:19.190132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:09:19,192 INFO - 2018-07-16 10:09:19.190165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:09:19,192 INFO - 2018-07-16 10:09:19.190181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:09:19,192 INFO - 2018-07-16 10:09:19.190376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:09:19,192 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1214
2018-07-16 03:09:19,938 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:09:19,972 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:10:37,717 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:10:37
2018-07-16 03:10:37,718 INFO - INFO:tensorflow:Saving dict for global step 1214: global_step = 1214, loss = 0.042168878
2018-07-16 03:10:37,718 INFO - {'loss': 0.042168878, 'global_step': 1214}
2018-07-16 03:10:37,718 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0631bc6210>]
2018-07-16 03:10:37,719 INFO - Starting a training cycle.
2018-07-16 03:10:38,319 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:10:41,139 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:10:41,140 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:10:41,736 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:10:41,737 INFO - 2018-07-16 10:10:41.735006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:10:41,737 INFO - 2018-07-16 10:10:41.735151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:10:41,737 INFO - 2018-07-16 10:10:41.735217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:10:41,738 INFO - 2018-07-16 10:10:41.735266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:10:41,738 INFO - 2018-07-16 10:10:41.735479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:10:41,738 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1214
2018-07-16 03:10:42,518 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:10:42,554 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:10:47,707 INFO - INFO:tensorflow:Saving checkpoints for 1215 into /output/model.ckpt.
2018-07-16 03:10:48,555 INFO - INFO:tensorflow:loss = 0.032089744, step = 1214
2018-07-16 03:11:27,532 INFO - INFO:tensorflow:global_step/sec: 2.56556
2018-07-16 03:11:27,533 INFO - INFO:tensorflow:loss = 0.026815586, step = 1314 (38.980 sec)
2018-07-16 03:12:06,197 INFO - INFO:tensorflow:global_step/sec: 2.58629
2018-07-16 03:12:06,198 INFO - INFO:tensorflow:loss = 0.04090564, step = 1414 (38.665 sec)
2018-07-16 03:12:44,821 INFO - INFO:tensorflow:global_step/sec: 2.58913
2018-07-16 03:12:44,822 INFO - INFO:tensorflow:loss = 0.03357749, step = 1514 (38.623 sec)
2018-07-16 03:13:23,505 INFO - INFO:tensorflow:global_step/sec: 2.58507
2018-07-16 03:13:23,506 INFO - INFO:tensorflow:loss = 0.028702974, step = 1614 (38.684 sec)
2018-07-16 03:14:02,157 INFO - INFO:tensorflow:global_step/sec: 2.58718
2018-07-16 03:14:02,158 INFO - INFO:tensorflow:loss = 0.030651528, step = 1714 (38.652 sec)
2018-07-16 03:14:40,817 INFO - INFO:tensorflow:global_step/sec: 2.58663
2018-07-16 03:14:40,818 INFO - INFO:tensorflow:loss = 0.024815004, step = 1814 (38.660 sec)
2018-07-16 03:14:43,048 INFO - INFO:tensorflow:Saving checkpoints for 1821 into /output/model.ckpt.
2018-07-16 03:14:44,047 INFO - Starting to evaluate.
2018-07-16 03:14:44,048 INFO - INFO:tensorflow:Loss for final step: 0.028095068.
2018-07-16 03:14:44,577 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:14:47,342 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:14:47,365 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:14:47
2018-07-16 03:14:48,275 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:14:48,276 INFO - 2018-07-16 10:14:48.274060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:14:48,276 INFO - 2018-07-16 10:14:48.274187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:14:48,276 INFO - 2018-07-16 10:14:48.274231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:14:48,276 INFO - 2018-07-16 10:14:48.274262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:14:48,277 INFO - 2018-07-16 10:14:48.274455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:14:48,277 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1821
2018-07-16 03:14:49,078 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:14:49,116 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:16:07,088 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:16:07
2018-07-16 03:16:07,088 INFO - INFO:tensorflow:Saving dict for global step 1821: global_step = 1821, loss = 0.02690732
2018-07-16 03:16:07,089 INFO - {'loss': 0.02690732, 'global_step': 1821}
2018-07-16 03:16:07,089 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0628011dd0>]
2018-07-16 03:16:07,089 INFO - Starting a training cycle.
2018-07-16 03:16:07,709 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:16:10,497 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:16:10,497 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:16:11,094 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:16:11,095 INFO - 2018-07-16 10:16:11.092521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:16:11,095 INFO - 2018-07-16 10:16:11.092668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:16:11,095 INFO - 2018-07-16 10:16:11.092713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:16:11,095 INFO - 2018-07-16 10:16:11.092783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:16:11,095 INFO - 2018-07-16 10:16:11.093006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:16:11,096 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-1821
2018-07-16 03:16:11,871 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:16:11,907 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:16:17,096 INFO - INFO:tensorflow:Saving checkpoints for 1822 into /output/model.ckpt.
2018-07-16 03:16:18,094 INFO - INFO:tensorflow:loss = 0.025381958, step = 1821
2018-07-16 03:16:57,180 INFO - INFO:tensorflow:global_step/sec: 2.55837
2018-07-16 03:16:57,181 INFO - INFO:tensorflow:loss = 0.028696753, step = 1921 (39.088 sec)
2018-07-16 03:17:35,784 INFO - INFO:tensorflow:global_step/sec: 2.5904
2018-07-16 03:17:35,785 INFO - INFO:tensorflow:loss = 0.024793472, step = 2021 (38.604 sec)
2018-07-16 03:18:14,421 INFO - INFO:tensorflow:global_step/sec: 2.58822
2018-07-16 03:18:14,422 INFO - INFO:tensorflow:loss = 0.026167642, step = 2121 (38.637 sec)
2018-07-16 03:18:53,056 INFO - INFO:tensorflow:global_step/sec: 2.58833
2018-07-16 03:18:53,057 INFO - INFO:tensorflow:loss = 0.029475011, step = 2221 (38.635 sec)
2018-07-16 03:19:31,687 INFO - INFO:tensorflow:global_step/sec: 2.58861
2018-07-16 03:19:31,687 INFO - INFO:tensorflow:loss = 0.031367753, step = 2321 (38.630 sec)
2018-07-16 03:20:10,324 INFO - INFO:tensorflow:global_step/sec: 2.58819
2018-07-16 03:20:10,324 INFO - INFO:tensorflow:loss = 0.023750514, step = 2421 (38.637 sec)
2018-07-16 03:20:12,540 INFO - INFO:tensorflow:Saving checkpoints for 2428 into /output/model.ckpt.
2018-07-16 03:20:13,338 INFO - Starting to evaluate.
2018-07-16 03:20:13,338 INFO - INFO:tensorflow:Loss for final step: 0.032847315.
2018-07-16 03:20:13,849 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:20:16,890 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:20:16,918 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:20:16
2018-07-16 03:20:17,629 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:20:17,630 INFO - 2018-07-16 10:20:17.628775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:20:17,630 INFO - 2018-07-16 10:20:17.628914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:20:17,630 INFO - 2018-07-16 10:20:17.628982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:20:17,631 INFO - 2018-07-16 10:20:17.629060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:20:17,631 INFO - 2018-07-16 10:20:17.629436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:20:17,631 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-2428
2018-07-16 03:20:18,423 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:20:18,459 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:21:36,749 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:21:36
2018-07-16 03:21:36,749 INFO - INFO:tensorflow:Saving dict for global step 2428: global_step = 2428, loss = 0.026627304
2018-07-16 03:21:36,750 INFO - {'loss': 0.026627304, 'global_step': 2428}
2018-07-16 03:21:36,750 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f062808df90>]
2018-07-16 03:21:36,750 INFO - Starting a training cycle.
2018-07-16 03:21:37,352 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:21:40,133 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:21:40,134 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:21:40,893 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:21:40,893 INFO - 2018-07-16 10:21:40.891882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:21:40,894 INFO - 2018-07-16 10:21:40.892036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:21:40,894 INFO - 2018-07-16 10:21:40.892081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:21:40,894 INFO - 2018-07-16 10:21:40.892133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:21:40,894 INFO - 2018-07-16 10:21:40.892356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:21:40,895 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-2428
2018-07-16 03:21:41,668 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:21:41,703 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:21:46,874 INFO - INFO:tensorflow:Saving checkpoints for 2429 into /output/model.ckpt.
2018-07-16 03:21:47,725 INFO - INFO:tensorflow:loss = 0.03068332, step = 2428
2018-07-16 03:22:26,735 INFO - INFO:tensorflow:global_step/sec: 2.56339
2018-07-16 03:22:26,736 INFO - INFO:tensorflow:loss = 0.02880114, step = 2528 (39.011 sec)
2018-07-16 03:23:05,344 INFO - INFO:tensorflow:global_step/sec: 2.59004
2018-07-16 03:23:05,346 INFO - INFO:tensorflow:loss = 0.023862667, step = 2628 (38.611 sec)
2018-07-16 03:23:44,015 INFO - INFO:tensorflow:global_step/sec: 2.58598
2018-07-16 03:23:44,015 INFO - INFO:tensorflow:loss = 0.030073397, step = 2728 (38.669 sec)
2018-07-16 03:24:22,671 INFO - INFO:tensorflow:global_step/sec: 2.58691
2018-07-16 03:24:22,672 INFO - INFO:tensorflow:loss = 0.039397493, step = 2828 (38.657 sec)
2018-07-16 03:25:01,338 INFO - INFO:tensorflow:global_step/sec: 2.58611
2018-07-16 03:25:01,340 INFO - INFO:tensorflow:loss = 0.024659168, step = 2928 (38.668 sec)
2018-07-16 03:25:40,019 INFO - INFO:tensorflow:global_step/sec: 2.58531
2018-07-16 03:25:40,019 INFO - INFO:tensorflow:loss = 0.02135265, step = 3028 (38.679 sec)
2018-07-16 03:25:42,239 INFO - INFO:tensorflow:Saving checkpoints for 3035 into /output/model.ckpt.
2018-07-16 03:25:43,031 INFO - INFO:tensorflow:Loss for final step: 0.024648609.
2018-07-16 03:25:43,032 INFO - Starting to evaluate.
2018-07-16 03:25:43,539 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:25:46,447 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:25:46,471 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:25:46
2018-07-16 03:25:47,328 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:25:47,415 INFO - 2018-07-16 10:25:47.326456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:25:47,416 INFO - 2018-07-16 10:25:47.326605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:25:47,416 INFO - 2018-07-16 10:25:47.326646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:25:47,416 INFO - 2018-07-16 10:25:47.326701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:25:47,416 INFO - 2018-07-16 10:25:47.326914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:25:47,416 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3035
2018-07-16 03:25:48,163 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:25:48,204 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:27:06,422 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:27:06
2018-07-16 03:27:06,422 INFO - INFO:tensorflow:Saving dict for global step 3035: global_step = 3035, loss = 0.020604685
2018-07-16 03:27:06,422 INFO - {'loss': 0.020604685, 'global_step': 3035}
2018-07-16 03:27:06,422 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f06276de2d0>]
2018-07-16 03:27:06,423 INFO - Starting a training cycle.
2018-07-16 03:27:07,033 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:27:09,686 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:27:09,687 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:27:10,421 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:27:10,421 INFO - 2018-07-16 10:27:10.419433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:27:10,421 INFO - 2018-07-16 10:27:10.419580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:27:10,422 INFO - 2018-07-16 10:27:10.419623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:27:10,422 INFO - 2018-07-16 10:27:10.419678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:27:10,422 INFO - 2018-07-16 10:27:10.419902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:27:10,422 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3035
2018-07-16 03:27:11,206 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:27:11,245 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:27:16,392 INFO - INFO:tensorflow:Saving checkpoints for 3036 into /output/model.ckpt.
2018-07-16 03:27:17,221 INFO - INFO:tensorflow:loss = 0.023330076, step = 3035
2018-07-16 03:27:56,311 INFO - INFO:tensorflow:global_step/sec: 2.55818
2018-07-16 03:27:56,312 INFO - INFO:tensorflow:loss = 0.019722741, step = 3135 (39.091 sec)
2018-07-16 03:28:34,927 INFO - INFO:tensorflow:global_step/sec: 2.58957
2018-07-16 03:28:34,929 INFO - INFO:tensorflow:loss = 0.027512807, step = 3235 (38.617 sec)
2018-07-16 03:29:13,533 INFO - INFO:tensorflow:global_step/sec: 2.59031
2018-07-16 03:29:13,534 INFO - INFO:tensorflow:loss = 0.023662595, step = 3335 (38.604 sec)
2018-07-16 03:29:52,175 INFO - INFO:tensorflow:global_step/sec: 2.58787
2018-07-16 03:29:52,176 INFO - INFO:tensorflow:loss = 0.02397204, step = 3435 (38.642 sec)
2018-07-16 03:30:30,797 INFO - INFO:tensorflow:global_step/sec: 2.58921
2018-07-16 03:30:30,798 INFO - INFO:tensorflow:loss = 0.027670301, step = 3535 (38.623 sec)
2018-07-16 03:31:09,406 INFO - INFO:tensorflow:global_step/sec: 2.59008
2018-07-16 03:31:09,406 INFO - INFO:tensorflow:loss = 0.02794463, step = 3635 (38.608 sec)
2018-07-16 03:31:11,633 INFO - INFO:tensorflow:Saving checkpoints for 3642 into /output/model.ckpt.
2018-07-16 03:31:12,563 INFO - INFO:tensorflow:Loss for final step: 0.024918836.
2018-07-16 03:31:12,564 INFO - Starting to evaluate.
2018-07-16 03:31:13,077 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:31:15,957 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:31:15,984 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:31:15
2018-07-16 03:31:16,960 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:31:16,961 INFO - 2018-07-16 10:31:16.960129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:31:16,961 INFO - 2018-07-16 10:31:16.960261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:31:16,961 INFO - 2018-07-16 10:31:16.960303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:31:16,962 INFO - 2018-07-16 10:31:16.960356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:31:16,962 INFO - 2018-07-16 10:31:16.960546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:31:16,962 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3642
2018-07-16 03:31:17,737 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:31:17,774 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:32:35,635 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:32:35
2018-07-16 03:32:35,636 INFO - INFO:tensorflow:Saving dict for global step 3642: global_step = 3642, loss = 0.018940382
2018-07-16 03:32:35,636 INFO - {'loss': 0.018940382, 'global_step': 3642}
2018-07-16 03:32:35,636 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0627aed090>]
2018-07-16 03:32:35,636 INFO - Starting a training cycle.
2018-07-16 03:32:36,258 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:32:39,062 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:32:39,063 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:32:39,651 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:32:39,652 INFO - 2018-07-16 10:32:39.650726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:32:39,652 INFO - 2018-07-16 10:32:39.650865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:32:39,653 INFO - 2018-07-16 10:32:39.650906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:32:39,653 INFO - 2018-07-16 10:32:39.650960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:32:39,653 INFO - 2018-07-16 10:32:39.651154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:32:39,653 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-3642
2018-07-16 03:32:40,423 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:32:40,455 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:32:45,632 INFO - INFO:tensorflow:Saving checkpoints for 3643 into /output/model.ckpt.
2018-07-16 03:32:46,661 INFO - INFO:tensorflow:loss = 0.02898778, step = 3642
2018-07-16 03:33:25,701 INFO - INFO:tensorflow:global_step/sec: 2.56139
2018-07-16 03:33:25,702 INFO - INFO:tensorflow:loss = 0.019567482, step = 3742 (39.042 sec)
2018-07-16 03:34:04,303 INFO - INFO:tensorflow:global_step/sec: 2.59052
2018-07-16 03:34:04,304 INFO - INFO:tensorflow:loss = 0.028895114, step = 3842 (38.603 sec)
2018-07-16 03:34:42,928 INFO - INFO:tensorflow:global_step/sec: 2.589
2018-07-16 03:34:42,930 INFO - INFO:tensorflow:loss = 0.024386756, step = 3942 (38.626 sec)
2018-07-16 03:35:21,555 INFO - INFO:tensorflow:global_step/sec: 2.58888
2018-07-16 03:35:21,557 INFO - INFO:tensorflow:loss = 0.021417174, step = 4042 (38.626 sec)
2018-07-16 03:36:00,169 INFO - INFO:tensorflow:global_step/sec: 2.58975
2018-07-16 03:36:00,170 INFO - INFO:tensorflow:loss = 0.021436598, step = 4142 (38.614 sec)
2018-07-16 03:36:38,803 INFO - INFO:tensorflow:global_step/sec: 2.58838
2018-07-16 03:36:38,805 INFO - INFO:tensorflow:loss = 0.022236176, step = 4242 (38.635 sec)
2018-07-16 03:36:41,029 INFO - INFO:tensorflow:Saving checkpoints for 4249 into /output/model.ckpt.
2018-07-16 03:36:41,818 INFO - INFO:tensorflow:Loss for final step: 0.031240867.
2018-07-16 03:36:41,818 INFO - Starting to evaluate.
2018-07-16 03:36:42,328 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:36:45,334 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:36:45,362 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:36:45
2018-07-16 03:36:46,023 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:36:46,023 INFO - 2018-07-16 10:36:46.021882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:36:46,023 INFO - 2018-07-16 10:36:46.022035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:36:46,024 INFO - 2018-07-16 10:36:46.022079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:36:46,024 INFO - 2018-07-16 10:36:46.022114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:36:46,024 INFO - 2018-07-16 10:36:46.022293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:36:46,024 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4249
2018-07-16 03:36:46,800 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:36:46,843 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:38:04,895 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:38:04
2018-07-16 03:38:04,896 INFO - INFO:tensorflow:Saving dict for global step 4249: global_step = 4249, loss = 0.01792569
2018-07-16 03:38:04,897 INFO - {'loss': 0.01792569, 'global_step': 4249}
2018-07-16 03:38:04,897 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f06276de290>]
2018-07-16 03:38:04,897 INFO - Starting a training cycle.
2018-07-16 03:38:05,497 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:38:08,285 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:38:08,286 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:38:09,036 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:38:09,037 INFO - 2018-07-16 10:38:09.035125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:38:09,037 INFO - 2018-07-16 10:38:09.035257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:38:09,037 INFO - 2018-07-16 10:38:09.035296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:38:09,038 INFO - 2018-07-16 10:38:09.035326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:38:09,038 INFO - 2018-07-16 10:38:09.035508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:38:09,038 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4249
2018-07-16 03:38:09,842 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:38:09,876 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:38:14,986 INFO - INFO:tensorflow:Saving checkpoints for 4250 into /output/model.ckpt.
2018-07-16 03:38:15,944 INFO - INFO:tensorflow:loss = 0.025027521, step = 4249
2018-07-16 03:38:54,946 INFO - INFO:tensorflow:global_step/sec: 2.56392
2018-07-16 03:38:54,948 INFO - INFO:tensorflow:loss = 0.02650905, step = 4349 (39.005 sec)
2018-07-16 03:39:33,513 INFO - INFO:tensorflow:global_step/sec: 2.59293
2018-07-16 03:39:33,515 INFO - INFO:tensorflow:loss = 0.025246225, step = 4449 (38.567 sec)
2018-07-16 03:40:12,152 INFO - INFO:tensorflow:global_step/sec: 2.58804
2018-07-16 03:40:12,153 INFO - INFO:tensorflow:loss = 0.025537178, step = 4549 (38.638 sec)
2018-07-16 03:40:50,783 INFO - INFO:tensorflow:global_step/sec: 2.58861
2018-07-16 03:40:50,784 INFO - INFO:tensorflow:loss = 0.026803564, step = 4649 (38.632 sec)
2018-07-16 03:41:29,424 INFO - INFO:tensorflow:global_step/sec: 2.58849
2018-07-16 03:41:29,425 INFO - INFO:tensorflow:loss = 0.032123264, step = 4749 (38.632 sec)
2018-07-16 03:42:08,070 INFO - INFO:tensorflow:global_step/sec: 2.58705
2018-07-16 03:42:08,071 INFO - INFO:tensorflow:loss = 0.022899888, step = 4849 (38.654 sec)
2018-07-16 03:42:10,287 INFO - INFO:tensorflow:Saving checkpoints for 4856 into /output/model.ckpt.
2018-07-16 03:42:11,069 INFO - INFO:tensorflow:Loss for final step: 0.029737856.
2018-07-16 03:42:11,069 INFO - Starting to evaluate.
2018-07-16 03:42:11,579 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:42:14,986 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:42:15,014 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:42:15
2018-07-16 03:42:15,924 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:42:15,924 INFO - 2018-07-16 10:42:15.922462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:42:15,925 INFO - 2018-07-16 10:42:15.922611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:42:15,925 INFO - 2018-07-16 10:42:15.922641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:42:15,925 INFO - 2018-07-16 10:42:15.922659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:42:15,925 INFO - 2018-07-16 10:42:15.922860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:42:15,925 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4856
2018-07-16 03:42:16,684 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:42:16,716 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:43:34,407 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:43:34
2018-07-16 03:43:34,408 INFO - INFO:tensorflow:Saving dict for global step 4856: global_step = 4856, loss = 0.01804525
2018-07-16 03:43:34,408 INFO - {'loss': 0.01804525, 'global_step': 4856}
2018-07-16 03:43:34,408 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0627ac0b10>]
2018-07-16 03:43:34,408 INFO - Starting a training cycle.
2018-07-16 03:43:35,029 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:43:37,839 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:43:37,839 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:43:38,430 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:43:38,430 INFO - 2018-07-16 10:43:38.428662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:43:38,430 INFO - 2018-07-16 10:43:38.428844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:43:38,431 INFO - 2018-07-16 10:43:38.428890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:43:38,431 INFO - 2018-07-16 10:43:38.428917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:43:38,431 INFO - 2018-07-16 10:43:38.429113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:43:38,431 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-4856
2018-07-16 03:43:39,205 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:43:39,236 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:43:44,522 INFO - INFO:tensorflow:Saving checkpoints for 4857 into /output/model.ckpt.
2018-07-16 03:43:45,536 INFO - INFO:tensorflow:loss = 0.021457177, step = 4856
2018-07-16 03:44:24,592 INFO - INFO:tensorflow:global_step/sec: 2.56026
2018-07-16 03:44:24,593 INFO - INFO:tensorflow:loss = 0.024052806, step = 4956 (39.059 sec)
2018-07-16 03:45:03,181 INFO - INFO:tensorflow:global_step/sec: 2.59144
2018-07-16 03:45:03,181 INFO - INFO:tensorflow:loss = 0.022045638, step = 5056 (38.588 sec)
2018-07-16 03:45:41,782 INFO - INFO:tensorflow:global_step/sec: 2.59056
2018-07-16 03:45:41,783 INFO - INFO:tensorflow:loss = 0.024035556, step = 5156 (38.602 sec)
2018-07-16 03:46:20,433 INFO - INFO:tensorflow:global_step/sec: 2.58734
2018-07-16 03:46:20,433 INFO - INFO:tensorflow:loss = 0.024453467, step = 5256 (38.650 sec)
2018-07-16 03:46:59,071 INFO - INFO:tensorflow:global_step/sec: 2.58806
2018-07-16 03:46:59,072 INFO - INFO:tensorflow:loss = 0.019713532, step = 5356 (38.639 sec)
2018-07-16 03:47:37,665 INFO - INFO:tensorflow:global_step/sec: 2.59106
2018-07-16 03:47:37,666 INFO - INFO:tensorflow:loss = 0.029800143, step = 5456 (38.594 sec)
2018-07-16 03:47:39,896 INFO - INFO:tensorflow:Saving checkpoints for 5463 into /output/model.ckpt.
2018-07-16 03:47:40,653 INFO - Starting to evaluate.
2018-07-16 03:47:40,653 INFO - INFO:tensorflow:Loss for final step: 0.029183932.
2018-07-16 03:47:41,161 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:47:43,875 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:47:43,898 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:47:43
2018-07-16 03:47:44,948 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:47:45,039 INFO - 2018-07-16 10:47:44.947264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:47:45,039 INFO - 2018-07-16 10:47:44.947420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:47:45,040 INFO - 2018-07-16 10:47:44.947454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:47:45,040 INFO - 2018-07-16 10:47:44.947475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:47:45,040 INFO - 2018-07-16 10:47:44.947668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:47:45,040 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-5463
2018-07-16 03:47:45,794 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:47:45,832 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:49:03,722 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:49:03
2018-07-16 03:49:03,722 INFO - INFO:tensorflow:Saving dict for global step 5463: global_step = 5463, loss = 0.018647188
2018-07-16 03:49:03,723 INFO - {'loss': 0.018647188, 'global_step': 5463}
2018-07-16 03:49:03,723 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0627c96fd0>]
2018-07-16 03:49:03,723 INFO - Starting a training cycle.
2018-07-16 03:49:04,330 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:49:07,136 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:49:07,137 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:49:07,729 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:49:07,729 INFO - 2018-07-16 10:49:07.728411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:49:07,729 INFO - 2018-07-16 10:49:07.728561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:49:07,730 INFO - 2018-07-16 10:49:07.728604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:49:07,730 INFO - 2018-07-16 10:49:07.728655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:49:07,731 INFO - 2018-07-16 10:49:07.728881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:49:07,731 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-5463
2018-07-16 03:49:08,508 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:49:08,548 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:49:13,759 INFO - INFO:tensorflow:Saving checkpoints for 5464 into /output/model.ckpt.
2018-07-16 03:49:14,771 INFO - INFO:tensorflow:loss = 0.027939528, step = 5463
2018-07-16 03:49:53,798 INFO - INFO:tensorflow:global_step/sec: 2.56228
2018-07-16 03:49:53,800 INFO - INFO:tensorflow:loss = 0.022956874, step = 5563 (39.029 sec)
2018-07-16 03:50:32,377 INFO - INFO:tensorflow:global_step/sec: 2.59211
2018-07-16 03:50:32,379 INFO - INFO:tensorflow:loss = 0.025884798, step = 5663 (38.579 sec)
2018-07-16 03:51:11,003 INFO - INFO:tensorflow:global_step/sec: 2.58889
2018-07-16 03:51:11,004 INFO - INFO:tensorflow:loss = 0.021645699, step = 5763 (38.626 sec)
2018-07-16 03:51:49,632 INFO - INFO:tensorflow:global_step/sec: 2.58879
2018-07-16 03:51:49,633 INFO - INFO:tensorflow:loss = 0.019706067, step = 5863 (38.628 sec)
2018-07-16 03:52:28,280 INFO - INFO:tensorflow:global_step/sec: 2.58743
2018-07-16 03:52:28,281 INFO - INFO:tensorflow:loss = 0.02360426, step = 5963 (38.648 sec)
2018-07-16 03:53:06,935 INFO - INFO:tensorflow:global_step/sec: 2.58701
2018-07-16 03:53:06,936 INFO - INFO:tensorflow:loss = 0.023699682, step = 6063 (38.655 sec)
2018-07-16 03:53:09,167 INFO - INFO:tensorflow:Saving checkpoints for 6070 into /output/model.ckpt.
2018-07-16 03:53:09,952 INFO - INFO:tensorflow:Loss for final step: 0.028932134.
2018-07-16 03:53:09,953 INFO - Starting to evaluate.
2018-07-16 03:53:10,461 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:53:13,856 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:53:13,883 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:53:13
2018-07-16 03:53:14,598 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:53:14,599 INFO - 2018-07-16 10:53:14.597650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:53:14,599 INFO - 2018-07-16 10:53:14.597786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:53:14,599 INFO - 2018-07-16 10:53:14.597822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:53:14,599 INFO - 2018-07-16 10:53:14.597855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:53:14,600 INFO - 2018-07-16 10:53:14.598022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:53:14,600 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6070
2018-07-16 03:53:15,472 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:53:15,518 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:54:33,646 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-10:54:33
2018-07-16 03:54:33,647 INFO - INFO:tensorflow:Saving dict for global step 6070: global_step = 6070, loss = 0.016862437
2018-07-16 03:54:33,647 INFO - {'loss': 0.016862437, 'global_step': 6070}
2018-07-16 03:54:33,647 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f05ceed3310>]
2018-07-16 03:54:33,647 INFO - Starting a training cycle.
2018-07-16 03:54:34,246 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:54:37,004 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:54:37,005 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 03:54:37,749 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:54:37,750 INFO - 2018-07-16 10:54:37.748859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:54:37,750 INFO - 2018-07-16 10:54:37.749000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:54:37,750 INFO - 2018-07-16 10:54:37.749045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:54:37,751 INFO - 2018-07-16 10:54:37.749099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:54:37,751 INFO - 2018-07-16 10:54:37.749317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:54:37,751 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6070
2018-07-16 03:54:38,535 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:54:38,568 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 03:54:43,705 INFO - INFO:tensorflow:Saving checkpoints for 6071 into /output/model.ckpt.
2018-07-16 03:54:44,535 INFO - INFO:tensorflow:loss = 0.02870389, step = 6070
2018-07-16 03:55:23,750 INFO - INFO:tensorflow:global_step/sec: 2.54999
2018-07-16 03:55:23,752 INFO - INFO:tensorflow:loss = 0.027582042, step = 6170 (39.218 sec)
2018-07-16 03:56:02,397 INFO - INFO:tensorflow:global_step/sec: 2.58752
2018-07-16 03:56:02,398 INFO - INFO:tensorflow:loss = 0.024907291, step = 6270 (38.646 sec)
2018-07-16 03:56:41,025 INFO - INFO:tensorflow:global_step/sec: 2.58881
2018-07-16 03:56:41,026 INFO - INFO:tensorflow:loss = 0.025403786, step = 6370 (38.628 sec)
2018-07-16 03:57:19,670 INFO - INFO:tensorflow:global_step/sec: 2.58766
2018-07-16 03:57:19,672 INFO - INFO:tensorflow:loss = 0.022241702, step = 6470 (38.646 sec)
2018-07-16 03:57:58,344 INFO - INFO:tensorflow:global_step/sec: 2.58572
2018-07-16 03:57:58,345 INFO - INFO:tensorflow:loss = 0.023470663, step = 6570 (38.673 sec)
2018-07-16 03:58:37,031 INFO - INFO:tensorflow:global_step/sec: 2.58483
2018-07-16 03:58:37,033 INFO - INFO:tensorflow:loss = 0.020294651, step = 6670 (38.688 sec)
2018-07-16 03:58:39,273 INFO - INFO:tensorflow:Saving checkpoints for 6677 into /output/model.ckpt.
2018-07-16 03:58:40,041 INFO - Starting to evaluate.
2018-07-16 03:58:40,041 INFO - INFO:tensorflow:Loss for final step: 0.028447429.
2018-07-16 03:58:40,548 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 03:58:43,593 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 03:58:43,621 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-10:58:43
2018-07-16 03:58:44,518 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 03:58:44,593 INFO - 2018-07-16 10:58:44.513738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 03:58:44,593 INFO - 2018-07-16 10:58:44.513921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 03:58:44,594 INFO - 2018-07-16 10:58:44.514015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 03:58:44,594 INFO - 2018-07-16 10:58:44.514093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 03:58:44,594 INFO - 2018-07-16 10:58:44.514370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 03:58:44,594 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6677
2018-07-16 03:58:45,350 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 03:58:45,383 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:00:03,696 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:00:03
2018-07-16 04:00:03,697 INFO - INFO:tensorflow:Saving dict for global step 6677: global_step = 6677, loss = 0.016778985
2018-07-16 04:00:03,697 INFO - {'loss': 0.016778985, 'global_step': 6677}
2018-07-16 04:00:03,697 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0628942cd0>]
2018-07-16 04:00:03,697 INFO - Starting a training cycle.
2018-07-16 04:00:04,293 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:00:07,103 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:00:07,104 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:00:07,697 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:00:07,698 INFO - 2018-07-16 11:00:07.696534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:00:07,698 INFO - 2018-07-16 11:00:07.696675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:00:07,698 INFO - 2018-07-16 11:00:07.696712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:00:07,698 INFO - 2018-07-16 11:00:07.696742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:00:07,699 INFO - 2018-07-16 11:00:07.696919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:00:07,699 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-6677
2018-07-16 04:00:08,490 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:00:08,529 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:00:13,717 INFO - INFO:tensorflow:Saving checkpoints for 6678 into /output/model.ckpt.
2018-07-16 04:00:14,551 INFO - INFO:tensorflow:loss = 0.022422092, step = 6677
2018-07-16 04:00:53,631 INFO - INFO:tensorflow:global_step/sec: 2.55879
2018-07-16 04:00:53,631 INFO - INFO:tensorflow:loss = 0.021560472, step = 6777 (39.082 sec)
2018-07-16 04:01:32,225 INFO - INFO:tensorflow:global_step/sec: 2.59105
2018-07-16 04:01:32,227 INFO - INFO:tensorflow:loss = 0.02335257, step = 6877 (38.596 sec)
2018-07-16 04:02:10,836 INFO - INFO:tensorflow:global_step/sec: 2.58995
2018-07-16 04:02:10,837 INFO - INFO:tensorflow:loss = 0.027076697, step = 6977 (38.610 sec)
2018-07-16 04:02:49,502 INFO - INFO:tensorflow:global_step/sec: 2.58642
2018-07-16 04:02:49,504 INFO - INFO:tensorflow:loss = 0.034757283, step = 7077 (38.667 sec)
2018-07-16 04:03:28,187 INFO - INFO:tensorflow:global_step/sec: 2.58479
2018-07-16 04:03:28,188 INFO - INFO:tensorflow:loss = 0.024613054, step = 7177 (38.684 sec)
2018-07-16 04:04:06,816 INFO - INFO:tensorflow:global_step/sec: 2.58878
2018-07-16 04:04:06,817 INFO - INFO:tensorflow:loss = 0.01853323, step = 7277 (38.630 sec)
2018-07-16 04:04:09,027 INFO - INFO:tensorflow:Saving checkpoints for 7284 into /output/model.ckpt.
2018-07-16 04:04:10,030 INFO - INFO:tensorflow:Loss for final step: 0.024636978.
2018-07-16 04:04:10,030 INFO - Starting to evaluate.
2018-07-16 04:04:10,543 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:04:13,168 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:04:13,191 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:04:13
2018-07-16 04:04:14,105 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:04:14,105 INFO - 2018-07-16 11:04:14.097161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:04:14,106 INFO - 2018-07-16 11:04:14.097319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:04:14,106 INFO - 2018-07-16 11:04:14.097355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:04:14,106 INFO - 2018-07-16 11:04:14.097381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:04:14,107 INFO - 2018-07-16 11:04:14.097576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:04:14,107 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7284
2018-07-16 04:04:15,084 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:04:15,124 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:05:32,956 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:05:32
2018-07-16 04:05:32,957 INFO - INFO:tensorflow:Saving dict for global step 7284: global_step = 7284, loss = 0.015960857
2018-07-16 04:05:32,957 INFO - {'loss': 0.015960857, 'global_step': 7284}
2018-07-16 04:05:32,957 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0631af0cd0>]
2018-07-16 04:05:32,957 INFO - Starting a training cycle.
2018-07-16 04:05:33,571 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:05:36,368 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:05:36,369 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:05:36,963 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:05:36,964 INFO - 2018-07-16 11:05:36.962204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:05:36,964 INFO - 2018-07-16 11:05:36.962359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:05:36,965 INFO - 2018-07-16 11:05:36.962403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:05:36,965 INFO - 2018-07-16 11:05:36.962458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:05:36,965 INFO - 2018-07-16 11:05:36.962723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:05:36,965 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7284
2018-07-16 04:05:37,742 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:05:37,775 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:05:42,941 INFO - INFO:tensorflow:Saving checkpoints for 7285 into /output/model.ckpt.
2018-07-16 04:05:44,016 INFO - INFO:tensorflow:loss = 0.03441489, step = 7284
2018-07-16 04:06:23,242 INFO - INFO:tensorflow:global_step/sec: 2.54919
2018-07-16 04:06:23,243 INFO - INFO:tensorflow:loss = 0.019963987, step = 7384 (39.228 sec)
2018-07-16 04:07:01,832 INFO - INFO:tensorflow:global_step/sec: 2.59139
2018-07-16 04:07:01,832 INFO - INFO:tensorflow:loss = 0.024418296, step = 7484 (38.589 sec)
2018-07-16 04:07:40,479 INFO - INFO:tensorflow:global_step/sec: 2.58747
2018-07-16 04:07:40,480 INFO - INFO:tensorflow:loss = 0.021511488, step = 7584 (38.648 sec)
2018-07-16 04:08:19,151 INFO - INFO:tensorflow:global_step/sec: 2.58588
2018-07-16 04:08:19,153 INFO - INFO:tensorflow:loss = 0.018229244, step = 7684 (38.673 sec)
2018-07-16 04:08:57,774 INFO - INFO:tensorflow:global_step/sec: 2.58915
2018-07-16 04:08:57,774 INFO - INFO:tensorflow:loss = 0.019313639, step = 7784 (38.622 sec)
2018-07-16 04:09:36,411 INFO - INFO:tensorflow:global_step/sec: 2.58816
2018-07-16 04:09:36,412 INFO - INFO:tensorflow:loss = 0.021614123, step = 7884 (38.637 sec)
2018-07-16 04:09:38,665 INFO - INFO:tensorflow:Saving checkpoints for 7891 into /output/model.ckpt.
2018-07-16 04:09:39,440 INFO - INFO:tensorflow:Loss for final step: 0.022679824.
2018-07-16 04:09:39,441 INFO - Starting to evaluate.
2018-07-16 04:09:39,961 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:09:43,223 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:09:43,251 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:09:43
2018-07-16 04:09:43,999 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:09:44,000 INFO - 2018-07-16 11:09:43.997932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:09:44,000 INFO - 2018-07-16 11:09:43.998064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:09:44,000 INFO - 2018-07-16 11:09:43.998107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:09:44,000 INFO - 2018-07-16 11:09:43.998148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:09:44,001 INFO - 2018-07-16 11:09:43.998318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:09:44,001 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7891
2018-07-16 04:09:44,809 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:09:44,846 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:11:02,923 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:11:02
2018-07-16 04:11:02,923 INFO - INFO:tensorflow:Saving dict for global step 7891: global_step = 7891, loss = 0.01564516
2018-07-16 04:11:02,924 INFO - {'loss': 0.01564516, 'global_step': 7891}
2018-07-16 04:11:02,924 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0627868a50>]
2018-07-16 04:11:02,924 INFO - Starting a training cycle.
2018-07-16 04:11:03,531 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:11:06,307 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:11:06,308 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:11:07,057 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:11:07,057 INFO - 2018-07-16 11:11:07.055185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:11:07,057 INFO - 2018-07-16 11:11:07.055355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:11:07,057 INFO - 2018-07-16 11:11:07.055399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:11:07,058 INFO - 2018-07-16 11:11:07.055428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:11:07,058 INFO - 2018-07-16 11:11:07.055616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:11:07,058 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-7891
2018-07-16 04:11:07,842 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:11:07,873 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:11:13,052 INFO - INFO:tensorflow:Saving checkpoints for 7892 into /output/model.ckpt.
2018-07-16 04:11:13,909 INFO - INFO:tensorflow:loss = 0.020584311, step = 7891
2018-07-16 04:11:53,018 INFO - INFO:tensorflow:global_step/sec: 2.55684
2018-07-16 04:11:53,019 INFO - INFO:tensorflow:loss = 0.021234363, step = 7991 (39.111 sec)
2018-07-16 04:12:31,587 INFO - INFO:tensorflow:global_step/sec: 2.59276
2018-07-16 04:12:31,589 INFO - INFO:tensorflow:loss = 0.024436459, step = 8091 (38.570 sec)
2018-07-16 04:13:10,226 INFO - INFO:tensorflow:global_step/sec: 2.58809
2018-07-16 04:13:10,226 INFO - INFO:tensorflow:loss = 0.018463772, step = 8191 (38.638 sec)
2018-07-16 04:13:48,880 INFO - INFO:tensorflow:global_step/sec: 2.58702
2018-07-16 04:13:48,880 INFO - INFO:tensorflow:loss = 0.020601805, step = 8291 (38.655 sec)
2018-07-16 04:14:27,508 INFO - INFO:tensorflow:global_step/sec: 2.58882
2018-07-16 04:14:27,508 INFO - INFO:tensorflow:loss = 0.022418287, step = 8391 (38.628 sec)
2018-07-16 04:15:06,135 INFO - INFO:tensorflow:global_step/sec: 2.58881
2018-07-16 04:15:06,137 INFO - INFO:tensorflow:loss = 0.01818015, step = 8491 (38.628 sec)
2018-07-16 04:15:08,364 INFO - INFO:tensorflow:Saving checkpoints for 8498 into /output/model.ckpt.
2018-07-16 04:15:09,127 INFO - Starting to evaluate.
2018-07-16 04:15:09,127 INFO - INFO:tensorflow:Loss for final step: 0.027443659.
2018-07-16 04:15:09,643 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:15:12,537 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:15:12,565 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:15:12
2018-07-16 04:15:13,408 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:15:13,409 INFO - 2018-07-16 11:15:13.407121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:15:13,410 INFO - 2018-07-16 11:15:13.407261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:15:13,410 INFO - 2018-07-16 11:15:13.407306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:15:13,410 INFO - 2018-07-16 11:15:13.407358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:15:13,411 INFO - 2018-07-16 11:15:13.407558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:15:13,411 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-8498
2018-07-16 04:15:14,338 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:15:14,376 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:16:32,379 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:16:32
2018-07-16 04:16:32,380 INFO - INFO:tensorflow:Saving dict for global step 8498: global_step = 8498, loss = 0.015619421
2018-07-16 04:16:32,380 INFO - {'loss': 0.015619421, 'global_step': 8498}
2018-07-16 04:16:32,380 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0631bc6c50>]
2018-07-16 04:16:32,380 INFO - Starting a training cycle.
2018-07-16 04:16:32,994 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:16:35,656 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:16:35,657 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:16:36,402 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:16:36,403 INFO - 2018-07-16 11:16:36.400467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:16:36,403 INFO - 2018-07-16 11:16:36.400619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:16:36,403 INFO - 2018-07-16 11:16:36.400662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:16:36,403 INFO - 2018-07-16 11:16:36.400689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:16:36,403 INFO - 2018-07-16 11:16:36.400897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:16:36,404 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-8498
2018-07-16 04:16:37,174 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:16:37,206 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:16:42,367 INFO - INFO:tensorflow:Saving checkpoints for 8499 into /output/model.ckpt.
2018-07-16 04:16:43,209 INFO - INFO:tensorflow:loss = 0.020609643, step = 8498
2018-07-16 04:17:22,220 INFO - INFO:tensorflow:global_step/sec: 2.56334
2018-07-16 04:17:22,220 INFO - INFO:tensorflow:loss = 0.021978434, step = 8598 (39.012 sec)
2018-07-16 04:18:00,814 INFO - INFO:tensorflow:global_step/sec: 2.59105
2018-07-16 04:18:00,815 INFO - INFO:tensorflow:loss = 0.021054743, step = 8698 (38.595 sec)
2018-07-16 04:18:39,448 INFO - INFO:tensorflow:global_step/sec: 2.58837
2018-07-16 04:18:39,450 INFO - INFO:tensorflow:loss = 0.029831737, step = 8798 (38.635 sec)
2018-07-16 04:19:18,101 INFO - INFO:tensorflow:global_step/sec: 2.58711
2018-07-16 04:19:18,102 INFO - INFO:tensorflow:loss = 0.02368741, step = 8898 (38.652 sec)
2018-07-16 04:19:56,758 INFO - INFO:tensorflow:global_step/sec: 2.58697
2018-07-16 04:19:56,760 INFO - INFO:tensorflow:loss = 0.019650951, step = 8998 (38.658 sec)
2018-07-16 04:20:35,384 INFO - INFO:tensorflow:global_step/sec: 2.58887
2018-07-16 04:20:35,385 INFO - INFO:tensorflow:loss = 0.018568598, step = 9098 (38.625 sec)
2018-07-16 04:20:37,616 INFO - INFO:tensorflow:Saving checkpoints for 9105 into /output/model.ckpt.
2018-07-16 04:20:38,609 INFO - INFO:tensorflow:Loss for final step: 0.03301587.
2018-07-16 04:20:38,610 INFO - Starting to evaluate.
2018-07-16 04:20:39,119 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:20:41,672 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:20:41,697 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:20:41
2018-07-16 04:20:42,675 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:20:42,675 INFO - 2018-07-16 11:20:42.673735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:20:42,676 INFO - 2018-07-16 11:20:42.673867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:20:42,676 INFO - 2018-07-16 11:20:42.673909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:20:42,676 INFO - 2018-07-16 11:20:42.673941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:20:42,676 INFO - 2018-07-16 11:20:42.674109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:20:42,676 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9105
2018-07-16 04:20:43,451 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:20:43,498 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:22:01,496 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:22:01
2018-07-16 04:22:01,496 INFO - INFO:tensorflow:Saving dict for global step 9105: global_step = 9105, loss = 0.015359037
2018-07-16 04:22:01,497 INFO - {'loss': 0.015359037, 'global_step': 9105}
2018-07-16 04:22:01,497 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f05dc0a3150>]
2018-07-16 04:22:01,497 INFO - Starting a training cycle.
2018-07-16 04:22:02,117 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:22:04,933 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:22:04,934 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:22:05,532 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:22:05,533 INFO - 2018-07-16 11:22:05.531637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:22:05,533 INFO - 2018-07-16 11:22:05.531773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:22:05,533 INFO - 2018-07-16 11:22:05.531844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:22:05,533 INFO - 2018-07-16 11:22:05.531909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:22:05,534 INFO - 2018-07-16 11:22:05.532139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:22:05,534 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9105
2018-07-16 04:22:06,480 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:22:06,517 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:22:11,916 INFO - INFO:tensorflow:Saving checkpoints for 9106 into /output/model.ckpt.
2018-07-16 04:22:12,832 INFO - INFO:tensorflow:loss = 0.020100407, step = 9105
2018-07-16 04:22:51,821 INFO - INFO:tensorflow:global_step/sec: 2.56475
2018-07-16 04:22:51,822 INFO - INFO:tensorflow:loss = 0.02445941, step = 9205 (38.991 sec)
2018-07-16 04:23:30,439 INFO - INFO:tensorflow:global_step/sec: 2.58945
2018-07-16 04:23:30,440 INFO - INFO:tensorflow:loss = 0.03239622, step = 9305 (38.618 sec)
2018-07-16 04:24:09,037 INFO - INFO:tensorflow:global_step/sec: 2.59083
2018-07-16 04:24:09,038 INFO - INFO:tensorflow:loss = 0.02422108, step = 9405 (38.597 sec)
2018-07-16 04:24:47,655 INFO - INFO:tensorflow:global_step/sec: 2.58946
2018-07-16 04:24:47,656 INFO - INFO:tensorflow:loss = 0.028446717, step = 9505 (38.618 sec)
2018-07-16 04:25:26,262 INFO - INFO:tensorflow:global_step/sec: 2.59024
2018-07-16 04:25:26,262 INFO - INFO:tensorflow:loss = 0.023695175, step = 9605 (38.606 sec)
2018-07-16 04:26:04,861 INFO - INFO:tensorflow:global_step/sec: 2.59068
2018-07-16 04:26:04,863 INFO - INFO:tensorflow:loss = 0.020974724, step = 9705 (38.600 sec)
2018-07-16 04:26:07,103 INFO - INFO:tensorflow:Saving checkpoints for 9712 into /output/model.ckpt.
2018-07-16 04:26:07,955 INFO - Starting to evaluate.
2018-07-16 04:26:07,955 INFO - INFO:tensorflow:Loss for final step: 0.034723647.
2018-07-16 04:26:08,457 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:26:11,303 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:26:11,326 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:26:11
2018-07-16 04:26:11,932 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:26:11,932 INFO - 2018-07-16 11:26:11.930974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:26:11,933 INFO - 2018-07-16 11:26:11.931122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:26:11,933 INFO - 2018-07-16 11:26:11.931158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:26:11,934 INFO - 2018-07-16 11:26:11.931176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:26:11,934 INFO - 2018-07-16 11:26:11.931358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:26:11,934 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9712
2018-07-16 04:26:12,716 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:26:12,741 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:27:30,966 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:27:30
2018-07-16 04:27:30,966 INFO - INFO:tensorflow:Saving dict for global step 9712: global_step = 9712, loss = 0.0152332615
2018-07-16 04:27:30,966 INFO - {'loss': 0.0152332615, 'global_step': 9712}
2018-07-16 04:27:30,967 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f05dca4da90>]
2018-07-16 04:27:30,967 INFO - Starting a training cycle.
2018-07-16 04:27:31,580 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:27:34,333 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:27:34,333 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:27:35,081 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:27:35,082 INFO - 2018-07-16 11:27:35.080348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:27:35,082 INFO - 2018-07-16 11:27:35.080477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:27:35,082 INFO - 2018-07-16 11:27:35.080509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:27:35,082 INFO - 2018-07-16 11:27:35.080531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:27:35,083 INFO - 2018-07-16 11:27:35.080744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:27:35,083 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-9712
2018-07-16 04:27:35,850 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:27:35,884 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:27:41,138 INFO - INFO:tensorflow:Saving checkpoints for 9713 into /output/model.ckpt.
2018-07-16 04:27:42,062 INFO - INFO:tensorflow:loss = 0.017977826, step = 9712
2018-07-16 04:28:21,117 INFO - INFO:tensorflow:global_step/sec: 2.56042
2018-07-16 04:28:21,119 INFO - INFO:tensorflow:loss = 0.022801964, step = 9812 (39.058 sec)
2018-07-16 04:28:59,728 INFO - INFO:tensorflow:global_step/sec: 2.59
2018-07-16 04:28:59,728 INFO - INFO:tensorflow:loss = 0.028891373, step = 9912 (38.609 sec)
2018-07-16 04:29:38,354 INFO - INFO:tensorflow:global_step/sec: 2.58893
2018-07-16 04:29:38,354 INFO - INFO:tensorflow:loss = 0.019966135, step = 10012 (38.626 sec)
2018-07-16 04:30:17,001 INFO - INFO:tensorflow:global_step/sec: 2.58752
2018-07-16 04:30:17,001 INFO - INFO:tensorflow:loss = 0.02593035, step = 10112 (38.647 sec)
2018-07-16 04:30:55,621 INFO - INFO:tensorflow:global_step/sec: 2.58933
2018-07-16 04:30:55,622 INFO - INFO:tensorflow:loss = 0.022322431, step = 10212 (38.621 sec)
2018-07-16 04:31:34,268 INFO - INFO:tensorflow:global_step/sec: 2.58748
2018-07-16 04:31:34,270 INFO - INFO:tensorflow:loss = 0.0251065, step = 10312 (38.648 sec)
2018-07-16 04:31:36,501 INFO - INFO:tensorflow:Saving checkpoints for 10319 into /output/model.ckpt.
2018-07-16 04:31:37,420 INFO - INFO:tensorflow:Loss for final step: 0.026657606.
2018-07-16 04:31:37,420 INFO - Starting to evaluate.
2018-07-16 04:31:37,946 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:31:40,712 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:31:40,745 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:31:40
2018-07-16 04:31:41,740 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:31:41,740 INFO - 2018-07-16 11:31:41.738407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:31:41,741 INFO - 2018-07-16 11:31:41.738605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:31:41,741 INFO - 2018-07-16 11:31:41.738687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:31:41,741 INFO - 2018-07-16 11:31:41.738750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:31:41,742 INFO - 2018-07-16 11:31:41.738959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:31:41,742 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10319
2018-07-16 04:31:42,521 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:31:42,571 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:33:00,498 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:33:00
2018-07-16 04:33:00,498 INFO - INFO:tensorflow:Saving dict for global step 10319: global_step = 10319, loss = 0.015417974
2018-07-16 04:33:00,498 INFO - {'loss': 0.015417974, 'global_step': 10319}
2018-07-16 04:33:00,498 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0631b09050>]
2018-07-16 04:33:00,499 INFO - Starting a training cycle.
2018-07-16 04:33:01,126 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:33:03,942 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:33:03,943 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:33:04,533 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:33:04,534 INFO - 2018-07-16 11:33:04.532652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:33:04,534 INFO - 2018-07-16 11:33:04.532818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:33:04,534 INFO - 2018-07-16 11:33:04.532863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:33:04,534 INFO - 2018-07-16 11:33:04.532887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:33:04,534 INFO - 2018-07-16 11:33:04.533078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:33:04,534 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10319
2018-07-16 04:33:05,315 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:33:05,352 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:33:10,583 INFO - INFO:tensorflow:Saving checkpoints for 10320 into /output/model.ckpt.
2018-07-16 04:33:11,728 INFO - INFO:tensorflow:loss = 0.025025241, step = 10319
2018-07-16 04:33:50,703 INFO - INFO:tensorflow:global_step/sec: 2.56571
2018-07-16 04:33:50,705 INFO - INFO:tensorflow:loss = 0.019528655, step = 10419 (38.977 sec)
2018-07-16 04:34:29,293 INFO - INFO:tensorflow:global_step/sec: 2.59139
2018-07-16 04:34:29,294 INFO - INFO:tensorflow:loss = 0.02014086, step = 10519 (38.588 sec)
2018-07-16 04:35:07,914 INFO - INFO:tensorflow:global_step/sec: 2.58921
2018-07-16 04:35:07,915 INFO - INFO:tensorflow:loss = 0.02386647, step = 10619 (38.622 sec)
2018-07-16 04:35:46,558 INFO - INFO:tensorflow:global_step/sec: 2.58778
2018-07-16 04:35:46,558 INFO - INFO:tensorflow:loss = 0.020888712, step = 10719 (38.643 sec)
2018-07-16 04:36:25,201 INFO - INFO:tensorflow:global_step/sec: 2.58775
2018-07-16 04:36:25,202 INFO - INFO:tensorflow:loss = 0.016232802, step = 10819 (38.643 sec)
2018-07-16 04:37:03,856 INFO - INFO:tensorflow:global_step/sec: 2.58699
2018-07-16 04:37:03,857 INFO - INFO:tensorflow:loss = 0.016907137, step = 10919 (38.655 sec)
2018-07-16 04:37:06,086 INFO - INFO:tensorflow:Saving checkpoints for 10926 into /output/model.ckpt.
2018-07-16 04:37:06,872 INFO - INFO:tensorflow:Loss for final step: 0.026694484.
2018-07-16 04:37:06,872 INFO - Starting to evaluate.
2018-07-16 04:37:07,384 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:37:10,466 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:37:10,490 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:37:10
2018-07-16 04:37:11,099 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:37:11,198 INFO - 2018-07-16 11:37:11.093074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:37:11,198 INFO - 2018-07-16 11:37:11.093214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:37:11,199 INFO - 2018-07-16 11:37:11.093259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:37:11,199 INFO - 2018-07-16 11:37:11.093317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:37:11,199 INFO - 2018-07-16 11:37:11.093558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:37:11,199 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10926
2018-07-16 04:37:11,883 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:37:11,931 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:38:30,302 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:38:30
2018-07-16 04:38:30,303 INFO - INFO:tensorflow:Saving dict for global step 10926: global_step = 10926, loss = 0.014725811
2018-07-16 04:38:30,303 INFO - {'loss': 0.014725811, 'global_step': 10926}
2018-07-16 04:38:30,303 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0628011a50>]
2018-07-16 04:38:30,303 INFO - Starting a training cycle.
2018-07-16 04:38:30,923 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:38:33,700 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:38:33,701 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:38:34,454 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:38:34,455 INFO - 2018-07-16 11:38:34.453589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:38:34,455 INFO - 2018-07-16 11:38:34.453741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:38:34,455 INFO - 2018-07-16 11:38:34.453785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:38:34,456 INFO - 2018-07-16 11:38:34.453823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:38:34,456 INFO - 2018-07-16 11:38:34.454066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:38:34,456 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-10926
2018-07-16 04:38:35,240 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:38:35,274 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:38:40,531 INFO - INFO:tensorflow:Saving checkpoints for 10927 into /output/model.ckpt.
2018-07-16 04:38:41,478 INFO - INFO:tensorflow:loss = 0.024905253, step = 10926
2018-07-16 04:39:20,454 INFO - INFO:tensorflow:global_step/sec: 2.56562
2018-07-16 04:39:20,455 INFO - INFO:tensorflow:loss = 0.028681522, step = 11026 (38.978 sec)
2018-07-16 04:39:59,076 INFO - INFO:tensorflow:global_step/sec: 2.5892
2018-07-16 04:39:59,077 INFO - INFO:tensorflow:loss = 0.021933604, step = 11126 (38.623 sec)
2018-07-16 04:40:37,686 INFO - INFO:tensorflow:global_step/sec: 2.59003
2018-07-16 04:40:37,686 INFO - INFO:tensorflow:loss = 0.017831538, step = 11226 (38.609 sec)
2018-07-16 04:41:16,314 INFO - INFO:tensorflow:global_step/sec: 2.58879
2018-07-16 04:41:16,315 INFO - INFO:tensorflow:loss = 0.019387182, step = 11326 (38.628 sec)
2018-07-16 04:41:54,950 INFO - INFO:tensorflow:global_step/sec: 2.58821
2018-07-16 04:41:54,952 INFO - INFO:tensorflow:loss = 0.02261607, step = 11426 (38.638 sec)
2018-07-16 04:42:33,588 INFO - INFO:tensorflow:global_step/sec: 2.58817
2018-07-16 04:42:33,589 INFO - INFO:tensorflow:loss = 0.026149068, step = 11526 (38.637 sec)
2018-07-16 04:42:35,821 INFO - INFO:tensorflow:Saving checkpoints for 11533 into /output/model.ckpt.
2018-07-16 04:42:36,583 INFO - Starting to evaluate.
2018-07-16 04:42:36,584 INFO - INFO:tensorflow:Loss for final step: 0.019252634.
2018-07-16 04:42:37,090 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:42:40,059 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:42:40,082 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:42:40
2018-07-16 04:42:40,852 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:42:40,852 INFO - 2018-07-16 11:42:40.850624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:42:40,852 INFO - 2018-07-16 11:42:40.850758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:42:40,853 INFO - 2018-07-16 11:42:40.850799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:42:40,853 INFO - 2018-07-16 11:42:40.850826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:42:40,853 INFO - 2018-07-16 11:42:40.851001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:42:40,853 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-11533
2018-07-16 04:42:41,861 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:42:41,906 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:44:00,222 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:44:00
2018-07-16 04:44:00,222 INFO - INFO:tensorflow:Saving dict for global step 11533: global_step = 11533, loss = 0.0141015705
2018-07-16 04:44:00,223 INFO - {'loss': 0.0141015705, 'global_step': 11533}
2018-07-16 04:44:00,223 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0627b5a110>]
2018-07-16 04:44:00,223 INFO - Starting a training cycle.
2018-07-16 04:44:00,837 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:44:03,524 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:44:03,525 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:44:04,275 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:44:04,276 INFO - 2018-07-16 11:44:04.273953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:44:04,276 INFO - 2018-07-16 11:44:04.274130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:44:04,277 INFO - 2018-07-16 11:44:04.274176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:44:04,277 INFO - 2018-07-16 11:44:04.274195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:44:04,277 INFO - 2018-07-16 11:44:04.274374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:44:04,277 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-11533
2018-07-16 04:44:05,066 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:44:05,099 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:44:10,302 INFO - INFO:tensorflow:Saving checkpoints for 11534 into /output/model.ckpt.
2018-07-16 04:44:11,310 INFO - INFO:tensorflow:loss = 0.023519054, step = 11533
2018-07-16 04:44:50,312 INFO - INFO:tensorflow:global_step/sec: 2.56398
2018-07-16 04:44:50,314 INFO - INFO:tensorflow:loss = 0.029134292, step = 11633 (39.004 sec)
2018-07-16 04:45:28,894 INFO - INFO:tensorflow:global_step/sec: 2.59184
2018-07-16 04:45:28,896 INFO - INFO:tensorflow:loss = 0.021380477, step = 11733 (38.583 sec)
2018-07-16 04:46:07,531 INFO - INFO:tensorflow:global_step/sec: 2.58823
2018-07-16 04:46:07,532 INFO - INFO:tensorflow:loss = 0.017473806, step = 11833 (38.635 sec)
2018-07-16 04:46:46,170 INFO - INFO:tensorflow:global_step/sec: 2.58808
2018-07-16 04:46:46,170 INFO - INFO:tensorflow:loss = 0.03073866, step = 11933 (38.639 sec)
2018-07-16 04:47:24,809 INFO - INFO:tensorflow:global_step/sec: 2.58805
2018-07-16 04:47:24,810 INFO - INFO:tensorflow:loss = 0.021808108, step = 12033 (38.640 sec)
2018-07-16 04:48:03,443 INFO - INFO:tensorflow:global_step/sec: 2.58838
2018-07-16 04:48:03,444 INFO - INFO:tensorflow:loss = 0.024840942, step = 12133 (38.634 sec)
2018-07-16 04:48:05,678 INFO - INFO:tensorflow:Saving checkpoints for 12140 into /output/model.ckpt.
2018-07-16 04:48:06,618 INFO - Starting to evaluate.
2018-07-16 04:48:06,620 INFO - INFO:tensorflow:Loss for final step: 0.030920716.
2018-07-16 04:48:07,125 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:48:09,819 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:48:09,843 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:48:09
2018-07-16 04:48:10,774 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:48:10,775 INFO - 2018-07-16 11:48:10.772743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:48:10,776 INFO - 2018-07-16 11:48:10.772889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:48:10,776 INFO - 2018-07-16 11:48:10.772957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:48:10,776 INFO - 2018-07-16 11:48:10.773020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:48:10,777 INFO - 2018-07-16 11:48:10.773253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:48:10,777 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12140
2018-07-16 04:48:11,723 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:48:11,760 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:49:29,648 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:49:29
2018-07-16 04:49:29,648 INFO - INFO:tensorflow:Saving dict for global step 12140: global_step = 12140, loss = 0.01382733
2018-07-16 04:49:29,649 INFO - {'loss': 0.01382733, 'global_step': 12140}
2018-07-16 04:49:29,649 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0628b07590>]
2018-07-16 04:49:29,649 INFO - Starting a training cycle.
2018-07-16 04:49:30,271 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:49:33,074 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:49:33,075 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:49:33,667 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:49:33,668 INFO - 2018-07-16 11:49:33.665658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:49:33,668 INFO - 2018-07-16 11:49:33.665824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:49:33,668 INFO - 2018-07-16 11:49:33.665880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:49:33,668 INFO - 2018-07-16 11:49:33.665942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:49:33,668 INFO - 2018-07-16 11:49:33.666157: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:49:33,668 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12140
2018-07-16 04:49:34,444 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:49:34,477 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:49:39,625 INFO - INFO:tensorflow:Saving checkpoints for 12141 into /output/model.ckpt.
2018-07-16 04:49:40,645 INFO - INFO:tensorflow:loss = 0.021711044, step = 12140
2018-07-16 04:50:19,663 INFO - INFO:tensorflow:global_step/sec: 2.56294
2018-07-16 04:50:19,664 INFO - INFO:tensorflow:loss = 0.02386149, step = 12240 (39.019 sec)
2018-07-16 04:50:58,250 INFO - INFO:tensorflow:global_step/sec: 2.59154
2018-07-16 04:50:58,250 INFO - INFO:tensorflow:loss = 0.025855385, step = 12340 (38.586 sec)
2018-07-16 04:51:36,874 INFO - INFO:tensorflow:global_step/sec: 2.58904
2018-07-16 04:51:36,876 INFO - INFO:tensorflow:loss = 0.025936551, step = 12440 (38.624 sec)
2018-07-16 04:52:15,535 INFO - INFO:tensorflow:global_step/sec: 2.58657
2018-07-16 04:52:15,537 INFO - INFO:tensorflow:loss = 0.027927425, step = 12540 (38.662 sec)
2018-07-16 04:52:54,177 INFO - INFO:tensorflow:global_step/sec: 2.58788
2018-07-16 04:52:54,178 INFO - INFO:tensorflow:loss = 0.02240007, step = 12640 (38.641 sec)
2018-07-16 04:53:32,803 INFO - INFO:tensorflow:global_step/sec: 2.58896
2018-07-16 04:53:32,803 INFO - INFO:tensorflow:loss = 0.025927238, step = 12740 (38.626 sec)
2018-07-16 04:53:35,038 INFO - INFO:tensorflow:Saving checkpoints for 12747 into /output/model.ckpt.
2018-07-16 04:53:35,838 INFO - INFO:tensorflow:Loss for final step: 0.016961979.
2018-07-16 04:53:35,838 INFO - Starting to evaluate.
2018-07-16 04:53:36,367 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:53:39,545 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:53:39,574 INFO - INFO:tensorflow:Starting evaluation at 2018-07-16-11:53:39
2018-07-16 04:53:40,261 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:53:40,262 INFO - 2018-07-16 11:53:40.260534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:53:40,262 INFO - 2018-07-16 11:53:40.260653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:53:40,263 INFO - 2018-07-16 11:53:40.260674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:53:40,263 INFO - 2018-07-16 11:53:40.260695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:53:40,263 INFO - 2018-07-16 11:53:40.260868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:53:40,263 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12747
2018-07-16 04:53:41,188 INFO - INFO:tensorflow:Running local_init_op.
2018-07-16 04:53:41,269 INFO - INFO:tensorflow:Done running local_init_op.
2018-07-16 04:54:59,398 INFO - INFO:tensorflow:Finished evaluation at 2018-07-16-11:54:59
2018-07-16 04:54:59,398 INFO - INFO:tensorflow:Saving dict for global step 12747: global_step = 12747, loss = 0.014075892
2018-07-16 04:54:59,399 INFO - {'loss': 0.014075892, 'global_step': 12747}
2018-07-16 04:54:59,399 INFO - [<tensorflow.python.training.basic_session_run_hooks.LoggingTensorHook object at 0x7f0625fd2c10>]
2018-07-16 04:54:59,399 INFO - Starting a training cycle.
2018-07-16 04:55:00,001 INFO - INFO:tensorflow:Calling model_fn.
2018-07-16 04:55:02,829 INFO - INFO:tensorflow:Done calling model_fn.
2018-07-16 04:55:02,830 INFO - INFO:tensorflow:Create CheckpointSaverHook.
2018-07-16 04:55:03,423 INFO - INFO:tensorflow:Graph was finalized.
2018-07-16 04:55:03,424 INFO - 2018-07-16 11:55:03.421917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-16 04:55:03,424 INFO - 2018-07-16 11:55:03.422053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-16 04:55:03,424 INFO - 2018-07-16 11:55:03.422096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-07-16 04:55:03,424 INFO - 2018-07-16 11:55:03.422117: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-07-16 04:55:03,424 INFO - 2018-07-16 11:55:03.422290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10652 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
2018-07-16 04:55:03,425 INFO - INFO:tensorflow:Restoring parameters from /output/model.ckpt-12747
2018-07-16 04:55:04,363 INFO - INFO:tensorflow:Running local_init_op.