Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incompatible shapes #4

Open
vijtad opened this issue Feb 9, 2017 · 12 comments
Open

Incompatible shapes #4

vijtad opened this issue Feb 9, 2017 · 12 comments

Comments

@vijtad
Copy link

vijtad commented Feb 9, 2017

I used import_meta_graph and restore and have used my own image.

I am getting this error

Incompatible shapes: [1,30,40,21] vs. [1,18,28,21] at

image_np, pred_np = sess.run([image_tensor, pred], feed_dict=feed_dict_to_use)
@warmspringwinds
Copy link
Owner

Hi,

Please, provide the full code that you run.

@ahundt
Copy link

ahundt commented Feb 9, 2017

Please correct the formatting of your post to be displayed as code:
https://help.github.com/articles/creating-and-highlighting-code-blocks/

@vijtad
Copy link
Author

vijtad commented Feb 10, 2017 •
edited
Loading

I am using Python 3.5 running TF v012 on Windows 10.
I don't see model_fcn8s_final.ckpt in your drop box and used meta data.
Please review my code.

from __future__ import division

import os
import sys
import tensorflow as tf
import skimage.io as io
import numpy as np

sys.path.append("tf-image-segmentation/")
sys.path.append("/home/dpakhom1/workspace/my_models/slim/")

fcn_16s_checkpoint_path = \
 'C:/TensorFlow/checkpoints/vgg_16.ckpt'

os.environ["CUDA_VISIBLE_DEVICES"] = '1'

slim = tf.contrib.slim

from tf_image_segmentation.models.fcn_8s import FCN_8s
from tf_image_segmentation.utils.inference import adapt_network_for_any_size_input
from tf_image_segmentation.utils.pascal_voc import pascal_segmentation_lut

number_of_classes = 21

image_filename = 'C:/Tensorflow/sticker/me.jpg'

#image_filename = 'C:/Tensorflow/sticker/small_cat.jpg'

image_filename_placeholder = tf.placeholder(tf.string)

feed_dict_to_use = {image_filename_placeholder: image_filename}

image_tensor = tf.read_file(image_filename_placeholder)

image_tensor = tf.image.decode_jpeg(image_tensor, channels=3)

# Fake batch for image and annotation by adding
# leading empty axis.
image_batch_tensor = tf.expand_dims(image_tensor, axis=0)

# Be careful: after adaptation, network returns final labels
# and not logits
FCN_8s = adapt_network_for_any_size_input(FCN_8s, 32)


pred, fcn_16s_variables_mapping = FCN_8s(image_batch_tensor=image_batch_tensor,
                                          number_of_classes=number_of_classes,
                                          is_training=False)

# The op for initializing the variables.
initializer = tf.global_variables_initializer()

#saver = tf.train.Saver()
saver = tf.train.import_meta_graph('C:/TensorFlow/checkpoints/fcn_8s_checkpoint/model_fcn8s_final.ckpt.meta', clear_devices=True)

with tf.Session() as sess:
    

    sess.run(initializer)
    #saver.restore(sess,fcn_16s_checkpoint_path)
    saver.restore(sess, "C:/TensorFlow/checkpoints/fcn_8s_checkpoint/model_fcn8s_final.ckpt")
    image_np, pred_np = sess.run([image_tensor, pred], feed_dict=feed_dict_to_use)
    
    io.imshow(image_np)
    io.show()
    
    io.imshow(pred_np.squeeze())
    io.show()

@warmspringwinds
Copy link
Owner

@vijtad , I didn't test this code for python 3 yet.
It makes it hard for me to understand what's the reason for your mistake.

If you solve it, feel free to post your workaround.

@naharsingh
Copy link

Hi @vijtad, @warmspringwinds, i am getting same issue with python 2.7. Would you please suggest solution.

@vijtad
Copy link
Author

vijtad commented Mar 11, 2017

I highlighted my workaround to make it work in Python 3.5 as described in #11
But still segmentation is not perfect.

Waiting for your help on perfect segmentation.

@ahundt
Copy link

ahundt commented Mar 13, 2017

@vijtad Perfect segmentation remains an open research question. In other words, nobody has solved it yet. Here is a leaderboard where you can see what it means to have state of the art results: http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?challengeid=11&compid=6

@sebaschaal
Copy link

I have a similar problem.

I made no changes to your code, but I also get compatibility issues. I narrowed it down to the following:

fcn_8s.py line 131:
fused_last_layer_and_pool4_logits = pool4_logits + last_layer_upsampled_by_factor_2_logits

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,38,38,21] vs. [1,26,26,21]
[[Node: fcn_8s/add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](fcn_8s/pool4_fc/BiasAdd, fcn_8s/conv2d_transpose)]]

Some wiring is not done correctly in my view. I experimented with different input size:

  • 616x616 --> Incompatible shapes: [1,38,38,21] vs. [1,26,26,21]

  • 500x500 --> Incompatible shapes: [1,32,32,21] vs. [1,20,20,21]

  • 598x800 --> Incompatible shapes: [1,38,50,21] vs. [1,26,38,21]

@naharsingh
Copy link

naharsingh commented Mar 24, 2017 via email

@arasharchor
Copy link

arasharchor commented Jul 24, 2017 •
edited
Loading

I have changed nothing in running FCN_32s_train.py except changing image size and number of classes.
image_train_size = [850, 850]
number_of_classes = 2
I had created tfrecord successfully, but after running fcn_32s_train.py I get the following error:
@vijtad @naharsingh @warmspringwinds @sebaschaal

Total memory: 11.90GiB
Free memory: 11.76GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247] PoolAllocator: After 44669 get requests, put_count=44143 evicted_count=1000 eviction_rate=0.0226536 and unsatisfied allocation rate=0.0364011
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259] Raising pool_size_limit_ from 100 to 110
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
  File "fcn_32s_train.py", line 145, in <module>
    train_step ])
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'adam_vars/gradients/GatherNd_1_grad/ScatterNd', defined at:
  File "fcn_32s_train.py", line 100, in <module>
    train_step = tf.train.AdamOptimizer(learning_rate=0.000001).minimize(cross_entropy_sum)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 269, in minimize
    grad_loss=grad_loss)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 335, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 482, in gradients
    in_grads = grad_fn(op, *out_grads)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/array_grad.py", line 354, in _GatherNdGrad
    ref_grad = array_ops.scatter_nd(indices, grad, ref_shape)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2757, in scatter_nd
    shape=shape, name=name)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

...which was originally created as op u'GatherNd_1', defined at:
  File "fcn_32s_train.py", line 82, in <module>
    class_labels=class_labels)
  File "/home/azim_se/projects/laneSegmentation/code/tf_image_segmentation/utils/training.py", line 167, in get_valid_logits_and_labels
    valid_logits_batch_tensor = tf.gather_nd(params=logits_batch_tensor, indices=valid_batch_indices)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1407, in gather_nd
    name=name)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

basically this line shows the error:

InvalidArgumentError (see above for traceback): Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]

@warmspringwinds
Copy link
Owner

warmspringwinds commented Jul 24, 2017 via email

@arasharchor
Copy link

thanks Daniil @warmspringwinds. I am not so familiar with torch. I have worked mostly with tensorflow. That would be super great if you could give me a hint what causes the mentioned error in this library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ahundt @warmspringwinds @vijtad @arasharchor @naharsingh @sebaschaal