Importing model and visualizing it with Lucid

[ricardobarroslourenco]

I'm trying to open an autoencoder model I've trained myself, on Lucid, and I'm using as reference the notebook Importing a graph into modelzoo.

I'm mostly in doubt on how to use the provided class:

  model_path = 'nasnet_mobile_graphdef_frozen.pb.modelzoo'
  image_shape = [224, 224, 3]
  image_value_range = (0, 1)
  input_name = 'input' 

What should I define as image_shape, image_value_range? For what images I'm considering this? The output of a certain convolutional layer?

Also, for what is defined the input_name?

[tschwabe]

Hi Ricardo,

the input_name refers to the input-node of your Tensorflow-graph. The image-shape, as far as i know, refers to the size of images your model expects. The image_value_range specifies the values, that are possible for every channel in your Picture. For a typical picture, those are between 0 and 255. However, for a CNN you normalize them to be between -1 and 1 or 0 and 1.

When i tried to import my own model into Lucid, it only worked when i removed all constraints for the expected size of the images, exept the number of channels(3).

[ricardobarroslourenco]

Thanks for the highlight @tschwabe . Another question, if I want to run Lucid on a model with more channels (>3), is it possible to use it? I wonder that perhaps I need to constrain it for three channels due to the RGB mapping.

[tschwabe]

Im not exactly sure. I would recommend to remove the constraints from the height and the width and keep it for the channels, thats what i did.

[ricardobarroslourenco]

First question: When you said to remove width and height constraints, and keep channels, would be something like going from [224, 224, 3] to [:, :, 3]?

Second: I've now tested just passing the model_path and input_name arguments, and got this error:
ValueError: Dimensions must be equal, but are 3 and 7 for 'import/conv2d_1/convolution' (op: 'Conv2D') with input shapes: [1,?,?,3], [3,3,7,16]. It is weird, because I've not specified a image_shape argument.

[tschwabe]

1.) Yeah, [None,None,3], to be exactly. But I removed such constraints inside of my model, not in image_shape !

2.) I would assume this Error arises, just like in my case, because the shapes inside your model are fixed and therefore not aligned with image_shape

[ricardobarroslourenco]

Oh, I see. How did you change/edit your trained model? (I'm new to Tensorflow)

[tschwabe]

I trained my model with fixed size of the pictures. Then i constructed a new model with the same architecture, but no constraints, and loaded the wheights from the other model Into the new one..
I guess there is a prettier way to do it..

[ludwigschubert]

image_shape, image_value_range, and input_name are all required even in theory.

In a nutshell, when lucid imports your model's graph, it needs to replace your model's input (usually a tf.placeholder) with it's own parameterization. To construct this input, it needs to know about the following:

• image_shape so the input lucid creates fits onto your model. (See details after this listing.)
• image_value_range so the input lucid creates provides pixel values similar to those your model was trained on. Different architectures use different pre-processing, so we can not guess. Many architectures normalize to between 0 and 1, but some use -1 to 1, yet others expect 0 to 225.
• input_name is the name of the tensor (usually a tf.placeholder) that is used as an input in your original model. We need to know about it so we know where to feed data into your model. Many architectures simply call this input, but different architectures use different names, so we can not guess.

image_shape has an additional layer of complexity, because we try to allow for flexibility in the input size. In models that pool, smaller inputs lead to smaller spatial dimensions in higher layers—which can lead to shape issues when e.g. your pooling layer has a kernel size that is larger than the remaining spatial dimensions.

As a first step, set image_shape to the input size your original model expects, and use param.image(...) with that same size. This should avoid most shape-related issues.

Also, @ricardobarroslourenco, feel free to create colab notebooks that let me reproduce your issue. I can not promise actual support, but if you can make it easier for me to work on your problem… it increases the probability of me doing just that. Thanks for being an early adopter! :-)

[ludwigschubert]

@tschwabe 's extra step should not be necessary when you tell lucid to use the same input size as you use during training.

[ricardobarroslourenco]

@ludwigschubert thanks for the thorough answer. Let me give you some background on my application.

In my project, I'm building a convolutional autoencoder, but the application is meant for remote sensing images I'm parsing from Google Earth Engine. More specifically I'm ingesting several multispectral images (7 channels, each one associated with a spectral band acquired at the satellite sensor) in which I want to replicate cloud texture patterns. We believe that such cloud class we feed to the autoencoder is too broad, and we expect to have a reasonable amount of variance at the hidden layers, even the most compact ones, even after convergence (by class misclassification of a cloud class, or due to cloud classes that are not described in the present framework, but were enclosed into a hyper class due to lack of a more proper label).

The reason I'm willing to use Lucid is to streamline the process of visualizing the embeddings I'm learning because I will probably drive my network architecture development, and parametrization, by the features I'm able to represent, and their quality. Ideally, this embeddings would show textures present in the samples I'm providing and would be subject to analysis by geophysicists on their feasibility as new cloud classes.

Currently, I've been working on a colab notebook which needs some cleanup, and I'll be glad to share it with you 😃

What google account should I share it to?

[ludwigschubert]

Sounds like an exciting application!

My Google account is [EDIT]

[ricardobarroslourenco]

Thanks @ludwigschubert . I've just shared with you a more clean and commented notebook. Thanks a lot for your availability :)

[ricardobarroslourenco]

@ludwigschubert I would like to know if you were able to run the notebook. I've changed the dataset permissions, so it should be ok.

[hegman12]

@ricardobarroslourenco I am not sure if you have resolved your issues of not able to visualize if your model takes input something like [batch_size,w,h,1]. I was able to visualize by making a small change To this image module

something like this at line number 37

return rgb[...,1]

The resultant image rendered is black and white.

[ricardobarroslourenco]

@hegman12 oh nice. I'll try to change that and see how it goes.

[ebwendol]

Sorry for tacking this on, but a similar question on the subject of image_value_range, based on https://github.com/tensorflow/lucid/blob/master/lucid/modelzoo/vision_models.py .

In the following class, image_value_range is set as image_value_range = (-117, 255-117). I have seen this with InceptionV1 as well as InceptionV3. Does anyone know where this comes from? I don't see these values in the InceptionV1 preprocessing codes.

class InceptionV1(Model):
model_path = 'gs://modelzoo/InceptionV1.pb'
labels_path = 'gs://modelzoo/InceptionV1-labels.txt'
image_shape = [224, 224, 3]
image_value_range = (-117, 255-117)
input_name = 'input:0'

[ludwigschubert]

@ebwendol The specific pre-trained weights we use were trained on that input range. We are not using the weights trained with the slim reimplementation. @colah is currently working on getting more models and more versions of models into modelzoo—check out #85 if you're curious. In particular, this will include slim's InceptionV1, with the value range that you'd expect from a slim model—(-1,1).

[ebwendol]

@ludwigschubert Thank you very much for clarifying this!

[MahrRah]

Hi everyone. I run into the same issues with the input size.
I gave lucid the same input sizes as I used during training but still get the following error:

ValueError: Dimensions must be equal, but are 3 and 1 for 'import/model_2/conv2d_6/convolution' (op: 'Conv2D') with input shapes: [?,?,?,3], [4,4,1,64].

Is there a simpler way to solve it than what to have two models one with fixed and one with unconstrained sizes?

p.s. I am rather new to tf, keras etc.

[colah]

@ricardobarroslourenco @tschwabe @ebwendol @MahrRah @hegman12

We've been working on a new workflow for importing models into Lucid, which aims to make it much easier! We'd love your feedback. You can find code and discussion at PR #152.

[ricardobarroslourenco]

Hi @colah ! Thanks for the heads up. I've just moved back to Brazil (I've left the PhD at UChicago), so I am still accomodating things. I'll let you know once I test it, but this is a nice contribution :)