Inspirational generation

[JanineCHEN]

Hi @Molugan , thank you for this great repo!

I have trained the stylegan using my own dataset and tried to do the inspirational generation following your instruction. I have conducted
python save_feature_extractor.py vgg19 feature_extractor_345 --layers 3 4 5
And then run
python eval.py inspirational_generation -n stylegan_outdoor -m StyleGAN --input_image fake_data_outdoor/gen_0.jpg -f feature_extractor_345
Yet I got error as following:

0 :  56.091831
Traceback (most recent call last):
  File "eval.py", line 54, in <module>
    out = module.test(parser, visualisation=vis_module)
  File "/models/eval/inspirational_generation.py", line 429, in test
    outPathSave=outPathDescent)
  File "/models/eval/inspirational_generation.py", line 258, in gradientDescentOnInput
    loss.sum(dim=0).backward(retain_graph=retainGraph)
  File "/.virtualenvs/cv/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/.virtualenvs/cv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Besides, there might be a typo in the readme file, as the command should be
--input_image $pathTotheInputImage
instead of
--inputImage $pathTotheInputImage

Any insights on how to solve the RuntimeError issue? Any help would be greatly appreciated!

[JanineCHEN]

Hi, I just solved this issue by amending this line
to
loss.sum(dim=0).backward(retain_graph=True)

[JanineCHEN]

Hi @Molugan , one more question about the functionality of inspirational generation.
I used one of the generated samples as input image, namely the following

And I got this as output image:

Together with this json file:

{
  "gen_0_inspiration": [
    52.50878143310547
  ],
  "Barycenter": {
    "meanDist": 0.0,
    "stdAngles": NaN,
    "meanAngles": 1.0
  },
  "kwargs": {
    "evaluation_name": "inspirational_generation",
    "no_vis": false,
    "np_vis": false,
    "module": "StyleGAN",
    "name": "stylegan_outdoor",
    "dir": "output_networks",
    "iter": null,
    "scale": null,
    "config": null,
    "partition_value": null,
    "statsFile": null,
    "featureExtractor": [
      "feature_extractor_345"
    ],
    "inputImage": "fake_data_outdoor/gen_0.jpg",
    "nRuns": 1,
    "learningRate": 1,
    "suffix": "inspiration",
    "lambdaD": 0.03,
    "nSteps": 6000,
    "weights": null,
    "gradient_descent": false,
    "random_search": false,
    "size": 128,
    "nevergrad": null,
    "save_descent": false
  }
}

and two pt files gen_0_inspirationvector.pt and gen_0_inspirationvectors.pt.

it would be great if you could share more detailed information on this workflow, namely how to interpret the result derived using this function?

[Molugan]

Hello,

Sorry for the delay I do not spend enough time on this repo !

Can you make a PR with your changes ? If not I can manage to push one soon :)

About the workflow.

The inspiration pipeline performs a gradient descent (or a non-gradient optimization if you use the nevergrad methods) from a random latent vector and will try to minimize the following loss:
L = Distance(image target, image generated) + lamda(Realism penalty)

Distance(image target, image generated) is computed using the image features provided by a simple vgg network trained on image net while the "Realism penalty" is the score provided by the discriminator D trained with the generator. More details are available here : https://arxiv.org/pdf/1906.11661.pdf .

The inspiration pipeline can generate several images and the "Barycenter" option here will give you the mean distance between the "mean" results of the pipelines (ie the generated vectors are averaged and a new image is produced from it) and each of the vectors obtained by gradient descent.

In order to get nice looking results, I'd advise to use the nevergrad optimizers they work pretty well (LBFGS typically).