+ inference code

.gitignore

@@ -0,0 +1,2 @@
+.vscode
+**__pycache__

README.md

@@ -0,0 +1,176 @@
+# Multimodal Garment Designer
+### Human-Centric Latent Diffusion Models for Fashion Image Editing
+[**Alberto Baldrati**](https://scholar.google.com/citations?hl=en&user=I1jaZecAAAAJ)**\***,
+[**Davide Morelli**](https://scholar.google.com/citations?user=UJ4D3rYAAAAJ&hl=en)**\***,
+[**Giuseppe Cartella**](https://scholar.google.com/citations?hl=en&user=0sJ4VCcAAAAJ),
+[**Marcella Cornia**](https://scholar.google.com/citations?hl=en&user=DzgmSJEAAAAJ),
+[**Marco Bertini**](https://scholar.google.com/citations?user=SBm9ZpYAAAAJ&hl=en),
+[**Rita Cucchiara**](https://scholar.google.com/citations?hl=en&user=OM3sZEoAAAAJ)
+
+**\*** Equal contribution.
+
+[![arXiv](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)](https://arxiv.org/abs/2304.02051)
+[![GitHub Stars](https://img.shields.io/github/stars/aimagelab/multimodal-garment-designer?style=social)](https://github.com/aimagelab/multimodal-garment-designer)
+
+This is the **official repository** for the [**paper**](https://arxiv.org/abs/2304.02051) "*Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing*".
+
+## Overview
+
+<p align="center">
+    <img src="images/1.gif" style="max-width:500px">
+</p>
+
+>**Abstract**: <br>
+> Fashion illustration is used by designers to communicate their vision and to bring the design idea from conceptualization to realization, showing how clothes interact with the human body. In this context, computer vision can thus be used to improve the fashion design process. Differently from previous works that mainly focused on the virtual try-on of garments, we propose the task of multimodal-conditioned fashion image editing, guiding the generation of human-centric fashion images by following multimodal prompts, such as text, human body poses, and garment sketches. We tackle this problem by proposing a new architecture based on latent diffusion models, an approach that has not been used before in the fashion domain. Given the lack of existing datasets suitable for the task, we also extend two existing fashion datasets, namely Dress Code and VITON-HD, with multimodal annotations collected in a semi-automatic manner. Experimental results on these new datasets demonstrate the effectiveness of our proposal, both in terms of realism and coherence with the given multimodal inputs.
+
+## Citation
+If you make use of our work, please cite our paper:
+
+```bibtex
+@article{baldrati2023multimodal,
+  title={Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing},
+  author={Baldrati, Alberto and Morelli, Davide and Cartella, Giuseppe and Cornia, Marcella and Bertini, Marco and Cucchiara, Rita},
+  journal={arXiv preprint arXiv:2304.02051},
+  year={2023}
+}
+```
+
+## Inference
+
+To run the inference please use the following:
+
+```
+python eval.py --dataset_path <path> --batch_size <int> --mixed_precision fp16 --output_dir <path> --save_name <string> --num_workers_test <int> --sketch_cond_rate 0.2 --dataset <dresscode|vitonhd> --start_cond_rate 0.0
+```
+
+- ```dataset_path``` is the path to the dataset (change accordingly to the dataset parameter)
+- ```dataset``` dataset name to be used
+- ```output_dir``` path to the output directory
+- ```save_name``` name of the output dir subfolder where the generated images are saved
+- ```start_cond_rate``` rate {0.0,1.0} of denoising steps in which sketch cond is applied
+- ```sketch_cond_rate``` rate {0.0,1.0} of denoising steps that will be used as offset to start sketch conditioning
+- ```test_order``` test setting (paired | unpaired)
+
+Note that we provide few sample images to test MGD simply cloning this repo (*i.e.*, assets/data). To execute the code set 
+- Dress Code Multimodal dataset
+    - ```dataset_path``` to ```assets/data/dresscode```
+    - ```dataset``` to ```dresscode```
+- Viton-HD Multimodal dataset
+    - ```dataset_path``` to ```assets/data/vitonhd```
+    - ```dataset``` to ```vitonhd```
+
+It is possible to run the inference on the whole Dress Code Multimodal or Viton-HD Multimodal dataset simply changing the ```dataset_path``` and ```dataset``` according with the downloaded and prepared datasets (see sections below).
+
+
+## Pre-trained models
+The model and checkpoints are available via torch.hub.
+
+Load the MGD denoising UNet model using the following code:
+
+```
+unet = torch.hub.load(
+    dataset=<dataset>, 
+    repo_or_dir='aimagelab/multimodal-garment-designer', 
+    source='github', 
+    model='mgd', 
+    pretrained=True
+    )
+```
+
+- ```dataset``` dataset name (dresscode | vitonhd)
+
+Use the denoising network with our custom diffusers pipeline as follow:
+
+```
+from pipes.sketch_posemap_inpaint_pipe import StableDiffusionSketchPosemapInpaintPipeline
+from diffusers import AutoencoderKL, DDIMScheduler
+from transformers import CLIPTextModel, CLIPTokenizer
+
+pretrained_model_name_or_path = "runwayml/stable-diffusion-inpainting"
+
+text_encoder = CLIPTextModel.from_pretrained(
+    pretrained_model_name_or_path, 
+    subfolder="text_encoder"
+    )
+
+vae = AutoencoderKL.from_pretrained(
+    pretrained_model_name_or_path, 
+    subfolder="vae"
+    )
+
+tokenizer = CLIPTokenizer.from_pretrained(
+    pretrained_model_name_or_path,
+    subfolder="tokenizer",
+    )
+
+val_scheduler = DDIMScheduler.from_pretrained(
+    pretrained_model_name_or_path,
+    subfolder="scheduler"
+    )
+val_scheduler.set_timesteps(50)
+
+val_pipe = ValPipe(
+    text_encoder=text_encoder,
+    vae=vae,
+    unet=unet,
+    tokenizer=tokenizer,
+    scheduler=val_scheduler,
+    )
+```
+
+For an extensive usage case see the file ```eval.py``` in the main repo.
+
+## Datasets
+You can download the Dress Code Multimodal and Viton-HD Multimodal additional data annotations from here.
+
+- Dress Code Multimodal **[[link](https://drive.google.com/file/d/1GABxne7cEHyFgmVoffgssfYKvy91KLos/view?usp=share_link)]**
+- Viton-HD Multimodal **[[link](https://drive.google.com/file/d/1Z2b9YkyBPA_9ZDC54Y5muW9Q8yfAqWSH/view?usp=share_link)]**
+
+### Dress Code Multimodal Data Preparation
+Once data is downloaded prepare the dataset folder as follow:
+
+<pre>
+Dress Code
+| <b>fine_captions.json</b>
+| <b>coarse_captions.json</b>
+| test_pairs_paired.txt
+| test_pairs_unpaired.txt
+| train_pairs.txt
+| <b>test_stitch_map</b>
+|---- [category]
+|-------- images
+|-------- keypoints
+|-------- skeletons
+|-------- dense
+|-------- <b>im_sketch</b>
+|-------- <b>im_sketch_unpaired</b>
+...
+</pre>
+
+### Viton-HD Multimodal Data Preparation
+Once data is downloaded prepare the dataset folder as follow:
+
+<pre>
+Viton-HD
+| <b>captions.json</b>
+|---- Train
+|-------- image
+|-------- cloth
+|-------- image-parse-v3
+|-------- openpose_json
+|-------- <b>im_sketch</b>
+|-------- <b>im_sketch_unpaired</b>
+...
+|---- Test
+...
+|-------- <b>im_sketch</b>
+|-------- <b>im_sketch_unpaired</b>
+...
+</pre>
+
+
+## TODO
+- [ ] training code
+
+## Acknowledgements
+This work has partially been supported by the PNRR project “Future Artificial Intelligence Research (FAIR)”, by the PRIN project “CREATIVE: CRoss-modal understanding and gEnerATIon of Visual and tExtual content” (CUP B87G22000460001), both co-funded by the Italian Ministry of University and Research, and by the European Commission under European Horizon 2020 Programme, grant number 101004545 - ReInHerit.

assets/data/dresscode/dresses/keypoints/051994_2.json

@@ -0,0 +1 @@
+{"keypoints": [[205.0, 0.0, 0.7454180121421814, 0.0], [193.0, 54.0, 0.9695595502853394, 1.0], [153.0, 46.0, 0.9210028052330017, 2.0], [163.0, 133.0, 0.8196305632591248, 3.0], [182.0, 193.0, 0.8251658082008362, 4.0], [234.0, 59.0, 0.9184455871582031, 5.0], [252.0, 133.0, 0.9567239880561829, 6.0], [186.0, 139.0, 0.8934434652328491, 7.0], [180.0, 199.0, 0.803178071975708, 8.0], [164.0, 323.0, 0.9721584916114807, 9.0], [140.0, 438.0, 0.8884657025337219, 10.0], [230.0, 199.0, 0.7910050749778748, 11.0], [223.0, 322.0, 0.8632205128669739, 12.0], [211.0, 436.0, 0.8701340556144714, 13.0], [197.0, 0.0, 0.5216715931892395, 14.0], [214.0, 0.0, 0.610032856464386, 15.0], [186.0, 0.0, 0.5596458911895752, 16.0], [224.0, 0.0, 0.6544457077980042, 17.0]]}

assets/data/dresscode/dresses/keypoints/052012_2.json

@@ -0,0 +1 @@
+{"keypoints": [[209.0, 0.0, 0.5854654312133789, 0.0], [194.0, 46.0, 0.9134835004806519, 1.0], [152.0, 37.0, 0.9346101880073547, 2.0], [137.0, 118.0, 0.8949743509292603, 3.0], [142.0, 188.0, 0.9262046813964844, 4.0], [235.0, 53.0, 0.9443597793579102, 5.0], [242.0, 134.0, 0.8160827159881592, 6.0], [249.0, 208.0, 0.7443605065345764, 7.0], [167.0, 194.0, 0.8199771046638489, 8.0], [160.0, 319.0, 0.907497763633728, 9.0], [143.0, 447.0, 0.8496435284614563, 10.0], [224.0, 195.0, 0.7766034007072449, 11.0], [214.0, 307.0, 0.8796323537826538, 12.0], [195.0, 430.0, 0.9041213393211365, 13.0], [201.0, 0.0, 0.36648842692375183, 14.0], [217.0, 0.0, 0.44042539596557617, 15.0], [187.0, 0.0, 0.40841901302337646, 16.0], [227.0, 0.0, 0.2826925814151764, 17.0]]}

assets/data/dresscode/dresses/test_pairs_paired.txt

@@ -0,0 +1 @@
+052012_0.jpg    052012_1.jpg

assets/data/dresscode/dresses/test_pairs_unpaired.txt

@@ -0,0 +1 @@
+052012_0.jpg    052033_1.jpg

assets/data/dresscode/fine_captions.json

@@ -0,0 +1,2 @@
+{"052033": ["blue belted dress", "light teal blue", "long loose blue dress"], "052896": ["cream dress", "natural sleeveless v-neck dress", "sleevless beige dress"], "049951": ["flowy top", "long sleeved and colored light blue", "sheer blue blouse"], "049534": ["black sweatshirt", "grey crew neck", "long sleeved and dark grey"], "050908": ["skinny denim", "skinny jeans", "skinny mid-rise jeans"], "051078": ["beige shorts", "mid rise shorts", "relaxed fit shorts"], "052012": ["black bell-sleeve mini dress", "black bow detail jersey dress", "long sleeve short black dress"], "051994": ["cream dress", "short cream colored dress", "white short sleeved dress"], "048462": ["black mockneck top", "long sleeved and black color", "long sleeved black top"], "048466": ["short sleeved and picture graphic", "yellow abstract print t-shirt", "yellow graphic tee"], "050855": ["black lexi midi jersey tube skirt", "black maddie jersey midi length tube skirt", "black ttya midi tube skirt"], "050915": ["blue original jeans", "blue straight leg", "blue tapered-leg denim trousers"]
+}

assets/data/dresscode/lower_body/keypoints/050855_2.json

@@ -0,0 +1 @@
+{"keypoints": [[183.0, 0.0, 0.5823147296905518, 0.0], [208.0, 53.0, 0.9465193152427673, 1.0], [166.0, 48.0, 0.9355934262275696, 2.0], [149.0, 138.0, 0.8791153430938721, 3.0], [142.0, 213.0, 0.9196943640708923, 4.0], [249.0, 54.0, 0.9031239151954651, 5.0], [257.0, 143.0, 0.8434740900993347, 6.0], [269.0, 214.0, 0.9049280881881714, 7.0], [181.0, 196.0, 0.7393415570259094, 8.0], [162.0, 321.0, 0.8804817795753479, 9.0], [125.0, 435.0, 0.8968222141265869, 10.0], [235.0, 199.0, 0.7837328314781189, 11.0], [233.0, 322.0, 0.927562415599823, 12.0], [217.0, 439.0, 0.8323052525520325, 13.0], [176.0, 0.0, 0.2889324426651001, 14.0], [190.0, 0.0, 0.395557701587677, 15.0], [-1.0, -1.0, 0.0, -1.0], [213.0, 0.0, 0.4816431403160095, 16.0]]}

assets/data/dresscode/lower_body/keypoints/050915_2.json

@@ -0,0 +1 @@
+{"keypoints": [[204.0, 0.0, 0.7293938398361206, 0.0], [199.0, 62.0, 0.9325352907180786, 1.0], [152.0, 67.0, 0.9000401496887207, 2.0], [144.0, 146.0, 0.9149616956710815, 3.0], [132.0, 213.0, 0.8797067999839783, 4.0], [246.0, 59.0, 0.8685039281845093, 5.0], [279.0, 135.0, 0.9230703115463257, 6.0], [228.0, 163.0, 0.908068835735321, 7.0], [160.0, 218.0, 0.8140541315078735, 8.0], [160.0, 327.0, 0.9113076329231262, 9.0], [162.0, 432.0, 0.8555663228034973, 10.0], [220.0, 218.0, 0.7834039926528931, 11.0], [227.0, 327.0, 0.8965063691139221, 12.0], [225.0, 434.0, 0.876929759979248, 13.0], [190.0, 0.0, 0.510018527507782, 14.0], [211.0, 0.0, 0.48887312412261963, 15.0], [172.0, 0.0, 0.6903271079063416, 16.0], [219.0, 0.0, 0.19232189655303955, 17.0]]}

assets/data/dresscode/lower_body/test_pairs_paired.txt

@@ -0,0 +1,2 @@
+050855_0.jpg    050855_1.jpg
+050915_0.jpg    050915_1.jpg

assets/data/dresscode/lower_body/test_pairs_unpaired.txt

@@ -0,0 +1,2 @@
+050855_0.jpg    050908_1.jpg
+050915_0.jpg    051078_1.jpg

assets/data/dresscode/upper_body/keypoints/048462_2.json

@@ -0,0 +1 @@
+{"keypoints": [[141.0, 0.0, 0.5669732689857483, 0.0], [138.0, 52.0, 0.9372020363807678, 1.0], [97.0, 51.0, 0.9566869735717773, 2.0], [91.0, 127.0, 0.9176412224769592, 3.0], [92.0, 196.0, 0.9124826192855835, 4.0], [175.0, 52.0, 0.9164174795150757, 5.0], [190.0, 126.0, 0.9348884224891663, 6.0], [200.0, 192.0, 0.9007474780082703, 7.0], [121.0, 186.0, 0.8528367280960083, 8.0], [131.0, 298.0, 0.9579567909240723, 9.0], [147.0, 408.0, 0.9156394600868225, 10.0], [171.0, 184.0, 0.8514549732208252, 11.0], [188.0, 299.0, 0.98090660572052, 12.0], [207.0, 421.0, 0.8863993287086487, 13.0], [133.0, 0.0, 0.38882675766944885, 14.0], [148.0, 0.0, 0.43583664298057556, 15.0], [120.0, 0.0, 0.44555413722991943, 16.0], [159.0, 0.0, 0.3003842532634735, 17.0]]}

assets/data/dresscode/upper_body/keypoints/048466_2.json

@@ -0,0 +1 @@
+{"keypoints": [[200.0, 0.0, 0.688823401927948, 0.0], [201.0, 49.0, 0.9461612105369568, 1.0], [162.0, 46.0, 0.9666376709938049, 2.0], [151.0, 113.0, 0.938896894454956, 3.0], [146.0, 177.0, 0.9722222685813904, 4.0], [238.0, 52.0, 0.9425340890884399, 5.0], [247.0, 117.0, 0.9355710744857788, 6.0], [246.0, 180.0, 0.9147509932518005, 7.0], [170.0, 173.0, 0.8584168553352356, 8.0], [159.0, 281.0, 0.938974916934967, 9.0], [156.0, 372.0, 0.9198458790779114, 10.0], [221.0, 173.0, 0.8431934714317322, 11.0], [219.0, 280.0, 0.9141676425933838, 12.0], [214.0, 367.0, 0.944513201713562, 13.0], [192.0, 0.0, 0.4936670660972595, 14.0], [208.0, 0.0, 0.5329393148422241, 15.0], [182.0, 0.0, 0.4606328308582306, 16.0], [219.0, 0.0, 0.460784375667572, 17.0]]}

assets/data/dresscode/upper_body/test_pairs_paired.txt

@@ -0,0 +1 @@
+048462_0.jpg    048462_1.jpg

assets/data/dresscode/upper_body/test_pairs_unpaired.txt

@@ -0,0 +1 @@
+048462_0.jpg    049951_1.jpg

assets/data/vitonhd/captions.json

@@ -0,0 +1 @@
+{"12419": ["black curve cami rib lace", "black dentelle lace-detail camisole", "black bella cami"],  "01944": ["yellow scoop tee", "yellow jersey tee", "gold t-shirt"], "03191": ["white petite t-shirt only macy", "white perforated leather front tee", "white detail tee"], "00349": ["vero moda black high neck blouse", "high-neck top", "black vanessa bruno ath\u00e9 high-neck ruffled blouse"]}

assets/data/vitonhd/test/openpose_json/03191_00_keypoints.json

@@ -0,0 +1 @@
+{"version":1.3,"people":[{"person_id":[-1],"pose_keypoints_2d":[416.932,147.983,0.897289,419.886,323.564,0.823605,295.199,323.575,0.710266,247.078,553.188,0.845261,167.788,760.006,0.839517,547.427,317.9,0.678732,558.827,555.858,0.815472,572.928,779.771,0.816659,397.27,700.433,0.424303,317.926,697.611,0.385856,300.856,969.661,0.23729,0,0,0,485.062,703.303,0.382307,465.203,972.48,0.184967,0,0,0,383.025,122.292,0.896053,442.498,122.248,0.942278,351.819,145.093,0.790069,482.159,136.519,0.671479,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"face_keypoints_2d":[354.446,127.599,0.808671,356.595,142.643,0.809553,360.893,157.687,0.892167,363.042,174.163,0.84444,368.057,189.207,0.79902,378.086,202.818,0.82672,388.831,214.28,0.824691,401.01,224.309,0.842394,416.77,227.175,0.842865,431.097,223.593,0.78027,446.141,213.564,0.798111,457.603,201.386,0.764526,464.767,186.342,0.860417,469.781,169.865,0.836623,470.497,154.105,0.757279,474.796,138.345,0.686917,476.228,122.585,0.674373,361.609,111.123,0.890275,367.34,105.392,0.944614,378.086,103.959,0.885853,387.399,104.676,0.865673,395.995,106.825,0.928337,423.933,106.108,0.876463,433.963,101.094,0.847098,443.992,99.6609,0.891535,455.454,101.094,0.840891,464.05,108.974,0.840359,411.039,122.585,0.865094,411.755,134.047,0.900373,411.755,145.509,0.902167,412.472,156.971,0.886932,401.726,165.567,0.929022,407.457,167.716,0.912199,412.472,168.432,0.922348,418.202,167,0.88593,423.933,164.134,0.89659,372.355,123.301,0.937799,378.086,119.003,0.91301,388.115,119.719,0.852703,395.279,126.883,0.899933,387.399,128.316,0.889112,378.086,128.316,0.975661,428.948,124.018,0.915437,436.112,117.57,0.984511,446.141,116.854,0.92132,452.588,121.868,0.969951,446.857,126.167,0.888569,436.112,126.883,0.902612,393.13,186.342,0.886258,401.01,182.044,0.871408,408.173,179.894,0.916132,413.188,180.611,0.923757,418.919,179.894,0.908463,428.232,180.611,0.889455,436.112,184.909,0.896513,429.664,192.789,0.899923,421.784,197.087,0.877151,413.904,197.804,0.916279,407.457,197.804,0.952279,401.01,194.938,0.878204,395.995,186.342,0.910329,408.173,185.625,0.890515,413.904,185.625,0.906499,419.635,184.909,0.880725,433.246,185.625,0.876117,419.635,189.207,0.837431,413.904,190.64,0.878774,408.173,189.924,0.874297,383.817,122.585,0.863312,441.126,121.152,0.859384],"hand_left_keypoints_2d":[565.693,776.845,0.542976,552.89,801.536,0.678419,545.574,836.287,0.768888,550.146,871.038,0.833503,549.232,893.9,0.865609,580.325,852.748,0.609337,567.522,893.9,0.828608,551.975,914.019,0.795984,537.343,924.078,0.766036,583.983,855.491,0.627158,572.094,892.986,0.789977,552.89,910.361,0.538921,537.343,920.42,0.410428,585.812,850.919,0.656112,574.838,881.097,0.65846,556.548,901.216,0.531315,541.001,910.361,0.395474,581.239,846.346,0.689323,573.923,872.867,0.668061,560.206,880.183,0.523149,545.574,897.558,0.328769],"hand_right_keypoints_2d":[166.902,765.96,0.632211,166.902,790.336,0.735203,165.096,811.102,0.672094,153.36,845.409,0.939393,149.748,871.592,0.619071,132.594,830.061,0.808289,128.983,867.98,0.782519,138.011,891.454,0.756885,145.234,910.414,0.853907,134.4,835.478,0.617637,129.886,871.592,0.758741,143.428,895.968,0.768459,155.165,913.122,0.702863,139.817,837.284,0.584807,136.206,867.078,0.70512,150.651,888.746,0.801908,162.388,904.094,0.809094,149.748,833.673,0.443493,147.04,860.758,0.500639,154.262,875.203,0.701815,165.999,882.426,0.819218],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]}]}