segmentation-based controlled image generation

[twinkleyang1]

Thank you for your contributions. Could you please tell me how to do segmentation-based controlled image generation

[jiangzeyinzi]

Please check this #8

[twinkleyang1]

@jiangzeyinzi Thank you for your answer. I made the modification on the yaml file based on canny, but now I have encountered some problems. My data collection format Source: FILE, Target: FILE, Prompt. Source:FILE indicates the original image; Target:FILE indicates the segmentation image; Prompt: Convert to a segmentation map based on the prompt. But the result is similar to Target:FILE. I want to generate images that are similar to Source:FILE and based on the conditions of Target:FILE

[jiangzeyinzi]

Based on your description, I'm still a bit confused about which task you would like to execute:

1. Conditional generation task: Generate an image from a caption + segmentation map.
2. Editing Task A: Generate an image from a segmentation map + natural language instruction.
3. Editing Task B: Generate a segmentation map from an image + natural language instruction.

Our framework can handle all of these tasks, but the methods of use differ.

[twinkleyang1]

@jiangzeyinzi
What I'm going to do is Conditional generation task, Generate an image from prompt+ segmentation map.

1. The data set format is similar to that of scepter_img2img_hed_example.

2. Your description is Source:FILE,Target:FILE,Prompt
   images/src_002.jpeg,images/tar_002.png,Convert to an edge map based on the prompt: A Seamless pattern with
   different cupcakes.
   Where your Source:FILE is the source FILE, Target:FILE is the hed edge processing file,。Your goal is to generate a file
   that is similar to the source file based on the hed boundary

3. My description is Source:FILE,Target:FILE,Prompt
   images/src_R2407(1).png,images/tar_R2407(1).png,Convert to a segmentation map based on the prompt: disk and cup
   segmentation map
   My Source:FILE is a source FILE, and Target:FILE is a segmentation file. My goal is to generate a file similar to the source
   file based on segmentation conditions

[jiangzeyinzi]

The scepter_img2img_hed_example YAML or data format you are using is supported by another work, stylebooth, for image editing tasks.

In SCEdit, we extract image conditions through on-the-fly operation by registering ANNOTATORS, so the inputs are just the prompt and the target image, as shown in the several methods/scedit/ctr/xxx.yaml files you used earlier. The CONTROL_ANNO field in the corresponding YAML file is the operation of the condition we used.

You can achieve the required functionality by registering the ANNOTATORS of segment algo, such as Uniformer, in the modules/annotator directory, then use it in the yaml.