Lora files not being correctly applied

[TheHaBachi]

I just set up Dream Factory on top of my automatic1111 and noticed that Lora files seem to not be working correctly. Using Auto1111 on the same machine I can use the same prompts and seeds with a lora model and get drastically different results from what dream factory produces. It appears that the loras are not being applied at all?

I have tried inserting them in the prompt in the following ways:
<lora:[filename]:[strength]> ((the way it works in Auto1111 by default & the way it's copied out of the clipboard))
<[filename]:[strength]> ((as suggested in issue #28 ))
lora:[filename]:[strength] ((just to test))

Also noticed that in the gallery when you are reviewing the prompts the lora part is missing from the preview like it has been commented out in some way as well. I have verified that the files & prompts are working correctly in Auto1111 but not for dream factroy, is there a config that needs to be added for dream factory for loras to work properly?

Any help is appreciated, really enjoy being able to use dreamfactory so far

[rbbrdckybk]

I'm using a few LoRAs daily and they're definitely being applied (I run the same prompt/seed with several different loras appended to the end, and they all look radically different in the manner I'd expect to see from each lora). I haven't tested the results against images directly generated with Auto1111; it's possible the API call may be slightly different from native - I'll run a few tests tomorrow.

If you're recently updated Dream Factory, do a hard refresh of your gallery page (control-F5 in Chrome, or clear your browser cache in other browsers). The gallery had a bug where it was treating lora embeddings as HTML and not displaying them, even though they were present in the metadata - it was fixed a few weeks ago; here is an image that just came out of one of my machines; you can see the lora in the prompt:

[rbbrdckybk]

Ok, ran a test today and am getting the same image from Dream Factory as I am from Auto1111 natively when using loras. I'm using --xformers so there may be some very minor differences, but that is expected.

From Dream Factory:

From Auto1111:

Settings:

The easter bunny painting colorful easter eggs, <lora:warcraftTheLora_v10:0.6>
1536x1024 w/ hires fix enabled, scale 8.4, DDIM @ 55 steps, seed 196862028

For reference, this is the image I get if I keep everything the same but remove the lora reference:

It looks like everything is working properly. In Auto1111, do you see your lora files when you click "show extra networks" from the txt2img tab?

[TheHaBachi]

I tried a similar test this afternoon, please let me know if there is something bone headed I've done
Generated an image with Dream factory using the following .prompt file

# *****************************************************************************************************
# Dream Factory random prompt file
# created 2023-03-06 at 17:55:10 via the integrated prompt editor
# *****************************************************************************************************
# these are the default configuration parameters set in your config.txt file
# you may override anything in this section if you wish, otherwise skip down to the [prompts] section(s) below
[config]

!MODE = random							# random mode; queue random prompts from [prompts] sections below
!DELIM = " "							# delimiter to use between prompt sections, default is space
!WIDTH = 512							# output image width
!HEIGHT = 512							# output image height
!HIGHRES_FIX = no				        # fix for images significantly larger than 512x512, if enabled uses !STRENGTH setting
!STEPS = 20							# number of steps, more may improve image but increase generation time
!SAMPLER = Euler a            			# sampler to use; press ctrl+h for reference
!SAMPLES = 1					      	# number of images to generate per prompt
!MIN_SCALE = 7							# minimum guidance scale, default = 7.5
!MAX_SCALE = 7							# maximum guidance scale, set min and max to same number for no variance
!RANDOM_INPUT_IMAGE_DIR =				# specify directory of images here; a random image will be picked per prompt 
!MIN_STRENGTH = 0.75					# min strength of starting image influence, (0-1, 1 is lowest influence)
!MAX_STRENGTH = 0.75					# max strength of start image, set min and max to same number for no variance
!CKPT_FILE = mdjrny-v4.safetensors     # model to load, press ctrl+h for reference

# optional integrated upscaling

!USE_UPSCALE = no						# upscale output images?
!UPSCALE_AMOUNT = 2.0					# upscaling factor
!UPSCALE_CODEFORMER_AMOUNT = 0.0		# how visible codeformer enhancement is, 0-1
!UPSCALE_GFPGAN_AMOUNT = 0.0			# how visible gfpgan enhancement is, 0-1
!UPSCALE_KEEP_ORG = no				# keep the original non-upscaled image (yes/no)?

# optional negative prompt

!NEG_PROMPT = 

# *****************************************************************************************************
# prompt section
# *****************************************************************************************************
[prompts]

The easter bunny painting colorful easter eggs, <lora:rabbitRabbit_v10:0.7>

# put your prompts here; one per line
# you may also add additional [prompt] sections below, see 'example-random.prompts' for details

This is the following image generated

Using the Seed: 4128928803 I generated the image via A1111 using the following prompt

The easter bunny painting colorful easter eggs, <lora:rabbitRabbit_v10:0.7>
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 4128928803, Size: 512x512, Model hash: aba96b389d, Model: mdjrny-v4

Using the same seed in A1111 I generated another imaged with the following prompt

The easter bunny painting colorful easter eggs,
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 4128928803, Size: 512x512, Model hash: aba96b389d, Model: mdjrny-v4

The image generated from Dream factory matches the image from A1111 without the Lora prompt, However after checking for updates (was up to date) and clearing my browser cache I can now see the lora prompt inside dream factory gallery

Did all the image generation after checking for updates and clearing browser cache

Additionally I can see the Lora model inside A1111 when I click show extra networks

Still fairly new to stable diffusion / Automatic1111 etc so there is a potential for configs to be incorrect ect. any guidance on where to check would be appreciated

Below is the confix.txt file from the dream factory dir, the D:\AI\stable-diffusion-webui is actually my install location for Automatic1111

###############################################################################
### Dream Factory default configuration file
###############################################################################

# Path to installed Automatic1111 stable diffusion repo (e.g.: C:\Applications\stable-diffusion-webui).
# You must populate this with a working SD installation for Dream Factory to operate!
SD_LOCATION = D:\AI\stable-diffusion-webui

# Specify which GPU(s) to use here; auto will attempt to use all detected devices
# to specify individual GPUs, use a comma-separated list of device IDs.
# e.g. USE_GPU_DEVICES = 0, 1, 2
USE_GPU_DEVICES = 0, 1

# Starting port that SD instances will use; additional GPUs will each increment this 
# by one when deciding which port to operate on - e.g. if you leave this at 7861 and 
# have 3 GPUs, then ports 7861, 7862, and 7863 will be used by SD instances.
# These are not accessible outside of the local machine Dream Factory operates on and 
# there is generally no reason to change this.
SD_PORT = 7861

# If you have multiple GPUs, this is how many are allowed to initialize at once. 
# Initialization only happens once at startup, but it very memory-intensive (system RAM, 
# not VRAM). Each GPU will use nearly 10GB of system RAM during initialization, so 
# allowing them all to init simultaneously may cause issues depending on system resources.
GPU_INIT_STAGGER = 1

# Directory containing your prompt files.
# Locations are relative to Dream Factory's installation directory unless specified.
PROMPTS_LOCATION = prompts

# Directory containing your wildcard files.
WILDCARD_LOCATION = prompts\wildcards

# Directory where output images will be created.
OUTPUT_LOCATION = output

# Use the webserver (yes/no)? This is required to use the browser interface.
WEBSERVER_USE = yes

# Port the webserver listens on.
# Note that ports under 1024 require root permission on Linux.
WEBSERVER_PORT = 8080

# Set this to yes if you want to be able to access the webserver UI from other computers.
# At the default "no" setting the web UI is inaccessible anywhere except at the local computer running the server!
WEBSERVER_NETWORK_ACCESSIBLE = no

# Enable webserver authentication?
# This is the most basic form of HTTP authentication and should not be considered secure!
# However if you're going to make this accessible via WAN/internet it's better than nothing.
WEBSERVER_USE_AUTHENTICATION = no

# Webserver login credentials; make sure to change these if you're using authentication!
WEBSERVER_AUTH_USERNAME = admin
WEBSERVER_AUTH_PASSWORD = password

# Automatically attempt to open the server interface in your browser (yes/no)?
# This has caused some confusion, but it literally does just that - it'll pop open Chrome 
# or whatever your default browser is, with Dream Factory loaded. Doesn't work all on OSes.
WEBSERVER_OPEN_BROWSER = yes

# Log webserver messages to the console (yes/no)? For debugging, leave it disabled unless you like spam.
WEBSERVER_CONSOLE_LOG = no

# Gallery maximum images to display.
GALLERY_MAX_IMAGES = 200

# Gallery refresh interval in seconds; set to zero to disable auto-refresh of gallery.
GALLERY_REFRESH = 30

# Optional folder to display in the gallery (and alias to refer to it)
GALLERY_USER_FOLDER = 
GALLERY_USER_FOLDER_ALIAS = favorites

# The editor won't attempt to apply context-sensitive color styling to prompt files over this character limit.
# Values above 200000 will significantly increase loading time for prompt files that exceed that character threshold. 
# If you're working with massive .prompt files, it's best to edit them using an external editor.
EDITOR_MAX_STYLING_CHARS = 80000

# You can set your own defaults for prompt file settings below.
# Settings specified in individual prompt files will always override these.

PF_WIDTH = 512             			# output image width, default is 512
PF_HEIGHT = 512            			# output image height, default is 512
PF_HIGHRES_FIX = no					# helps a lot when generating images significantly larger than 512x512 at the cost of speed
PF_SAMPLER = DPM++ 2M Karras		# sampler to use (create a new prompt file using the web UI to see a full list of options)
PF_STEPS = 20              			# number of steps, default = 50
PF_SCALE = 7.5              		# guidance scale, default = 7.5
PF_SAMPLES = 1						# number of images to generate per prompt
PF_USE_UPSCALE = no					# use ESRGAN to upscale output images?
PF_UPSCALE_AMOUNT = 2.0				# upscaling factor (2 means you'll get an image with double the width and height of the original)
PF_UPSCALE_CODEFORMER_AMOUNT = 0.0	# visibility of codeformer face enhancement (0 = off, 1 = full effect)
PF_UPSCALE_GFPGAN_AMOUNT = 0.0		# visibility of gfpgan face enhancement (0 = off, 1 = full effect)
PF_UPSCALE_KEEP_ORG = no			# keep the original non-upscaled image (yes/no)?
PF_NEG_PROMPT = 					# if you have a 'catch all' negative prompt you typically use, enter it here
PF_FILENAME =						# set a default custom filename: https://github.com/rbbrdckybk/dream-factory#filename

# Set a default .ckpt file here so that whenever you set !CKPT_FILE to nothing (e.g. '!CKPT_FILE = '), this default 
# will be loaded. If you don't set anything here, !CKPT_FILE= will just use whatever happens to be loaded in Auto1111.
PF_CKPT_FILE =

# automatically insert the trigger word when using custom models?
# off = disable this
# start = at the start of the prompt (the default)
# end = at the end of the prompt
# first_comma = after the first comma (or end if prompt contains no commas)
# you must edit the model-triggers.txt file (auto-generated after first run) for these to work
PF_AUTO_INSERT_MODEL_TRIGGER = start


# Set the number of prompts to queue at once when using prompt files set to random mode.
# The only real reason you'd want to change this is if you're specifying multiple models (with !CKPT_FILE = model1, model2, etc).
# In random mode the models will rotate every time a batch of prompts is queued, so you can effectively control how often 
# your models will switch by setting this.
RANDOM_QUEUE_SIZE = 50

[rbbrdckybk]

I am stumped - everything looks fine in your config.txt and your prompt file, but to be 100% sure, I copied it and ran it myself. I downloaded the same rabbit LoRA and the midjourney-v4 model as well.

Dream Factory gives me the same image that Auto1111 generated for you (see below), so it seems to be working properly here. Is your Auto1111 repo up to date (from the command line in the Auto1111 base directory, you can type git pull to grab the latest version)? It's possible there is an issue with the Auto1111 API call in older versions, but I'm just guessing at this point.

I tested with --xformers enabled and disabled (I get the same image on both sides regardless), and I don't have anything else in my Auto1111 startup script.

If your Auto1111 is up to date, I'm honestly not sure what else could possibly cause this.

The prompt file I used is an exact copy of yours; I just added the !SEED you used and turned off model trigger insertion:

[TheHaBachi]

I guess I'll reinstall both Auto1111 and Dream factory to see if that resolves my issue, checked Auto1111 directory and it's up to date.

[TheHaBachi]

Uninstalled all remnants of Auto1111 & Dream factory, found some old python 2.x.x installs from old dev work, uninstalled them, uninstalled Anaconda, uninstalled python 3.10.6. Reinstalled everything clean and still experiencing the same problem. Only thing of notice is when running setup.py for the first time received the following error. Any suggestions for remedy or config files or logs I need to provide?
Personally stumped at this, any suggestions would be great :)

[rbbrdckybk]

On the setup.py installation error, you can try re-running with the shell option to fix; depending on your environment it might not be able to find a needed executable (git, etc). If you add the --verbose option it'll echo everything to the screen so you can see any errors that might pop up:

python setup.py --shell --force --verbose

Honestly I'm completely at a loss as to why LoRA embeddings in your Dream Factory prompts don't seem to be having any effect. I have 4 installations running DF (mixed windows/linux environments with various dissimilar hardware) and loras are working in all of them and I haven't had any other reports of loras not working.

Some things you can try, although I'm honestly not sure they'll be all that helpful:

• In your DF config.txt, try enabling console logging for the webserver and see if you see any errors when running prompt files containing loras:

WEBSERVER_CONSOLE_LOG = yes

• Also in your DF config.txt, try switching the Auto1111 port to 7860 (the default that Auto1111 uses when running it normally), then start Auto1111 manually (and wait for it to fully load), and finally start DF. DF will see that Auto1111 is already running on port 7860 and use it, which will allow you to monitor the Auto1111 console to see if there are any errors popping up there:

SD_PORT = 7860

• If you're running any command line arguments in your Auto1111 startup script (webui-user.bat in your Auto1111 root directory for Windows), disable them to see if that makes a difference. I'm running --xformers and --no-half-vae on some of my machines, but there are many options I haven't tested.

[TheHaBachi]

So after a fair amount of testing and a lot of testing later, I seem to have found a solution,
Reference: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7984

added command line arg --api to webui-users.bat
and in webui.py added modules.script_callbacks.before_ui_callback() on line 190 in def api_only()
for whatever reason this has solved my problems.

Small side note, I installed dreamfactory onto a completely new machine and experienced all same problems and resolved them with this same solution.

Really appreciate all the help, if you need any additional information from me please let me know happy to help!

Thanks!

[rbbrdckybk]

Aha - glad you found the issue! Very strange that none of my Auto1111 installations have the issue, but hopefully he'll patch a fix into an upcoming release so others don't experience the problem. At least I'll be able to point others to the resolution if anyone else mentions it - thanks for the info!

Edit: pinning this until the issue is resolved in in the Auto1111 API!

[Koneko349]

@TheHaBachi, looks like this has been fixed if you want to test, its working for me.