dream-conveyor

This is a framework on top of a framework to fascilitate (and automate) generation of large quantities of ungodly visual horrors and occasionall AI art. Sometimes the latter look more like the former, never vice versa. Use at your own risk.

Rationale

When initially generating AI art using my CPU (I'm cheap), waiting for individual results is time-consuming. A background script based on the diffusers library by Hugging Face would simplify the process. Initially, I wrote custom code for various tasks, but it lacked scalability and reusability. Over time, I identified recurring usage patterns and developed a framework to generate images efficiently in the background. Especially when parameters are changed slightly with each iteration (see Gradients).

DISCLAIMER: This piece of software is a result of me implementing my personal take on Stable Diffusion image generation workflow. If you find it useful — great, otherwise you should probably look elsewhere.

Quickstart

Start by installing poetry globally, it will help you manage the rest of the requirements and create a virtual environment for you.

pip install poetry

Now install the dependencies by running the following in the project folder.

poetry install

After poetry install you will probably be switched to a virtual environment (i.e poetry shell) automatically. If not, run poetry shell.

The next step would be to configure accelerate.

accelerate config

In some more complicated cases that might not be enough, refer to accelerate documentation if it's giving you issues.

Most of this can be run w/o accelerate but it does accelerate things immensely.

Now you're ready to start generating images. Run this command to start:

accelerate launch run.py ./example/

./example/ can be replaced with any folder on your machine. I have several folders on mine that I use for different projects.

When the script is run it will create the folder (if it doesn't exist) and add a configuration file in that folder that will probably look like this:

base_img: null
cfg: 9
mask_img: null
negative: ''
prompt: ''
seed: null
steps: 24
strength: 0.5

You can update this configuration even while the script is running, that's the whole point. If you change any of the parameters, they will take effect for the next image that will start generation.

IP-Adapter reference images

Add one or more image paths to condition generation with an IP-Adapter:

ip_adapter_images:
  - ./references/subject.png
  - ./references/style.png
ip_adapter_type: image
ip_adapter_scale:
  - 0.7
  - 0.3
ip_adapter_model: h94/IP-Adapter
ip_adapter_subfolder: null
ip_adapter_weight: null

ip_adapter_scale should normally be a single number controlling the combined influence of all reference images. For compatibility, a list is accepted and its mean is used. Set ip_adapter_images: null to use Conveyor exactly as before. Conveyor selects the standard SD 1.5 or SDXL checkpoint from the active pipeline. Set ip_adapter_subfolder and ip_adapter_weight explicitly only when using a different compatible adapter.

For stronger SDXL face identity, use tightly cropped, square face references with the face-specific Plus checkpoint:

ip_adapter_images:
  - ./references/face-front.png
  - ./references/face-three-quarter.png
ip_adapter_scale: 0.75
ip_adapter_model: h94/IP-Adapter
ip_adapter_subfolder: sdxl_models
ip_adapter_weight: ip-adapter-plus-face_sdxl_vit-h.safetensors
ip_adapter_image_encoder_folder: models/image_encoder

The face checkpoint uses the ViT-H encoder from models/image_encoder. A scale around 0.7 to 0.9 usually preserves identity more strongly, at the cost of some prompt flexibility.

FaceID

Use FaceID when identity should be preserved without copying the reference image's portrait framing:

ip_adapter_type: faceid
ip_adapter_images:
  - ./references/face-front.png
  - ./references/face-three-quarter.png
ip_adapter_scale: 0.7
ip_adapter_model: h94/IP-Adapter-FaceID
ip_adapter_subfolder: null
ip_adapter_weight: ip-adapter-faceid_sdxl.bin
ip_adapter_image_encoder_folder: null

FaceID currently requires SDXL. Conveyor detects exactly one face in each reference with InsightFace, averages and normalizes their identity embeddings, and caches the result until a reference file changes. Face analysis runs on CPU while diffusion continues on the configured device. If a reference has no face or multiple faces, generation stops with a descriptive error.

I will cover all the implications and various scenarios in series of more in-depth tutorials.

Gradients

TBD

Supported Models

Tested with the following models:

• dreamlike-art/dreamlike-photoreal-2.0
• wavymulder/Analog-Diffusion
• runwayml/stable-diffusion-v1-5
• runwayml/stable-diffusion-inpainting
• stabilityai/stable-diffusion-2
• stabilityai/stable-diffusion-2-inpainting
• darkstorm2150/Protogen_x3.4_Official_Release