ControlNet Deployment

A FastAPI-based deployment of ControlNet for edge detection and image generation.

Overview

This project provides a containerized deployment of ControlNet with the following key components:

• FastAPI Backend: Serves the model via REST API endpoints.
• Model Serialization & Deployment: Uses a pre-trained ControlNet model for synthetic image generation.
• Dynamic File Management: Utility functions select the correct input/output paths for Docker or local environments.
• Docker Containerization: Supports GPU acceleration using NVIDIA Container Toolkit.

Architecture

1. API Endpoints

   • /generate: Accepts an image file and form-based generation parameters. Returns a concatenated image (control + generated).
   • /generate_from_config: Accepts a JSON configuration file (located in inputs/configs/) defining the image path and generation parameters.

2. Model Wrapper

   • The model is loaded using a custom wrapper (ControlNetModel) that performs pre- and post-processing, including edge detection via Canny.

3. Dynamic File Handling

   • Helper functions (get_inputs_dir and get_outputs_dir) detect whether the application is running inside Docker or locally, and adjust paths accordingly.

4. Deployment

   • The project is containerized in Docker and can utilize NVIDIA GPUs.

System Requirements

Tested configuration:

• Intel core i5-6500 (3.2 GHz)

• 32 GB RAM

• NVIDIA GPU (NVIDIA GeForce GTX 1050 Ti): 4GB VRAM

• Docker with NVIDIA Container Toolkit

• Python 3.8

• CUDA 11.8

• PyTorch 2.1.0

Project Structure

controlnet-deployment/
├── app/                   # FastAPI application
│   ├── main.py            # API endpoints and middleware
│   └── model.py           # ControlNet model wrapper and inference code
├── annotator/             # Edge detection modules
├── cldm/                  # ControlNet model architecture
├── ldm/                   # Latent diffusion modules
├── models/                # Model checkpoints & configuration files
├── inputs/                # Input images and configuration JSONs
│   ├── images/
│   └── configs/
└── outputs/               # Generated images

Installation

1. Clone the repository:

   git clone https://github.com/rajanish4/controlnet_deployment.git
   cd controlnet-deployment

2. Download the model:

   • Download the ControlNet model from Hugging Face
   • Place the downloaded control_sd15_canny.pth file in the models/ directory

Deployment

Using Docker (Recommended)

1. Build and run using docker-compose:

   docker-compose up --build
   

The GUI API will be available at http://localhost:8000/docs Images can be generated by uploading an image and providing prompt parameters manually or by uploading a config file (stored in inputs/configs/parameters.json, ps: just enter parameters.json as the config file name, or add your own json files inside this folder).

Local Python Installation

1. Install Miniconda (if needed):
   Miniconda Installation Instructions

2. Create and Activate the Environment:

   conda env create -f environment.yaml
   conda activate control_fastapi

3. Run the FastAPI Server:

   python run.py

   • The API will be available at: http://localhost:8000/docs

API Usage

1. Direct Generation

Endpoint: POST /generate
Parameters:

• file: Image file to upload
• Generation parameters, such as prompt, ddim_steps, etc.

Example using api_test.py:

tester = ControlNetAPITester()
tester.test_generate_endpoint('inputs/images/mri_brain.jpg')

2. Config-based Generation

Endpoint: POST /generate_from_config
Parameters:

• config_file.json: Name of a JSON configuration file (located in inputs/configs/)

Sample config (inputs/configs/parameters.json):

{
  "image_path": "images/mri_brain.jpg",
  "params": {
    "prompt": "MRI brain scan",
    "a_prompt": "good quality",
    "n_prompt": "animal, drawing, painting, vivid colors, longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality",
    "num_samples": 1,
    "image_resolution": 128,
    "ddim_steps": 10,
    "guess_mode": false,
    "strength": 1.0,
    "scale": 9.0,
    "seed": 1,
    "eta": 0.0,
    "low_threshold": 50,
    "high_threshold": 100
  }
}

Example using api_test.py:

tester.test_generate_from_config('parameters.json')

Testing

Run the test script (make sure the server is already running):

python api_test.py

This script tests both endpoints and saves generated images locally.

Generation Parameters

The API accepts the following parameters (for form data):

• prompt: Text description for image generation.
• a_prompt: Additional positive prompt (default: "good quality").
• n_prompt: Negative prompt (default: see sample in config).
• num_samples: Number of images to generate (default: 1).
• image_resolution: Output resolution in pixels (default: 128).
• ddim_steps: Number of denoising steps (default: 10).
• guess_mode: Boolean flag enabling guess mode (default: false).
• strength: Strength parameter for image conditioning (default: 1.0).
• scale: Guidance scale for conditional generation (default: 9.0).
• seed: Random seed (default: 1).
• eta: DDIM sampling eta value (default: 0.0).
• low_threshold: Low threshold value for Canny edge detection (default: 50).
• high_threshold: High threshold value for Canny edge detection (default: 100).

Development Notes

• Model weights should be placed in the models/ directory
• Input images go in inputs/images/
• Configuration json must match the parameters in the main.py file and go in inputs/configs/
• Generated images are saved to outputs/