sd-webui-controlnet Project Detailed Introduction

Project Overview

sd-webui-controlnet is a ControlNet extension plugin developed for AUTOMATIC1111's Stable Diffusion WebUI. Maintained by Mikubill, it is one of the most popular and feature-complete ControlNet WebUI extensions available.

Project Information

• Project Address: https://github.com/Mikubill/sd-webui-controlnet
• Main Maintainer: Mikubill
• Project Type: WebUI Extension Plugin

Core Features

1. ControlNet Integration

• Full support for ControlNet 1.0 and 1.1 versions
• Instantly add ControlNet to the original Stable Diffusion model without merging
• Supports multiple control model types

2. Preprocessor Support

Supports various image preprocessors:

• Canny: Edge Detection
• Depth: Depth Map Generation (including Midas, Depth Anything V2)
• OpenPose: Human Pose Detection
• Scribble: Scribble Control
• Segmentation: Image Segmentation
• Normal: Normal Map
• Lineart: Line Drawing Extraction
• MLSD: Straight Line Detection
• HED: Holistically-Nested Edge Detection
• Reference-only: Reference Image Mode

3. Advanced Features

Multi ControlNet Support

• Use multiple ControlNet inputs simultaneously in a single generation
• Adjustable maximum model count in settings

Control Mode

• Balanced: ControlNet works on both sides of the CFG scale
• My prompt is more important: Gradually reduces the influence of ControlNet
• ControlNet is more important: Works only on the conditional side of CFG

Pixel Perfect Mode

• Automatically calculates the optimal preprocessor resolution
• Ensures each pixel perfectly matches Stable Diffusion

4. T2I-Adapter Support

Full support for TencentARC's T2I-Adapter models, including:

• Color Adapter
• Style Adapter
• Sketch Adapter
• Depth Adapter

5. IP-Adapter Functionality

• Image Prompt Adapter
• CLIP Mask Support
• IP-Adapter Auto Preprocessor
• Advanced Weight Control

Technical Features

Smart Image Resampling

• Outputs two control images during high-resolution fix
• Small image for base generation, large image for high-resolution fix
• Smart algorithm ensures high-quality control image resampling

Mask Integration

• Perfect support for A1111's various mask types
• Supports "Inpaint masked"/"Inpaint not masked"
• Supports "Whole picture"/"Only masked"
• Perfectly matches A1111's resize options

Batch Processing Mode

• Supports batch processing of images
• Specify a batch directory for each unit
• Usable in txt2img and img2img tabs

Installation Method

Via WebUI Installation

1. Open the "Extensions" tab
2. Open the "Install from URL" sub-tab
3. Enter the following in "URL for extension's git repository":

   https://github.com/Mikubill/sd-webui-controlnet.git
   

4. Click the "Install" button
5. Restart WebUI after installation is complete

Model Download

• Visit the official model download page: https://github.com/Mikubill/sd-webui-controlnet/wiki/Model-download
• Place the model files in the stable-diffusion-webui\extensions\sd-webui-controlnet\models directory

API Support

REST API

• Use the --api parameter when starting WebUI
• Visit http://webui-address/docs to view the API documentation
• Supports txt2img and img2img tasks

External Invocation

• Supports other scripts to control this extension
• Requires enabling "Allow other scripts to control this extension" in settings

Command Line Arguments

The extension adds the following command-line arguments:

--controlnet-dir <path>                           # Add ControlNet model directory
--controlnet-annotator-models-path <path>        # Set preprocessor model directory
--no-half-controlnet                             # Load ControlNet models in full precision
--controlnet-preprocessor-cache-size             # Preprocessor result cache size
--controlnet-loglevel                            # ControlNet extension log level
--controlnet-tracemalloc                         # Enable memory allocation tracking

System Requirements

Minimum Configuration

• Windows/Linux/macOS
• NVIDIA GPU (4GB+ VRAM recommended)
• Python 3.8+
• AUTOMATIC1111 WebUI

Performance Optimization

• Use the --xformers parameter to improve performance
• Enable "Low VRAM" mode for devices with low memory
• Supports up to 768x832 resolution (4GB VRAM)

Usage Examples

Basic Usage

1. Enable ControlNet in WebUI
2. Select a preprocessor type (e.g., Canny, Depth, etc.)
3. Upload a control image or let the preprocessor generate it automatically
4. Adjust the weight and guidance strength
5. Generate the image

Multi-ControlNet Usage

1. Increase the maximum model count in settings
2. Enable multiple ControlNet units simultaneously
3. Set different preprocessors and weights for each unit
4. Combine multiple control conditions to generate an image

Summary

sd-webui-controlnet is one of the most important extensions in the Stable Diffusion ecosystem, significantly expanding the controllability and precision of AI image generation. Whether you are a professional artist or an amateur enthusiast, you can achieve more precise image control and richer creative possibilities through this extension.