Fix potential issue with hydit controlnets. #112

Job	Run time
test-unix-nightly (macos, 3.11, 12.1, nightly)	11m 1s
test-stable (macos, 3.9, 12.1, stable)	10m 0s
test-unix-nightly (linux, 3.11, 12.1, nightly)	5m 58s
test-stable (macos, 3.10, 12.1, stable)	10m 4s
test-win-nightly (windows, 3.9, 12.1, nightly)	3m 31s
test-stable (macos, 3.11, 12.1, stable)	9m 57s
test-win-nightly (windows, 3.10, 12.1, nightly)	3m 38s
test-stable (macos, 3.12, 12.1, stable)	9m 40s
test-win-nightly (windows, 3.11, 12.1, nightly)	3m 38s
test-stable (linux, 3.9, 12.1, stable)	6m 3s
test-win-nightly (windows, 3.12, 12.1, nightly)	3m 44s
test-stable (linux, 3.10, 12.1, stable)	6m 1s
test-stable (linux, 3.11, 12.1, stable)	6m 2s
test-stable (linux, 3.12, 12.1, stable)	6m 4s
test-stable (windows, 3.9, 12.1, stable)	3m 29s
test-stable (windows, 3.10, 12.1, stable)	3m 45s
test-stable (windows, 3.11, 12.1, stable)	3m 35s
test-stable (windows, 3.12, 12.1, stable)	3m 36s
	1h 49m 46s

Provide feedback