Yes, you can absolutely use scrcpy for this without VNC. The trick is simply to use Termux-X11 instead of VNC to get full hardware GPU acceleration. Keyboard, mouse, and audio can work normally via UHID and Scrcpy. The only thing stopping me right now is changing the resolution to 1920x1080 inside Termux-X11.
Yes, you can absolutely use scrcpy for this without VNC. The trick is simply to use Termux-X11 instead of VNC to get full hardware GPU acceleration. Keyboard, mouse, and audio can work normally via UHID and Scrcpy. The only thing stopping me right now is changing the resolution to 1920x1080 inside Termux-X11.