Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Caddy on OPNSense as Layer 4 Proxy for Wireguard #295

Open
pvillmann opened this issue Feb 2, 2025 · 16 comments
Open

Caddy on OPNSense as Layer 4 Proxy for Wireguard #295

pvillmann opened this issue Feb 2, 2025 · 16 comments
Labels
bug Something isn't working

Comments

@pvillmann
Copy link

pvillmann commented Feb 2, 2025 •
edited
Loading

Hi together,

I'm trying to use Caddy as Layer 4 Proxy für Wireguard. (Proxy is reachable via IPv4 and IPv6 and forwards the traffic to my IPv6 CGNAT at home).

Basically everythings works fine as long as no connection is active :).

If i connect my mobile phone to the wireguard Server over Caddy and use the connection (active oder passive) the Caddy service stops after 2 - 5 min.

The only thing i have found in caddy.log is:

panic: send on closed channel

goroutine 76 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c561570, {0x165a45d39190, 0x86c337e00})
        github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 53
        github.com/mholt/[email protected]/layer4/app.go:85 +0x6c5

this is my config:

servers {
		protocols h1 h2
		trusted_proxies static 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 	108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 	172.64.0.0/13 131.0.72.0/22 2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32 2a06:98c0::/29 	2c0f:f248::/32
		listener_wrappers {
			layer4 {
				import /usr/local/etc/caddy/caddy.d/*.layer4listener
			}
			tls
		}
	}

layer4 {
	import /usr/local/etc/caddy/caddy.d/*.layer4global

	udp/:51822 {
		@c0ebf082-5d15-433f-9682-1b34676fa87b wireguard

		route @c0ebf082-5d15-433f-9682-1b34676fa87b {
			proxy udp/XXX:XXX::XXXX:51821 { #IPV6 Adresse :)
			}
		}
	}
}

Thanks :)
@mholt mholt added the bug Something isn't working label Feb 3, 2025
@vnxme
Copy link
Collaborator

vnxme commented Feb 4, 2025

Hi, Please post your full config, including imports. Put debug inside the global config section and post your Caddy log indicating the moment it panics.

@pvillmann
Copy link
Author

Caddyfile:
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file

# caddy_user=root

# Global Options
{
	log {
		include http.log.access.dd57a126-5520-47fb-b0b8-c422338f276c
		include http.log.access.cb08bf6d-bd00-4e35-b233-3a4058b20484
		output net unixgram//var/run/caddy/log.sock {
		}
		format json {
			time_format rfc3339
		}
		level DEBUG
	}

	servers {
		protocols h1 h2
		trusted_proxies static 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22 2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32 2a06:98c0::/29 2c0f:f248::/32
		listener_wrappers {
			layer4 {
				import /usr/local/etc/caddy/caddy.d/*.layer4listener
			}
			tls
		}
	}

	layer4 {
		import /usr/local/etc/caddy/caddy.d/*.layer4global

		udp/:51822 {
			@c0ebf082-5d15-433f-9682-1b34676fa87b wireguard

			route @c0ebf082-5d15-433f-9682-1b34676fa87b {
				proxy udp/[2a00:6020:1000:58::2771]:51821 {
				}
			}
		}
	}

	email [email protected]
	grace_period 10s
	import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Layer4 default HTTP port
:80 {
}
# Layer4 default HTTPS port
:443 {
}

# Reverse Proxy Domain: "dd57a126-5520-47fb-b0b8-c422338f276c"
xxx.xxx.de {
	log dd57a126-5520-47fb-b0b8-c422338f276c
	tls {
		issuer acme {
			dns cloudflare xxx
		}
	}

	handle {
		reverse_proxy https://127.0.0.1:8443 {
			transport http {
				tls_insecure_skip_verify
			}
		}
	}
}
# Reverse Proxy Domain: "cb08bf6d-bd00-4e35-b233-3a4058b20484"
xxx.xxx.xx {
	log cb08bf6d-bd00-4e35-b233-3a4058b20484
	tls {
		issuer acme {
			dns cloudflare xxxx
		}
	}

	handle {
		reverse_proxy https://xxx:5001 {
			transport http {
				tls_insecure_skip_verify
			}
		}
	}
}

import /usr/local/etc/caddy/caddy.d/*.conf

/usr/local/etc/caddy/caddy.d/* --> is complete empty

Log Settings are set to debug already. Here a copy of my logs where you see the problem 2 times with manual restarts. (/var/log/caddy.log)

panic: send on closed channel

goroutine 49 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8700ce7e0, {0x1e69b48e3388, 0x8703a0c60})
        github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
        github.com/mholt/[email protected]/layer4/app.go:85 +0x6c5
{"level":"info","ts":1738528871.584285,"msg":"using config from file","file":"/usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738528871.5849953,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738528871.585096,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738528871.5886347,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1738528871.5900762,"msg":"redirected default logger","from":"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=54330) - Caddy is running in the background
panic: send on closed channel

goroutine 162 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x8702f0af0, {0x21c307329298, 0x8703359a0})
        github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 131
        github.com/mholt/[email protected]/layer4/app.go:85 +0x6c5
{"level":"info","ts":1738529323.0152822,"msg":"using config from file","file":"/usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738529323.0162566,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738529323.0164282,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738529323.0165496,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738529323.0166857,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738529323.022801,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1738529323.024608,"msg":"redirected default logger","from":"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=22966) - Caddy is running in the background
panic: send on closed channel

goroutine 76 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x86c561570, {0x165a45d39190, 0x86c337e00})
        github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 53
        github.com/mholt/[email protected]/layer4/app.go:85 +0x6c5
{"level":"info","ts":1738586314.9854126,"msg":"using config from file","file":"/usr/local/etc/caddy/Caddyfile"}
{"level":"warn","ts":1738586314.9861212,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4listener"}
{"level":"warn","ts":1738586314.9862113,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.layer4global"}
{"level":"warn","ts":1738586314.986279,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.global"}
{"level":"warn","ts":1738586314.986357,"msg":"No files matching import glob pattern","pattern":"/usr/local/etc/caddy/caddy.d/*.conf"}
{"level":"info","ts":1738586314.989946,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1738586314.9914758,"msg":"redirected default logger","from":"stderr","to":"unixgram//var/run/caddy/log.sock"}
Successfully started Caddy (pid=66954) - Caddy is running in the background

@vnxme
Copy link
Collaborator

vnxme commented Feb 4, 2025

Thanks. Well, I don't think this is normal that Caddy panics in the circumstances you described, though the reason is still unclear for me.

According to the log, the panic is triggered by sending a packet to the closed channel here. And the only place I can see where it gets closed is here. We could probably move this line up, so that we close the packets channel after the UDP connection has been removed, but it requires proper testing. @WeidiDeng, what do you think?

@pvillmann, do you use the keepalive option? Are you facing the same issue if you add PersistentKeepalive = 25 for both ends of your WireGuard tunnel?

@pvillmann
Copy link
Author

I'm using keepalive yes, with 30 seconds... i changed it to 25 but i see that whyever the "last handshake" updates only every 2 min plus/minus from both sides.

I don't know if this is a Wireguard / iOS Problem.

@vnxme
Copy link
Collaborator

vnxme commented Feb 4, 2025

i changed it to 25 but i see that whyever the "last handshake" updates only every 2 min plus/minus from both sides.

Handshake update occurs every 2 minutes in WireGuard, it's OK. Keepalives are sent to make NAT/proxies in the middle keep your connections open. With keepalives changed to 25, are you still facing the issue or is it fine now? It's important so that we could understand if any fixes are required to eliminate panics.

@pvillmann
Copy link
Author

my feeling is that it works longer as with 30 seconds but after 20 - 25 minutes the service brokes again.
Last entry is caddy.log:

2025-02-04 09:48:16.338724288 +0100 CET m=+72581.445817463 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.338821291 +0100 CET m=+72581.445914437 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.338825094 +0100 CET m=+72581.445919477 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.337869067 +0100 CET m=+72581.444962246 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.339888257 +0100 CET m=+72581.446981482 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.337957322 +0100 CET m=+72581.445050476 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
2025-02-04 09:48:16.340672668 +0100 CET m=+72581.447765888 write error: write unixgram ->/var/run/caddy/log.sock: write: no buffer space available
panic: send on closed channel

goroutine 12773 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x18c00016d810, {0x30b8ac550dd0, 0x18c000344e00})
        github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
        github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 12753
        github.com/mholt/[email protected]/layer4/app.go:85 +0x6c5

@pvillmann
Copy link
Author

Can I help with addtional Infos ?

@vnxme
Copy link
Collaborator

vnxme commented Feb 5, 2025

Here I've implemented the idea suggested above. You may try it and communicate the results. Please be advised I haven't tested it.

@pvillmann
Copy link
Author

Stupid Question.... any idea how I can replace this field on OPNSense ? :)

@vnxme
Copy link
Collaborator

vnxme commented Feb 5, 2025

on OPNSense

You have to make your own build of Caddy using xcaddy. I would advice you try it locally on Windows/Linux/MacOS before deploying on OPNsense. You may also build Caddy in a Docker container like this.

@pvillmann
Copy link
Author

Okay. i have cloned the your repo and the proxy runs now with xcaddy... I will reach you if the connection brokes.

@pvillmann
Copy link
Author

i tested it 2 times. At both tries the connection brokes after 25-30 min.... sorry

panic: send on closed channel

goroutine 44 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0xc000244c40, {0x7d43d36eb558, 0xc00017fce0})
github.com/mholt/[email protected]/layer4/server.go:158 +0x46f
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
github.com/mholt/[email protected]/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 1
github.com/mholt/[email protected]/layer4/app.go:85 +0x68a
Error: exit status 2
exit status 2

@Monviech
Copy link

Monviech commented Feb 12, 2025 •
edited
Loading

@vnxme

Hey there. I have another report here. I dont know yet if its also with wireguard but I asked for the Caddyfile. The error looks like the same layer4 closed channel error.

https://forum.opnsense.org/index.php?topic=45777.msg229070#msg229070

@vnxme
Copy link
Collaborator

vnxme commented Feb 12, 2025

@Monviech Yes, thanks for reporting. The wireguard matcher is out of scope, since the panic occurs inside the proxy handler. I guess it might be relevant to any UDP traffic under certain circumstances, but WireGuard just triggers that wrong state somehow.

@Monviech
Copy link

@vnxme I think you are right it is out of scope of the Wireguard matcher itself. I got the Caddyfile attached in the linked issue and they use one udp block in the layer 4 global section:

            udp/:3478 {
                        route {
                                    proxy udp/192.168.130.20:3478     {
                                   }
                        }
           }

I have followed up and asked if they can test if the crashes still happen without the UDP section. I'll update here when I know.

@Monviech
Copy link

They reached back out to me and said Caddy is stable since removing the UDP configuration block.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mholt @vnxme @Monviech @pvillmann