Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

port_forward panics when connection to pod is lost #69

Open
bsamuels453 opened this issue Feb 22, 2024 · 0 comments
Open

port_forward panics when connection to pod is lost #69

bsamuels453 opened this issue Feb 22, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@bsamuels453
Copy link
Collaborator

Getting a panic here that seems to be caused by some sort of race condition:

if err = portForward.ForwardPorts(); err != nil {

No pods were restarted/failed during the connection, so this may be root caused by some tcp error that causes the connection to close. Should be handled by our retry logic, but instead the k8s go client panics.

The following channel is closed when PortForward() is called despite it has already being closed: https://github.com/kubernetes/client-go/blob/8c4efe8d079e405329f314fb789a41ac6af101dc/tools/portforward/portforward.go#L227-L229

What's interesting is the above code is the only place where the channel is ever closed. I suspect what's happening is the portforward object is not re-usable if it fails to connect - we'll have to re-build it for each dial attempt.

Stacktrace:

runtime.fatalpanic (panic.go:1188) runtime
runtime.gopanic (panic.go:1017) runtime
runtime.closechan (chan.go:365) runtime
portforward.(*PortForwarder).forward (portforward.go:228) k8s.io/client-go/tools/portforward
portforward.(*PortForwarder).ForwardPorts (portforward.go:200) k8s.io/client-go/tools/portforward
kubernetes.(*KubeClient).StartPortForwarding.func1 (port_forward.go:120) attacknet/cmd/pkg/kubernetes
kubernetes.(*KubeClient).StartPortForwarding.func3 (port_forward.go:130) attacknet/cmd/pkg/kubernetes
runtime.goexit (asm_arm64.s:1197) runtime
 - Async Stack Trace
kubernetes.(*KubeClient).StartPortForwarding (port_forward.go:118) attacknet/cmd/pkg/kubernetes

The console has the following messages, which may not be relevant but are worth including:

E0222 10:01:08.820313   85675 portforward.go:419] error closing listener: close tcp4 127.0.0.1:57176: use of closed network connection
E0222 10:01:08.820338   85675 portforward.go:419] error closing listener: close tcp6 [::1]:57176: use of closed network connection
bsamuels453 added a commit that referenced this issue Feb 23, 2024
@bsamuels453 bsamuels453 added the bug Something isn't working label Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant