Support HTTP tunneling on all Upstreams by ryanrolds · Pull Request #10712 · solo-io/gloo

ryanrolds · 2025-03-19T21:49:57Z

Description

Refactors the HTTP tunneling plugin to support all Upstreams. I have a PR that confirms a tunnel can be used with a remote JWKS (customer use case). This is Solution 3 in the design document.

While working on tests for the customer use case I discovered that the lifecycle differences with krt require introducing a new plugin interface that was krt-safe. After talking with Yuval, I implemented my understanding of his recommendation. This work should help with other uses of GeneratedResources in gateway2/kgateway as well.

The plugin has two paths that use common logic to transforming and creating the additional clusters/listeners:

Edge uses the now deprecated (not-krt-safe) GeneratedResources
Gateway uses the krt-safe UpstreamGeneratedResources

Tests for the customer case have been added to https://github.com/solo-io/solo-projects/pull/8080.

Last week I added kubernetes/e2e tests that confirm the works in Edge and GW APIs. This work refactors it to work in more cases. My plan is to add additional kubernetes/e2e tests to solo-projects to confirm that the tunneling works for the specific customer case (remote JWKS).

Changes:

Added Gateway test to CI that was missed in an earlier PR
Added ClusterResult (holds the additional clusters/listeners) and plumbed through the gateway2 translator and syncer
Plumbed the reporter through GeneratedResources so that the plugin could report issues with specific Upstreams
Adjusted the tunnel tests to tear down the gateway, forcing the tunnel closed and triggering the logs in Squid

Added golden YAML tests for gateway2 translation.

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works

solo-changelog-bot · 2025-03-19T21:55:10Z

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/7497

github-actions · 2025-03-20T15:27:50Z

Visit the preview URL for this PR (updated for commit 63a5227):

https://gloo-edge--pr10712-rolds-remote-jwks-up-43q77f4s.web.app

_{(expires Wed, 09 Apr 2025 22:24:34 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

…om/solo-io/gloo into rolds/remote_jwks_upstream_tunneling

sam-heilbron

We have an in-memory e2e test: https://github.com/solo-io/gloo/blob/main/test/e2e/dynamic_forward_proxy_test.go. Now that we support a new use case, is it worth it to add a test to prove that case?

projects/gloo/pkg/plugins/plugins.go

projects/gloo/pkg/plugins/tunneling/plugin.go

ryanrolds · 2025-03-24T17:34:55Z

We have an in-memory e2e test: https://github.com/solo-io/gloo/blob/main/test/e2e/dynamic_forward_proxy_test.go. Now that we support a new use case, is it worth it to add a test to prove that case?

Will do.

ryanrolds · 2025-03-24T17:55:03Z

We have an in-memory e2e test: https://github.com/solo-io/gloo/blob/main/test/e2e/dynamic_forward_proxy_test.go. Now that we support a new use case, is it worth it to add a test to prove that case?

I think those tests are unrelated. There are in-memory E2E tests for HTTP tunneling. I understand your ask to be that you would like a non-route bound Upstream to be confirmed to work.

ryanrolds · 2025-03-24T20:28:13Z

@sam-heilbron, I looked at non-EE cases to test and it's pretty much tracing. As noted in the description, I'm working on adding a test for the customer's case. Does this work or would you really like two tests to confirm that routes and non-routed Upstreams work in Gloo even though both use the same code path?

sam-heilbron

LGTM! I like the testing plan you proposed

sam-heilbron · 2025-03-24T20:33:20Z

projects/gloo/pkg/plugins/dynamic_forward_proxy/plugin.go

-func (p *plugin) GeneratedResources(params plugins.Params,
+func (p *plugin) GeneratedResources(
+	params plugins.Params,
+	proxy *v1.Proxy,


I really like this change! I feel like it makes the plugin more self-contained, and a set of resources are associated with the Proxy resource anyway, so I think this makes a lot of sense

projects/gloo/pkg/plugins/tunneling/plugin.go

sam-heilbron · 2025-03-24T20:45:55Z

projects/gloo/pkg/plugins/tunneling/plugin.go

+		cfg, err := utils.NewSslConfigTranslator().ResolveUpstreamSslConfig(params.Snapshot.Secrets,
+			us.GetHttpConnectSslConfig())
+		if err != nil {
+			// if we are configured to warn on missing tls secret and we match that error, add a


I like the idea here, and we may be required to have this logic built into this plugin. What I wish existed, was that we could just compile errors and warnings in reports (and basically not make plugins aware of how they are consumed), and then when we process them, we would inspect the validation settings and processes/consume those reports differently.

https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/translator/reporting.go#L173 is that in practice, https://github.com/solo-io/gloo/blob/main/projects/gloo/pkg/plugins/configuration_error.go.

Do you know if there is a way to ensure that plugins can be unaware of high-level settings like this?

The plugin does need to know about setting as the behavior (not standing-up the tunnel) depends on the setting and error.

It may be possible to push the check down into the Resolve<X>SslConfig methods, but the need to check for the error and branch is still needed. I'm not sure if we want to tightly couple the error for those methods to the ResourceReports.

ryanrolds · 2025-03-27T19:05:55Z

/kick-ci

…10724)

ryanrolds · 2025-03-31T19:36:28Z

I've merged a refactor that should support gateway2. I will be confirming it works with my solo-projects PR once CI finishes.

stevenctl · 2025-04-01T15:11:14Z

projects/gloo/pkg/plugins/tunneling/plugin.go

+	newInClusterName := clusterName + OriginalClusterSuffix
+
+	// use an in-memory pipe to ourselves (only works on linux)
+	forwardingPipe := "@/" + clusterName


Could consider

https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/internal_listener

at some point, not truly proposing this now

I will add a comment nudging future engineers to consider switching to an internal listener.

projects/gloo/pkg/plugins/tunneling/plugin.go

stevenctl · 2025-04-01T16:07:22Z

projects/gloo/pkg/translator/translator.go

-		endpoints = append(endpoints, generatedEndpoints...)
-		routeConfigs = append(routeConfigs, generatedRouteConfigs...)
-		listeners = append(listeners, generatedListeners...)
+		newClusters, newEndpoints, newRouteConfigs, newListeners := plugin.GeneratedResources(params, proxy,


GeneratedResources has a comment about being deprecated, this (the only non-test caller) doesn't have anything like a if (is ggv2) continue? Should we have that? Otherwise maybe we should clarify the deprecation comment.

There are a few other plugins that use GeneratedResources in the gateway2 translation path, so we can't skip it. The check here prevents the HTTP tunnel plugin from double adding the clusters.

When the other plugins are updated, we can pull what you're highlghting out and only use UpstreamGeneratedResources for both the gloo and gateway2 translation paths.

I will add a comment clarifying the intended approach to removing this.

I'm adding:

// GeneratedResources is deprecated and being replaced by UpstreamGeneratedResources // The plan is to move the few remaining plugins in Gloo/EE to use the new // interface. Once that is done and the new interface is called in the Gloo translation // path, we can remove this code. // During this transition period, gateway2 will call both interfaces and it's translator // has (must have) checks that prevent duplicate resources from being added.

stevenctl

It would also be good to have some of the YAML-only goldenfile style tests for this: see ggv2setup_test.go and assoc testdata.

projects/gateway2/proxy_syncer/upstreams.go

ryanrolds · 2025-04-01T17:07:14Z

It would also be good to have some of the YAML-only goldenfile style tests for this: see ggv2setup_test.go and assoc testdata.

Great callout. I will make sure that happens.

…om/solo-io/gloo into rolds/remote_jwks_upstream_tunneling

Partial refactor to support all upstreams

045e7bb

github-actions bot added the keep pr updated label Mar 19, 2025

Adding changelog

dc4b53b

ryanrolds changed the title ~~Support HTTP tunneling on all upstreams~~ Support HTTP tunneling on all Upstreams Mar 19, 2025

changelog-bot added 2 commits March 20, 2025 00:34

Adding changelog file to new location

9783f52

Deleting changelog file from old location

c17c025

ryanrolds marked this pull request as ready for review March 20, 2025 15:15

ryanrolds changed the title ~~Support HTTP tunneling on all Upstreams~~ [DRAFT] Support HTTP tunneling on all Upstreams Mar 20, 2025

ryanrolds added 5 commits March 20, 2025 11:11

Added tests

eebd3f5

Merge branch 'rolds/remote_jwks_upstream_tunneling' of ssh://github.c…

f4c6b05

…om/solo-io/gloo into rolds/remote_jwks_upstream_tunneling

Minor cleanup

b014155

Handle warnings before the error

c32c81c

More tests

6c69923

ryanrolds changed the title ~~[DRAFT] Support HTTP tunneling on all Upstreams~~ Support HTTP tunneling on all Upstreams Mar 21, 2025

ryanrolds added 2 commits March 21, 2025 15:30

Fixing linter errors

aa5fe34

Fixed typo in SNI test

f39aa28

ryanrolds added the work in progress label Mar 24, 2025

ryanrolds added 3 commits March 24, 2025 07:22

Merge branch 'main' into rolds/remote_jwks_upstream_tunneling

18136dd

Added detail to changelog

e6041dc

Merge branch 'rolds/remote_jwks_upstream_tunneling' of ssh://github.c…

3d309a2

…om/solo-io/gloo into rolds/remote_jwks_upstream_tunneling

sam-heilbron reviewed Mar 24, 2025

View reviewed changes

Addressing some feedback

5befaed

ryanrolds requested a review from sam-heilbron March 24, 2025 20:28

sam-heilbron reviewed Mar 24, 2025

View reviewed changes

Merge branch 'main' into rolds/remote_jwks_upstream_tunneling

37a68cf

ryanrolds marked this pull request as draft March 26, 2025 22:34

Adding TestK8sGateway/HTTPTunnel to a test runner

8a3ab8f

ryanrolds changed the title ~~Support HTTP tunneling on all Upstreams~~ [DRAFT] Support HTTP tunneling on all Upstreams Mar 27, 2025

ryanrolds marked this pull request as ready for review March 27, 2025 20:17

[WIP] Implementing a plugin interface for tunneling that is krt-safe (#…

ac6bd9f

…10724)

ryanrolds added 2 commits March 31, 2025 14:29

Improving tests

38a13e7

Fixing flipped apply

88d6b47

ryanrolds changed the title ~~[DRAFT] Support HTTP tunneling on all Upstreams~~ Support HTTP tunneling on all Upstreams Mar 31, 2025

Merge branch 'main' into rolds/remote_jwks_upstream_tunneling

fddbe56

stevenctl reviewed Apr 1, 2025

View reviewed changes

projects/gloo/pkg/plugins/tunneling/plugin.go Show resolved Hide resolved

stevenctl reviewed Apr 1, 2025

View reviewed changes

projects/gateway2/proxy_syncer/upstreams.go Show resolved Hide resolved

ryanrolds added 5 commits April 1, 2025 10:29

Merge branch 'main' into rolds/remote_jwks_upstream_tunneling

6e017f8

Addressing feedback

824df1e

Merge branch 'rolds/remote_jwks_upstream_tunneling' of ssh://github.c…

b9e84f2

…om/solo-io/gloo into rolds/remote_jwks_upstream_tunneling

Wrapped up reporting

39207cb

Adding HTTP tunneling ggv2 golden yaml

c493cde

ryanrolds requested a review from stevenctl April 2, 2025 15:47

ryanrolds added 2 commits April 2, 2025 09:39

Improving ggv2 setup test flakes

9575a46

Increasing timeout

5049c9d

stevenctl approved these changes Apr 2, 2025

View reviewed changes

Merge branch 'main' into rolds/remote_jwks_upstream_tunneling

63a5227

ryanrolds merged commit 078715c into main Apr 3, 2025
20 checks passed

ryanrolds deleted the rolds/remote_jwks_upstream_tunneling branch April 3, 2025 18:02

ymesika pushed a commit that referenced this pull request Apr 23, 2025

reenable TCPRoute e2e test (#10712)

e5535b3

Conversation

ryanrolds commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

solo-changelog-bot bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sam-heilbron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanrolds commented Mar 24, 2025

Uh oh!

ryanrolds commented Mar 24, 2025

Uh oh!

ryanrolds commented Mar 24, 2025

Uh oh!

sam-heilbron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanrolds commented Mar 27, 2025

Uh oh!

ryanrolds commented Mar 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenctl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryanrolds commented Apr 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryanrolds commented Mar 19, 2025 •

edited

Loading

github-actions bot commented Mar 20, 2025 •

edited

Loading