Design for session persistence - Part II

[salonichf5]

Proposed changes

Write a clear and concise description that helps reviewers understand the purpose and impact of your changes. Use the
following format:

Problem: Users want sessionPersistence for their server communication

Solution: Add a detailed approach on how to support sessionPersistence for NGF while being community conformant.

Testing: Did a couple of manual checks

• IP hash working over a set of upstreams and what happens when it fails.
• GRPC connections sending cookie header and responding with correct backend when that cookie is used to send a request again.
• HTTPRoutes verified sessionPersistence working when right cookie sent back in the curl. If it fails, defaults to round robin while generating cookie ID each session.

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #4051

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.06%. Comparing base (8d1b6bd) to head (462e832).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4221      +/-   ##
==========================================
- Coverage   86.08%   86.06%   -0.02%     
==========================================
  Files         131      131              
  Lines       14167    14162       -5     
  Branches       35       35              
==========================================
- Hits        12196    12189       -7     
+ Misses       1770     1768       -2     
- Partials      201      205       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[salonichf5]

I think this is a reasonable first approach for HTTPRoute path handling, but if it’s viewed as over derived then, I’m also comfortable simplifying it to rely on the default cookie path behavior (i.e., not computing a path on the NGF side at all).

[sjberman]

Be sure to fill out the security/testing/use case sections as provided in the proposal template.

[sjberman]

While I like the idea of these being on by default, I'm trying to think of the future implications. If these fields get added to the API spec, then in theory they're going to be disabled by default there. Which means our default behavior is going to change...and that's a breaking change.

[salonichf5]

That’s a fair concern. I wasn’t trying to predefine how secure / httpOnly should behave in the Gateway API, just to harden the default cookie behavior in NGF. But you’re right: if the spec later adds those fields with different defaults, we could box ourselves in.

I’ll update the design to:

• remove secure and httpOnly from the mapping table (so we’re not implying any API-level fields today), and
• call out separately that NGF currently sets secure and httpOnly on the session cookie as an implementation detail, not something controlled by the sessionPersistence spec.

If the community later adds secure / httpOnly fields, we can keep the current behavior when they’re unset (flags stay on by default) and only change behavior when users set them explicitly. That preserves today’s defaults, while still letting users opt out if they really need to.

I do think there’s real security value in having these enabled by default, given the usual session-token hijack and XSS concerns, especially since this directly influences which backend handles the session atleast users will opt into a using a less secure versions themselves

[sjberman]

If the community later adds secure / httpOnly fields, we can keep the current behavior when they’re unset (flags stay on by default) and only change behavior when users set them explicitly

The risk here is that if the API adds these fields and says that their default behavior is "off", then our default behavior doesn't fit that contract. NGINX also defaults these to off.

Another option is set them to on, with the caveat and documentation that this is an experimental API that could evolve and these default values could change. If that ever happens in the future, we would need upgrade docs to help users through that to ensure the new behavior doesn't break them.

The other question I have is that by defaulting them to "on", are we going to prevent any functionality for some users? This is an issue because they can't change the spec until the API exists for it.

[salonichf5]

Now that I’ve thought about it more, I agree that changing them from the usual NGINX defaults does change the “expected” behavior a bit.

I did think about the limiting functionality part:

For secure, the only place it really limits is if a client is talking to the gateway over plain HTTP. In that case nothing actually breaks — requests still go through — but persistence doesn’t work. The browser will accept the Set-Cookie; Secure and then never send it back on http://, so you effectively get a new cookie ID every time and no stickiness.

For httpOnly, all it does is stop JavaScript from reading or modifying the cookie; the server can still set and use it normally. Since this cookie is meant to be opaque and just used by the gateway/NGINX for routing, I don’t really see a normal case where frontend JS should be touching it. The only edge case I can imagine is someone doing something very custom in JS that expects to read that cookie directly. But for grpc connection this could be more limiting since they are expected to be teated as headers and be read.

I did test gPRC over HTTP with these directives and it seemed to be responding correctly but I was injecting the cookie-id

sticky cookie grpc_v1_session expires=1h domain=.bar.com secure httponly;

grpcurl -v -plaintext -proto grpc.proto -authority bar.com -H "cookie: grpc_v1_session=aab8d9166406b4d92a9dffa98eb2a0e2" -d '{"name": "check-upstream"}' localhost:8080 helloworld.Greeter/SayHello

and did have a response from the desired backend when i had last tested.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies

If there other scenarios that you know could cause issues I would love to factor it in.