Add monitoring phase for NetworkPolicies: log violations instead of dropping via annotation

[dottorblaster]

Is your feature request related to a problem? Please describe.
In the context of the Kubernetes-native network enforcer that me and others are building, we would like to have a monitoring phase for the deployed network policy so the user can verify everything is right before switching to "enforce mode". Policy violations should be logged, but the traffic should still be allowed.

Describe the solution you'd like
From the investigation we conducted:

• use a custom annotation on the k8s network policy to mark the policy as in "monitor" mode. Not ideal but would avoid a new CRD.
• if the annotation is present kube-router could change the iptables rules to just log instead of dropping the traffic. This would mean that the notification in monitor mode would be collected through system logs
• if the annotation is not present the policy is enforced in the cluster.

Describe alternatives you've considered
At first we thought about a dedicated CRD for this, but maybe having just the annotation would be a better fit for the current architecture

Additional context
We already looked into this from a user standpoint: rancher-sandbox/network-enforcer#58

We would still love to hear feedback about anything of this.

[aauren]

I think that this would be an interesting feature.

One nuance, is that it would be using NFLOG targets for the logging as kube-router doesn't have any components in the actual data plane and is purely control plane oriented. That means that users would have to have an NFLOG collector active on their system, not sure how many users have that.

Are you / k3s interested in helping develop this feature? If so, we could probably do some more detailing of the design here and then work towards a PR.

One other thing to note, the netfilter implementation is currently undergoing some pretty extreme changes. There is #2064 that would need to land first, and then after that, the project has ~a month long retrofit that we want to do to remove dependency on iptables across the rest of the controllers and make future implementations more flexible.

We would want to see all of that work land before this feature was worked on.

[dottorblaster]

Thanks for this @aauren!

Yes me/k3s is interested in developing this feature, and I'd like toward a design here and towards a PR later.

About the work in #2064, I think we have some time on our end and we can wait, thanks for letting me know!

About the design, this is what I thought: log instead of drop" only works when a pod is selected only by monitored policies. If a pod is selected by both an enforced and a monitored policy (the common case when validating a new policy on a protected workload), flipping the terminal REJECT drops enforcement for the whole pod.

To keep enforced + "just monitoring" side by side, we could run the monitored evaluation in parallel using a second, staging-only mark bit (e.g. 0x40000, pending a collision audit vs our 0x10000 and kube-proxy's marks, ideally configurable). Monitored policies get KUBE-NWPLCY-MONITORED-<hash> chains identical to the enforced ones except they set the staging bit instead of 0x10000.

Per pod, split target bookkeeping into enforced/monitored sets:

• monitored -> log-only chain, no REJECT, traffic always passes;
• enforced -> unchanged;
• both -> monitored eval first (log-only, always RETURN, touches only 0x40000), then existing enforced eval.

Example of a monitored sub-chain:

-A KUBE-POD-MONITORED-X -j KUBE-NWPLCY-STAGED-<P1>
-A KUBE-POD-MONITORED-X -j KUBE-NWPLCY-STAGED-<P2>
-A KUBE-POD-MONITORED-X -m mark ! --mark 0x20000/0x20000 \
     -m limit --limit 10/min --limit-burst 10 \
     -j NFLOG --nflog-group 101 --nflog-prefix "SNP <policy-hash>"
-A KUBE-POD-MONITORED-X -j MARK --set-xmark 0x0/0x20000
# no REJECT

Annotation: kube-router.io/network-policy.monitor: "true" or Annotation: kube-router.io/network-policy.staged: "true"

Let me know, I'm sure something is off here but I tried my best 😄

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.