nftables meta mark for accepting forwarding packets

[CrimsonFez]

Hello,
I'm trying to build a linux based router that has podman for running basic containers. I'm running podman 5.0.1 with nftables support enabled in netavark for my testing. The issue I'm running into is I want to set my forward policy to drop. But this drops packets for podman.
Here is a basic firewall rule I'm trying to implement,

table inet filter {
	chain forward {
		type filter hook forward priority 0; policy drop;
		iifname lan oifname wan accept;
	}
}

With this configured, packets are not forwarded out of my containers.

I'm thinking that if we have netavark mark packets, then I can configure my rules to allow them easily.
For example,

table inet filter {
	chain forward {
		type filter hook forward priority 0; policy drop;
		iifname lan oifname wan accept;
		meta mark <netavark_mark> accept;
	}
}

Is this something that we can do? Do you guys have any suggestions as to how I can handle this without needing to add this feature to netavark?

Thanks in advance!

[robertgzr]

same issue here but for input. i currently don't see a way to have the default verdict set to drop without manually filtering out podman traffic again:

table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;
                ip daddr 10.89.0.0/24 ct state established,related accept
                ip saddr 10.89.0.0/24 accept
                # my other rules here...
        }
}

[X-dark]

Hi,

I got it working adding iif podman0 accept but I am wondering as well if there is a better way (especially as interface podman0 is often not yet created when nft default rules are loaded).

[dsedivec]

@CrimsonFez any reason you don't want to just iifname podman0 accept? Do you have a lot of different Podman networks with names chosen at runtime?

@X-dark try using iifname instead of iif.

[CrimsonFez]

@dsedivec that is the exact reason. I would like to let podman manage it's own firewall rules instead of adding each bridge manually. Personally, I'm fine with just using one bridge, but when it comes to podman/docker compose, dynamically created bridges working out of the box would be nice.

[dsedivec]

@CrimsonFez makes sense. I don't know of any other way to do this. A crazy person might choose to explicitly name all Podman network devices and then statically assign all IPs, and then you could simply name those known values in rules.

Marks feel very ad hoc, especially if Netavark hard codes a particular mark value that the user can't change, but even that would be better than nothing. (Nothing in my servers has stepped on 0x2000 yet.)

To be clear, I would love to see Netavark adding marks on the exact host+port combinations that are valid for forwarding. This actually doesn't look super-difficult to add in nft.rs, but I feel like this is the kind of thing that people would demand to be optional, as they wouldn't want additional rules running for each packet? And once you make it optional, now I presumably have to plumb it through at least Podman as well?

[Luap99]

I would not be to worried about one extra rule. I also rather avoid making this optional, optional always means double the testing matrix, more difficult to debug user issues as you always need to know if they have this on/off and it would require extra documentation.

So if anyone wants to contribute a rule to mark all packages I am fine with merging that
cc @mheon

[mheon]

It might be better to do a user chain (similar to Docker) where all our traffic is guaranteed to pass through. We do use packet marking elsewhere in the nftables code - so we could do marking for outgoing traffic and not interfere, but if we wanted to do incoming traffic we couldn't use a mark and would have to use a chain instead if users wanted to inspect and act on inbound packets. I'd prefer to do both inbound and outbound as chains, rather than have inbound require a chain for inspection but outbound just mark the packets.

[dsedivec]

I would not be to worried about one extra rule. I also rather avoid making this optional

Oh, good, I feel the same way, for all the reasons you cite and more.

It might be better to do a user chain (similar to Docker) where all our traffic is guaranteed to pass through. We do use packet marking elsewhere in the nftables code - so we could do marking for outgoing traffic and not interfere, but if we wanted to do incoming traffic we couldn't use a mark

It seems like Netavark could probably put a oif <bridge name> ip daddr <container IP> udp dport <port> ct mark set 0x2000 accept for each port in a forward chain, then have the ct state established,related accept rule include meta mark set ct mark as well?

I bet there's some constraint I'm missing?

I think something like the Docker user chain wouldn't really change things for me, since I am angling to check that the inbound packet is going to exactly the IP and port of the container, and I have a whole parallel table of firewall rules (so as not to interfere with Netavark's or anyone else's). Netavark is in the best position to mark packets that have the exact expected container IP and port for containers running under it.

[X-dark]

@X-dark try using iifname instead of iif.

Thanks, this is what I needed (and make sense reading nft doc).

[apollo13]

Does anyone have an example setup of a safe configuration where input and forwards are dropped unless explicitly allowed that we can use as a base for the discussion?

[apollo13]

@X-dark:

I got it working adding iif podman0 accept but I am wondering as well if there is a better way (especially as interface podman0 is often not yet created when nft default rules are loaded).

I think I found a better solution. Initial testing seems to suggest (at least that works on my box so far) that the following works:

table inet firewall {
        chain FORWARD {
                type filter hook forward priority filter + 1; policy drop;
                ct status dnat accept; # Accept incoming forwards that went through dnat
                ct state new iifname "podman*" accept; # allow outgoing traffic from "podman*"
                ct state established,related accept; # reallow everything that is established and related, otherwise the final policy would hit.
        }
}

The priority here is filter + 1 so it runs after the rules in the netavark table. The policy is set to drop (jay!) and ct status dnat matches all packets that have been dnatted which happens in the NETAVARK-HOSTPORT-DNAT and subsequent chains.

One could possibly further limit this to the interfaces in question, but that looks relatively nice I think. Any downsides?

EDIT:// Updated it a bit to reflect reality.

What I dislike about my solution is that I have to basically reimplement everything the netavark tables do in the forward chain already.