netfilter buglog - Apr 2024 - [Bug 1748] New: nft masquerade commands make table nat unreadable by iptables-nft

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

            Bug ID: 1748
           Summary: nft masquerade commands make table nat unreadable by
                    iptables-nft
           Product: iptables
           Version: git (please indicate commit ID)
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: iptables
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: ts at ferncast.de

Hi,

If I inject a masquerade rule into the nat table by nft I cannot read the
table with iptables-nft (git hash 8bf2bab8eb2e4f5ae2fef859ea7c877662854101 and
also older versions like 1.8.9) anymore till I delete this single rule with nft
tool.

E.g. if I put the below configuration into file postrouting.nft and inject it
via `nft -f postrouting.nft`:
table ip nat {
    chain POSTROUTING {
        oifname != "br-mailcow" ip saddr 172.22.1.0/24 masquerade
    }
}

`iptables -t nat -L` than tells me `iptables v1.8.10 (nf_tables): table
`nat'
is incompatible, use 'nft' tool.` If I inject the same rule with
iptables-nft,
everything is working fine. I was able to fix this behavior by this patch to
iptables, but I am pretty sure that this is not the right way to do it:

diff --git a/iptables/nft.c b/iptables/nft.c
index 884cc77e..a7086014 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -3931,6 +3931,7 @@ static const char *supported_exprs[] = {
        "immediate",
        "lookup",
        "range",
+       "masq",
 };

In my further debugging I found that the rule injected by iptables seems to
have "target" as expression instead of "masq", but this only
as a hint for
someone who really knows the code. ;-)

Yesterday morning this "bug" caused a severe problem as libvirt was
not able to
read the nat table anymore because I used some nft commands before to insert
some masquerading rules. This also may affect others who update their libvirt
in, e.g., Ubuntu, and have inserted masquerading rules in the nat table via
`nft`.

Best regards,
Thomas

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/3972b950/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

Phil Sutter <phil at nwl.cc> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |phil at nwl.cc
         Resolution|---                         |INVALID

--- Comment #1 from Phil Sutter <phil at nwl.cc> ---
Hi,

(In reply to Thomas Schlien from comment #0)
> If I inject a masquerade rule into the nat table by nft I cannot read the
> table with iptables-nft (git hash 8bf2bab8eb2e4f5ae2fef859ea7c877662854101
> and also older versions like 1.8.9) anymore till I delete this single rule
> with nft
> tool.
This is expected behaviour. You should not mix iptables-nft and nft tools when
manipulating the ruleset. Doing so is a good way to shoot one's foot (as you
did).
> E.g. if I put the below configuration into file postrouting.nft and inject
it
> via `nft -f postrouting.nft`:
> table ip nat {
>     chain POSTROUTING {
>         oifname != "br-mailcow" ip saddr 172.22.1.0/24 masquerade
>     }
> }
> 
> `iptables -t nat -L` than tells me `iptables v1.8.10 (nf_tables): table
> `nat' is incompatible, use 'nft' tool.` If I inject the same
rule with
> iptables-nft, everything is working fine. I was able to fix this behavior
by
> this patch to iptables, but I am pretty sure that this is not the right way
> to do it:
> 
> diff --git a/iptables/nft.c b/iptables/nft.c
> index 884cc77e..a7086014 100644
> --- a/iptables/nft.c
> +++ b/iptables/nft.c
> @@ -3931,6 +3931,7 @@ static const char *supported_exprs[] = {
>         "immediate",
>         "lookup",
>         "range",
> +       "masq",
>  };
This will merely accept the unsupported expression in the rule but the output
is likely incomplete (or even broken).
> In my further debugging I found that the rule injected by iptables seems to
> have "target" as expression instead of "masq", but this
only as a hint for
> someone who really knows the code. ;-)
For NAT rules, iptables-nft uses xtables extensions (xt_MASQUERADE.ko) while
nft uses its native code (nft_masq.ko). There are "compat" nftables
expressions
named "match" and "target" to call xtables extensions from
an nftables rule.
> Yesterday morning this "bug" caused a severe problem as libvirt
was not able
> to read the nat table anymore because I used some nft commands before to
> insert some masquerading rules. This also may affect others who update
their
> libvirt in, e.g., Ubuntu, and have inserted masquerading rules in the nat
> table via `nft`.
Starting with v1.0.6, nft even warns when listing a ruleset which contains the
compat expressions mentioned above.

Cheers, Phil

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/d1a9c435/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

--- Comment #2 from Thomas Schlien <ts at ferncast.de> ---
Hi Phil,

thanks for the fast answer. I understand now that one should not mix up the
two, but I can give you one example on our system where I didn't even knew
that
a rule was added via nft.

We have a server where some VMs are running in libvirt/QEMU/KVM and some in
docker. One of the docker services is mailcow which itself starts 18 docker
container. One of them is netfilter container which handles all the firewall
rules settings. Unfortunately mailcow checks if nftables is installed on the
host and if so the nft tool is used to handle the necessary rule injection in
the firewall. This way I didn't even noticed that the table was
"poisoned".

Isn't the idea to replace iptables by nftables one day? I think at least a
lot
of people do believe this as the mailcow example shows. Also the description of
nftables on netfilter.org homepage leads to this assumption. But if this is the
case there will be a transition phase where both tools will be used in
parallel. So how should this be handled?

Btw. other rules that I inject, e.g., via `/usr/sbin/nft insert rule filter
LIBVIRT_FWI oifname kvmnat meta l4proto 6 ip daddr ${GUEST_IP} tcp dport
${GUEST_PORT_SSH} accept comment \"Accept SSH traffic\"` are handled
fine by
iptables-nft.

Best regards,
Thomas

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/e2723294/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

--- Comment #3 from Phil Sutter <phil at nwl.cc> ---
Hi Thomas,

(In reply to Thomas Schlien from comment #2)
> thanks for the fast answer. I understand now that one should not mix up the
> two, but I can give you one example on our system where I didn't even
knew
> that a rule was added via nft.
> 
> We have a server where some VMs are running in libvirt/QEMU/KVM and some in
> docker. One of the docker services is mailcow which itself starts 18 docker
> container. One of them is netfilter container which handles all the
firewall
> rules settings. Unfortunately mailcow checks if nftables is installed on
the
> host and if so the nft tool is used to handle the necessary rule injection
> in the firewall. This way I didn't even noticed that the table was
> "poisoned".
Yes, it's kind of a mess. Containers manipulating init netns firewall
ruleset
is really bad design choice: Even when sticking to the same tool, mixing
different versions may lead to problems, too. This is also not a new problem,
it existed with legacy iptables as well. Though in the past it was much less
common to use multiple userspace root fs for privileged operations at the same
time.
> Isn't the idea to replace iptables by nftables one day? I think at
least a
> lot of people do believe this as the mailcow example shows. Also the
> description of nftables on netfilter.org homepage leads to this assumption.
> But if this is the case there will be a transition phase where both tools
> will be used in parallel. So how should this be handled?
Yes, nftables is the designated successor of iptables. Mailcow is merely trying
to get things right as it has a hard time guessing whether iptables-nft or nft
may be used on the host. Though these types of games are expected when doing
such things.

Why should both tools be used in parallel? A transitioning mechanism may be to
pipe one's iptables ruleset dump through iptables-restore-translate and load
the output via nft after a reboot.

iptables-nft is a transitioning mechanism in that it allows one to disable some
legacy kernel modules, for instance. Also it is superior to legacy iptables in
that it doesn't have to depend on filesystem-based locking to avoid
concurrent
kernel ruleset access.

There may be a version of iptables-nft in the future which does not use any
xtables extensions anymore and thus creates a ruleset which nft is able to
parse and recreate entirely. Right now transitioning in that direction is
problematic because it breaks older user space and thus causes more problems to
your use case.

Also there will never be a version of iptables-nft which supports everything
nft is able to emit, so full compatibility is impossible anyway.
> Btw. other rules that I inject, e.g., via `/usr/sbin/nft insert rule filter
> LIBVIRT_FWI oifname kvmnat meta l4proto 6 ip daddr ${GUEST_IP} tcp dport
> ${GUEST_PORT_SSH} accept comment \"Accept SSH traffic\"` are
handled fine by
> iptables-nft.
Yes, in this case iptables-nft is able to parse the rule. If you
dump'n'restore
it (i.e., call iptables-nft-save | iptables-nft-restore), you'll notice it
added a counter to the rule. Also it will convert the native nftables comment
into an xtables one (xt_comment.ko). So the next time you list the same rule
using nft, the comment will be gone.

BTW: 'meta l4proto 6' there is redundant, the 'tcp dport' match
will
automatically add this as a dependency.

Cheers, Phil

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/f6e56500/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

--- Comment #4 from Thomas Schlien <ts at ferncast.de> ---
Hi Phil,

thanks for your detailed answer and comments. Hopefully one day the transition
to nftables will be done and we all can just use the nft tool without problems.
:-)

Best regards,
Thomas

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/2d9d64e8/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1748

--- Comment #5 from Phil Sutter <phil at nwl.cc> ---
Thomas,

(In reply to Thomas Schlien from comment #4)
> thanks for your detailed answer and comments. Hopefully one day the
> transition to nftables will be done and we all can just use the nft tool
> without problems. :-)
I hope so, too. Improving the situation is also on my radar, but there are
constantly more important things to work on, sadly.

In reality, there are chances though that this container-with-init-ns-access
thing will eventually bite us with nftables, too: We don't test for
compatibility between different versions of the tool and new features are
likely to break compat in one direction.

Cheers, Phil

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240418/6fd97f0f/attachment-0001.html>