netfilter buglog - Jul 2020 - [Bug 1439] New: Atomically updating/reloading a large set with nft -f is excessively slow

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

            Bug ID: 1439
           Summary: Atomically updating/reloading a large set with nft -f
                    is excessively slow
           Product: nftables
           Version: unspecified
          Hardware: x86_64
                OS: Debian GNU/Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: public_timo.s at silentcreek.de

Hi,

I'm trying to move my iptables/ipset setups to nftables. Some of these
setups
involve fairly large sets and updating those atomically does not seem to be
practical due to the excessive amount of time it takes to reload the set via a
script file.

Assume a set that was initialized like this:
  nft add set inet filter ipv6_bogons { type ipv6_addr; flags interval; }

And then load the attached example ipv6_bogons.nft with `nft -f' to populate
the set with elements. When you first load that file, it's a matter of
seconds
(or less). But if you try to load the set another time in order to reload it
(the script contains a flush set statement at the top), it takes excessively
long.

For reference: On an 8th generation Intel Core i7 with a turbo frequency of
5GHz, reloading the file takes over 3 minutes(!) as opposed to ~0.6s when
loaded initially. If you do this on less powerful hardware, such as a dual-core
1GHz ARMv7 SoC, first loading the file takes a few seconds, but the second time
seems to take forever - I aborted the experiment after 45 minutes! These
loading times are just not practical, even on powerful hardware, when you're
working with multiple large sets and looking for a way to reload them
atomically. (Obviously, you wouldn't want to reload the exact same set, but
after updating the file with added or deleted elements.)

The only workaround so far that I found was to reload the complete ruleset
(with  an include statement that references the attached ipv6_bogons.nft file).
This works quicky, but has a few drawbacks, naturally, such as resetting
counters and losing dynamically populated sets (with elements that are not
permanently saved to nft script files).

I have tested this both with nftables 0.9.0-2 (Debian 10) as well as 0.9.3-2
(Ubuntu 20.04).

If there is anything I can do to speed this up and reload a large set
atomically, please let me know.

Cheers,

Timo

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200701/bd639c52/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #1 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
Created attachment 598
  --> https://bugzilla.netfilter.org/attachment.cgi?id=598&action=edit
Example: Large ipv6 set (takes a long time to reload with nft -f)

Sorry, had to compress the example file to circumvent the upload file size
limit.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200701/1f95fd25/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #2 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
Oh, I forgot to mention: Another workaround to the excessive reload times of
such large sets is to flush the set before reloading it with `nft -f'. Then
it
happens as quickly as initially loading the set, but obviously the update
isn't
atomic anymore.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200702/bd82715c/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kfm at plushkava.net

--- Comment #3 from kfm at plushkava.net ---
This seems to be a duplicate of bug 1392, for which I still haven't
discerned
any operational workaround.

My view is that some serious thought should be given to supporting ipsets. As
you have noticed, the present implementation of native sets suffers from
various bugs and, though it pains me to say it, it just isn't stable enough
to
usurp ipsets at the current time.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200703/ab5f68a4/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugzilla.netfilter.
                   |                            |org/show_bug.cgi?id=1392

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200703/2fe05333/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|enhancement                 |normal

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200703/67c1a9d1/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #4 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
(In reply to kfm from comment #3)
> This seems to be a duplicate of bug 1392, for which I still haven't
> discerned any operational workaround.
Indeed, this seems to be the same issue. I missed that one.
> My view is that some serious thought should be given to supporting ipsets.
> As you have noticed, the present implementation of native sets suffers from
> various bugs and, though it pains me to say it, it just isn't stable
enough
> to usurp ipsets at the current time.
Yeah, I also checked whether it's possible to match ipsets with nftables,
but
concluded that wasn't the case as I couldn't find any information on
it...

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200703/75f07db5/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #5 from Pablo Neira Ayuso <pablo at netfilter.org> ---
Thanks for reporting.

Please give a test to current upstream nft git snapshot, specifically, this fix
this targetting at speeding up loading time in interval sets:

http://git.netfilter.org/nftables/commit/?id=40ef308e19b6db02017a8a650406b0c6d37be750

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200721/123af3b8/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #6 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
Thanks, I'll give this a shot, but it might take me a couple of days to find
the time.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200721/47c131ef/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #7 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
(In reply to Pablo Neira Ayuso from comment #5)
> Thanks for reporting.
> 
> Please give a test to current upstream nft git snapshot, specifically, this
> fix this targetting at speeding up loading time in interval sets:
> 
> http://git.netfilter.org/nftables/commit/
> ?id=40ef308e19b6db02017a8a650406b0c6d37be750
I'm afraid this has not solved the issue here. I've build nftables and
libnftnl
from source on Ubuntu 20.04 and the problem with the example file from above
persists.

`nft -v' shows:
  nftables v0.9.6 (Capital Idea #2)
I cloned the repositories I used for building at commit 7c9bef0 (nftables) and
commit 330ca1c(libnftnl) respectively.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200729/03c00836/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #8 from Pablo Neira Ayuso <pablo at netfilter.org> ---
Testing with nftables at 7c9bef0c

# nft add table inet filter
# nft add set inet filter ipv6_bogons { type ipv6_addr\; flags interval\; }

# nft list ruleset
table inet filter {
        set ipv6_bogons {
                type ipv6_addr
                flags interval
        }
}

... download your bogons tarball, decompress

# nft -f ipv6_bogons.nft
... few seconds
# nft -f ipv6_bogons.nft
... few seconds
# nft list ruleset | wc -l
113210

Are you sure you are running a fresh nft binary?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200730/e420c45e/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

--- Comment #9 from Timo Sigurdsson <public_timo.s at silentcreek.de> ---
(In reply to Pablo Neira Ayuso from comment #8)
> Testing with nftables at 7c9bef0c
> 
> # nft add table inet filter
> # nft add set inet filter ipv6_bogons { type ipv6_addr\; flags interval\; }
> 
> # nft list ruleset
> table inet filter {
>         set ipv6_bogons {
>                 type ipv6_addr
>                 flags interval
>         }
> }
> 
> ... download your bogons tarball, decompress
> 
> # nft -f ipv6_bogons.nft
> ... few seconds
> # nft -f ipv6_bogons.nft
> ... few seconds
> # nft list ruleset | wc -l
> 113210
> 
> Are you sure you are running a fresh nft binary?
As mentioned in the other bug. You're right. My installation of the fresh
binaries and libraries was somehow broken. I just assume I was running the new
build since nft -v returned 0.9.6 (whereas Ubuntu 20.04 ships 0.9.3). I
reinstalled the custom build from fresh and now it works. Loading the set for
the first time takes ~0.6s and repeated loads take only slightly longer with
~0.9s. All good! Thanks!

Another question: In a different bug you mentioned that your fix in nftables
requires also an updated libnftnl. Is there a specific commit in libnftnl that
is required by your change or does it depend on more changes? I'm asking
because I'd like to try to backport the fix to Debian stable if the changes
required are fairly small.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200730/4950001a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1439

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #10 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to Timo Sigurdsson from comment #9)
> (In reply to Pablo Neira Ayuso from comment #8)
[...]
> > Are you sure you are running a fresh nft binary?
> 
> As mentioned in the other bug. You're right. My installation of the
fresh
> binaries and libraries was somehow broken. I just assume I was running the
> new build since nft -v returned 0.9.6 (whereas Ubuntu 20.04 ships 0.9.3). I
> reinstalled the custom build from fresh and now it works. Loading the set
> for the first time takes ~0.6s and repeated loads take only slightly longer
> with ~0.9s. All good! Thanks!
Thank you for confirming this works for you.
> Another question: In a different bug you mentioned that your fix in
nftables
> requires also an updated libnftnl. Is there a specific commit in libnftnl
> that is required by your change or does it depend on more changes? I'm
> asking because I'd like to try to backport the fix to Debian stable if
the
> changes required are fairly small.
The next nftables version contains support for chain bindings:

https://git.netfilter.org/nftables/commit/?id=c330152b7f7779f15dba3e0862bf5616e7cb3eab

This requires this recent update for libftnl:

http://git.netfilter.org/libnftnl/commit/?id=20e0e8f2bec3b06ba5d77ec17d8e635750f4c085

This chain binding support requires Linux kernel 5.9-rc1 (yet to be release,
this is net-next material).

So, if you carefully pick the patches to backport - skipping the patches that
are related to the chain binding support - then you don't need the most
recent
version of libnftnl.

Closing this bug.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200730/7c917878/attachment.html>