netfilter buglog - Jan 2024 - [Bug 1735] New: Adding nftables interval sets progressively gets slower and makes the nft CLI less responsive with each added set

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

            Bug ID: 1735
           Summary: Adding nftables interval sets progressively gets
                    slower and makes the nft CLI less responsive with each
                    added set
           Product: nftables
           Version: 1.0.x
          Hardware: All
                OS: All
            Status: NEW
          Severity: major
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: anton.khazan at gmail.com

Created attachment 734
  --> https://bugzilla.netfilter.org/attachment.cgi?id=734&action=edit
shell script demonstrating the bug

The title basically describes the issue.

Steps to reproduce the issue:
Take an ip list for any country (in most of my tests I've been using the ip
list for GB fetched from ipdeny, which currently has 8589 ipv4 ip blocks),
measure time taking to load it into a new set, measure time adding the same ip
list again in a new set under a new name (without removing the 1st one), repeat
n times. Each new iteration takes a little longer. On the OpenWRT installation
in VM, 1st iteration takes 0.12s. 10th iteration takes 0.47s. 40th iteration
takes 1.59s.

Getting a response from the command 'nft list tables' also takes
progressively
longer with each added set, as well as any other nft CLI operation involving
the table under test. Starting with 0.04s after 1st iteration and up to 1.49s
after the 40th iteration.

Monitoring memory use while running this test also reveals that the transient
memory use spikes (similar to those reported in Bug 1584) get progressively
larger as you go, reaching about 180MB around the 40th iteration (i've been
just monitoring memory use via the 'top' utility which samples once a
second at
most, so I might have missed the peak of the spike).

Using the same ip list is not an essential part of the bug, as well as the
specific list for GB. I picked GB because it's a mid-sized list, and
re-using
it because this way it is possible to get comparable results. The issue also
extends to ipv6 ip lists.

nft CLI operations involving other tables seem unaffected.

This is a pretty big problem for applications like geoip, especially so on
embedded devices like routers with limited memory.

I'm not sure whether this belongs to the kernel component or nft userspace.
Filing this under nft userspace component in the meanwhile.

Hardware used for testing: Intel i7-4770
Same behavior on Linux Mint (kernel 6.2.0, nftables 1.0.2) and OpenWRT x86-64
installed in a VM (kernel 5.15.137, nftables 1.0.8).

A shell script reproducing the issue is attached.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240130/f11cf698/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #1 from anton.khazan at gmail.com ---
Also to clarify, the issue happens with or without 'auto-merge',
although
enabling auto-merge somewhat reduces the magnitude.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240130/89f94f6b/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

anton.khazan at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |anton.khazan at gmail.com

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240130/691e6750/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #2 from anton.khazan at gmail.com ---
Update: the author of BanIP (OpenWRT application doing similar tasks with
interval sets) shared their method of populating sets and listing the contents
of a table, which avoids the bug. Attaching their version of the test script
for comparison. Looking at the differences, I'm starting to suspect that
some
commands cause nftables to re-process all accumulated sets (which causes spikes
in memory use and reduced responsiveness), and some don't. 'nft list
tables'
does, while 'nft list ruleset' doesn't. BanIP method of populating
the sets
doesn't, while my method does. I still really don't completely
understand why
but this looks like a bug to me.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240131/3109d231/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #3 from anton.khazan at gmail.com ---
Created attachment 735
  --> https://bugzilla.netfilter.org/attachment.cgi?id=735&action=edit
shell script avoiding the bug

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240131/2cf99289/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

Phil Sutter <phil at nwl.cc> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |phil at nwl.cc
           Assignee|pablo at netfilter.org         |phil at nwl.cc

--- Comment #4 from Phil Sutter <phil at nwl.cc> ---
Pointless set element fetching with 'list tables' command fixed in
commit
674eb7fa28958 ("cache: Optimize caching for 'list tables'
command"). Sadly,
fixing the same for 'add element' command is sadly not as trivial, but
I'll
leave this ticket open as I think it should be doable.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240207/6f456666/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #5 from anton.khazan at gmail.com ---
Thank you for looking into this. Additional affected command:
'nft -t list table inet test'
(0.01s after 1st iteration, 0.5s after 40th iteration)

To my best understanding, it's not supposed to fetch elements when called
with
the '-t' option.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240208/a2ffebff/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #6 from anton.khazan at gmail.com ---
'nft -t list set' is also likely affected to some degree, as comparing
the time
it takes to list a small set (261 ip ranges) to a large set (11655 ip ranges),
I'm noticing a significant difference - 0.01s vs 0.07s on my VM (the actual
difference is likely higher but I'm limited to 0.01s measurement precision).
Note that this is also with the '-t' option so it's not supposed to
fetch
elements (to my understanding).

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240208/35d85553/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1735

--- Comment #7 from Phil Sutter <phil at nwl.cc> ---
(In reply to anton.khazan from comment #6)
> 'nft -t list set' is also likely affected to some degree, as
comparing the
> time it takes to list a small set (261 ip ranges) to a large set (11655 ip
> ranges), I'm noticing a significant difference - 0.01s vs 0.07s on my
VM
> (the actual difference is likely higher but I'm limited to 0.01s
measurement
> precision). Note that this is also with the '-t' option so it's
not supposed
> to fetch elements (to my understanding).
It is. To see for yourself, use --debug=netlink:

| nft --debug=netlink -t list set ...

If debug output contains lots of "element" lines while non-debug
output
doesn't, there likely is a bug (with list-like commands at least).

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240208/34690f35/attachment.html>