netfilter buglog - Feb 2018 - [Bug 1228] New: [REGRESSION] nft cannot load big set anymore

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

            Bug ID: 1228
           Summary: [REGRESSION] nft cannot load big set anymore
           Product: nftables
           Version: unspecified
          Hardware: x86_64
                OS: other
            Status: NEW
          Severity: normal
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: oleksandr at natalenko.name

Created attachment 530
  --> https://bugzilla.netfilter.org/attachment.cgi?id=530&action=edit
Huge set

Hello.

After nft v0.8.1 is released, I cannot load my huge set of blackholes anymore.
nft just eats CPU at 100% while loading the rules. I've waited for 5 minutes
before killing it. v0.8.0 performed the load well in ~15 seconds.

Next, I've recompiled the nft v0.8.2 with the debug info and then attached
to
it with gdb while it wastes CPU to get a backtrace:

==(gdb) bt
#0  0x00007fc9593d134b in __gmpz_cmp () from /usr/lib/libgmp.so.10
#1  0x00005600a1cb1097 in interval_overlap (e1=e1 at entry=0x5600aaf5efc0,
e2=e2 at entry=0x5600addf0240) at segtree.c:350
#2  0x00005600a1cb1794 in intervals_overlap (keylen=<optimized out>,
intervals=<optimized out>, msgs=<optimized out>)
    at segtree.c:386
#3  set_to_segtree (merge=<optimized out>, add=true, tree=0x7fff36409230,
init=0x5600a2fdc590, set=0x5600a2fdbe10, 
    msgs=0x7fff36409490) at segtree.c:416
#4  set_to_intervals (errs=0x7fff36409490, set=set at entry=0x5600a2fdbe10,
init=0x5600a2fdc590, add=add at entry=true, 
    debug_mask=<optimized out>, merge=false) at segtree.c:586
#5  0x00005600a1c94fe0 in do_add_set (flags=0, set=0x5600a2fdbe10,
h=0x5600a2fdcaf0, ctx=0x7fff364093e0) at rule.c:1015
#6  do_command_add (ctx=0x7fff364093e0, cmd=0x5600a2fdcab0,
excl=excl at entry=false) at rule.c:1054
#7  0x00005600a1c967a7 in do_command (ctx=ctx at entry=0x7fff364093e0,
cmd=cmd at entry=0x5600a2fdcab0) at rule.c:1805
#8  0x00005600a1c810e5 in nft_netlink (nf_sock=0x5600a2fdbb50,
msgs=0x7fff36409490, state=0x7fff364094a0, nft=0x5600a2fdba20)
    at libnftables.c:47
#9  nft_run (nft=nft at entry=0x5600a2fdba20, nf_sock=0x5600a2fdbb50,
scanner=scanner at entry=0x5600a2fdbb90, 
    state=state at entry=0x7fff364094a0, msgs=msgs at entry=0x7fff36409490) at
libnftables.c:95
#10 0x00005600a1c817dc in nft_run_cmd_from_filename (nft=0x5600a2fdba20,
filename=0x7fff3640be70 "/etc/nftables.conf")
    at libnftables.c:323
#11 0x00005600a1c80799 in main (argc=3, argv=0x7fff36409d58) at main.c:276
==
I think this has something to do with the changes related to checking for
overlapping ranges introduced in v0.8.1.

Here is how the set gets loaded:

==…
table inet filter {
       set blackhole {
               type ipv4_addr
               flags interval
               include "/etc/nftables-blackhole.conf"
       }
…
==
I'm attaching the nftables-blackhole.conf file to this ticket too.

Could you please fix this?

Thanks.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180218/0eebf499/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

Florian Westphal <fw at strlen.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fw at strlen.de

--- Comment #1 from Florian Westphal <fw at strlen.de> ---
(In reply to Oleksandr Natalenko from comment #0)
[..]
> I think this has something to do with the changes related to checking for
> overlapping ranges introduced in v0.8.1.
> 
> Here is how the set gets loaded:
> 
> ==> …
> table inet filter {
>        set blackhole {
>                type ipv4_addr
>                flags interval
>                include "/etc/nftables-blackhole.conf"
>        }
> …
> ==> 
> I'm attaching the nftables-blackhole.conf file to this ticket too.
Thanks for the detailed info.
Caused by commit 9a4b513014cfdeaad6d247b72a7924b3a536cfe9 (refs/bisect/bad)
src: Don't merge adjacent/overlapping ranges
> Could you please fix this?
We will look at fixing this, in the mean time, you should be able to get
performance (and old behaviour) back by adding auto-merge keyword:

table inet filter {
       set blackhole {
               type ipv4_addr
               auto-merge
               flags interval
               include "/etc/nftables-blackhole.conf"
       }

This restores old behaviour and makes nft not check for overlapping
addresses.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180218/8ce80093/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

--- Comment #2 from Oleksandr Natalenko <oleksandr at natalenko.name> ---
Hi.

Thanks for the response. The "auto-merge" statement brings things back
into a
working state indeed. I'll use it as a workaround then.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180218/446b42a5/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #3 from Pablo Neira Ayuso <pablo at netfilter.org> ---
Fixed upstream.

http://git.netfilter.org/nftables/commit/?id=4d6ad0f310d6cc3a1d776d32d9d7d678017c6dd7

BTW, is the blackhole file that you're attaching autogenerated? I'm
telling
this because I could add a shell script test that autogenerates it.

See tests/shell/testcases/0029named_ifname_dtype_0 for instance.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180227/06cde630/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

--- Comment #4 from Oleksandr Natalenko <oleksandr at natalenko.name> ---
(In reply to Pablo Neira Ayuso from comment #3)
> Fixed upstream.
> 
> http://git.netfilter.org/nftables/commit/
> ?id=4d6ad0f310d6cc3a1d776d32d9d7d678017c6dd7
Thank you.
> BTW, is the blackhole file that you're attaching autogenerated? I'm
telling
> this because I could add a shell script test that autogenerates it.
> 
> See tests/shell/testcases/0029named_ifname_dtype_0 for instance.
No, it is just one of those lists [1], parsed by the iprange tool

[1] https://www.iblocklist.com/lists

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180227/a0960ec7/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1228

Florian Westphal <fw at strlen.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20180328/2b7998ec/attachment.html>