netfilter buglog - Dec 2016 - [Bug 1101] New: SET target unreliable in iptables - add does not work as expected

https://bugzilla.netfilter.org/show_bug.cgi?id=1101

            Bug ID: 1101
           Summary: SET target unreliable in iptables - add does not work
                    as expected
           Product: netfilter/iptables
           Version: unspecified
          Hardware: x86_64
                OS: Debian GNU/Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: ip_tables (kernel)
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: koetter at luis.uni-hannover.de

Created attachment 486
  --> https://bugzilla.netfilter.org/attachment.cgi?id=486&action=edit
iptables -nvL special-unused:filter

I'm with debian Jessie, 

Linux <> 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64
GNU/Linux
iptables v1.4.21
ipset v6.23, protocol version: 6

I use the ipset SET target to create dynamic lists of addresses to block.
The problem: the SET target fails with ~50% of the cases to add an address
properly. A subsequent match on the ipset fails - the address is not added to
the set.

To provide an example, I modified my rules to add & match subsequently - one
would expect the counters to match, but they do not.

It is possible to verify an address is not added to the set using ipset
userspace as well.
The ipset has about 20k entries, adding via ipset cli always works as expected.

The machine I'm working does quite some traffic - so it may be a race
condition
and hard to reproduce.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161209/25862658/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1101

Jozsef Kadlecsik <kadlec at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kadlec at netfilter.org
             Status|NEW                         |ASSIGNED

--- Comment #1 from Jozsef Kadlecsik <kadlec at netfilter.org> ---
What parameter did you use at creating the set in question? I.e. what are the
hash size and maxelem parameters of the set?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161210/668a3a81/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1101

--- Comment #2 from Markus K�tter <koetter at luis.uni-hannover.de> ---
I do not see any parameters getting passed to "ipset create" when
creating.

According to "ipset save" 

"create blocked:host:dynamisch:net hash:net family inet hashsize 2048
maxelem
65536 timeout 0"

is used.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161212/c0a3169a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1101

--- Comment #3 from Markus K�tter <koetter at luis.uni-hannover.de> ---
Created attachment 487
  --> https://bugzilla.netfilter.org/attachment.cgi?id=487&action=edit
Rule hitcunters after some runtime.

The rules had some runtime, seems like the numbers get worse over time:

18636 / 193k ~ 9%
7843K / 19M ~40%
0 / 50128 ~ 0%
338 / 758k ~ 0.0004%
0 / 14585 ~ 0%
15 / 168k ~ 0.00008%

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161212/538d7389/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1101

Jozsef Kadlecsik <kadlec at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |INVALID

--- Comment #4 from Jozsef Kadlecsik <kadlec at netfilter.org> ---
This is the expected behaviour and documented in the manpage:

"In order to avoid clashes in the hash, a limited number of chaining,
and if that is exhausted, the doubling of the hash size is performed
when adding entries by the ipset command. When entries added by
the SET target of iptables/ip6tables, then the hash size is fixed
and the set won't be duplicated, even if the new entry cannot be
added to the set."

You have to create the set with a proper hashsize parameter if
the elements are added by the SET target.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161217/3f8415d5/attachment.html>