netfilter buglog - Feb 2012 - [Bug 773] New: iptables performance limits on # of rules using ipset

http://bugzilla.netfilter.org/show_bug.cgi?id=773

           Summary: iptables performance limits on # of rules using ipset
           Product: ipset
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: default
        AssignedTo: netfilter-buglog at lists.netfilter.org
        ReportedBy: aas029 at yahoo.com
   Estimated Hours: 0.0


Observing significant degradation in latency and packet loss of pass-through
traffic (FORWARD chain) when the number of iptables rules that use ipprotiphash
ipset matching exceeds 24 rules. This happens even when the ipsets themselves
are empty or have just a few entries each. 

The following is a striped down example to demonstrate the potential issue:
- create X number of ipportiphash ipsets:
ipset -N UDP-x ipportiphash --network 129.129.0.0/22
where x is from 1 to X
- add X number of iptables rules each of which matching on one of the above
created ipsets:
iptables -I FORWARD 1 -m set --match-set UDP-x src,src,dst -j ACCEPT;
where x is from 1 to X

When X is up to 24 (i.e. 24 ipsets and corrensponding 24 iptable rules), the
average latency of packets forwarded thought the system is in the order of
100us (micro seconds) and no packets are dropped. This result is close to the
case where the are no iptables rules at all. 

When X is around 29 or above, the latency of the very same system (no other
changes) is 5ms i.e. 50 times larger and 4% or more of the packets are dropped
by the system. 

The above is true even if the ipsets contain only a few entries each or are
completely empty. Somehow the number of iptables rules using ipportiphash ipset
above a given threshold (around 24 rules in my system) causes a huge
performance degradation. On the other hand, for X < 24, the results don't
seem
to depend on X and moreover, the number of entries in the ipsets doesn't
seem
to matter much either. For examples, average latency results stay <150us even
when the ipsets are full (contain 64k entries each) as long as there is no more
than 24 iptables rules referring to these largest size ipsets. 

Note that the traffic forwarded through the system in the above tests does NOT
match any of the rules (when all ipsets are empty, this is obviously true
regardless of the src and dst IPs in the packets). This 'dummy'
config/behavior
is intended in order to force the system to match against all rules (X of them
to be more specific) and be able measure/determine the latency cost of matching
on X number of rules. 

As a way of comparison, the same system works fine with 1000 iptables rules
that do NOT use ipsets like the following:
iptables -I FORWARD 1 -s 129.129.0.0/24 -d 29.30.0.0/16 -j ACCEPT
The rules like the above one were also chosen so that the dst IP of test
traffic never matches any of the rules' specified destination.

So to summarize, the system is capable of processing 1000 of 'normal'
iptables
rules (that use -s and -d matching) like the above for each packet but on the
other hand has significant performance degradation in processing just 29 rules
that use ipportiphash ipset even when the ipsets used are empty. 


Version information is below:
ipset v4.5, protocol version 4.
Kernel module protocol version 4.
iptables v1.4.7
Linux 2.6.32

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

Al <aas029 at yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P5                          |P4
                 CC|                            |aas029 at yahoo.com
           Severity|enhancement                 |minor

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

Jan Engelhardt <jengelh at medozas.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jengelh at medozas.de
         AssignedTo|netfilter-buglog at lists.netf |kadlec at netfilter.org
                   |ilter.org                   |

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

--- Comment #1 from Jozsef Kadlecsik <kadlec at netfilter.org> 2012-02-29
16:09:35 CET ---
This is the old ipset branch, which protects the sets with a single spinlock. I
suspect that is the reason for the performance degradation.

The development of ipset is focused on the ipset 6.x branch. I don't have
free
resource to work on the old branch, so please try to upgrade (but it needs at
least kernel 2.6.34).

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

--- Comment #2 from Al <aas029 at yahoo.com> 2012-02-29 18:43:30 CET ---
(In reply to comment #1)
> This is the old ipset branch, which protects the sets with a single
spinlock. I
> suspect that is the reason for the performance degradation.
> 
> The development of ipset is focused on the ipset 6.x branch. I don't
have free
> resource to work on the old branch, so please try to upgrade (but it needs
at
> least kernel 2.6.34).
Hi Jozsef,
Thanks for the very quick response. 

To give you a bit more details, if I create just one empty 1 ipset and then
build 29 identical dummy iptables rules (dummy in the sense that the rules will
never match any of the test packets) that use this single empty ipset , I still
can reproduce the problem. Just wanted to let you know this is not tied to the
number of ipsets but just the number of iptables rules that refer to ipset. 

I have not checked if the behavior applies to other types of ipsets than the
ipportiphash. This would be something fast for me to check if it helps knowing
/ confirming what you suspect.  

While I will not be able to use 2.6.34, I'll try to take a different Linux
distribution just to confirm whether this performs better on newer releases. 

Is this spin lock acquired every time a packet is processed? 

It's not easy to imagine how a lock related bottleneck would not seem to
have
almost any impact for up to 24 rules and then cause a very significant
performance impact lightly above that # of rules threshold.

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

--- Comment #3 from Al <aas029 at yahoo.com> 2012-02-29 21:34:31 CET ---
(In reply to comment #2)
> (In reply to comment #1)
> > This is the old ipset branch, which protects the sets with a single
spinlock. I
> > suspect that is the reason for the performance degradation.
> > 
> > The development of ipset is focused on the ipset 6.x branch. I
don't have free
> > resource to work on the old branch, so please try to upgrade (but it
needs at
> > least kernel 2.6.34).
> 
> Hi Jozsef,
> Thanks for the very quick response. 
> 
> To give you a bit more details, if I create just one empty 1 ipset and then
> build 29 identical dummy iptables rules (dummy in the sense that the rules
will
> never match any of the test packets) that use this single empty ipset , I
still
> can reproduce the problem. Just wanted to let you know this is not tied to
the
> number of ipsets but just the number of iptables rules that refer to ipset.
> 
> I have not checked if the behavior applies to other types of ipsets than
the
> ipportiphash. This would be something fast for me to check if it helps
knowing
> / confirming what you suspect.  
> 
> While I will not be able to use 2.6.34, I'll try to take a different
Linux
> distribution just to confirm whether this performs better on newer
releases.
> 
> Is this spin lock acquired every time a packet is processed? 
> 
> It's not easy to imagine how a lock related bottleneck would not seem
to have
> almost any impact for up to 24 rules and then cause a very significant
> performance impact lightly above that # of rules threshold.
Hi Jozsef,

One more related question if you can advise. Since I don't have the option
to
upgrade the OS, I may be able to limit my rules to not exceed the threshold
that causes performance issues. 
To do so, I think I need to have an ipsets with more than 64k entries (the
current max). Is changing this max as simple as doubling MAX_RANGE in ip_set.h
and recompiling ipset? or are there other dependency? Not I only use
ipportiphash in case this makes the workaround any simpler.  

/* General limit for the elements in a set */
#define MAX_RANGE 0x0000FFFF

Thanks a lot,
Adel.

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.

http://bugzilla.netfilter.org/show_bug.cgi?id=773

--- Comment #4 from Jozsef Kadlecsik <kadlec at netfilter.org> 2012-03-01
15:25:48 CET ---
> > To give you a bit more details, if I create just one empty 1 ipset and
then
> > build 29 identical dummy iptables rules (dummy in the sense that the
rules will
> > never match any of the test packets) that use this single empty ipset
, I still
> > can reproduce the problem. Just wanted to let you know this is not
tied to the
> > number of ipsets but just the number of iptables rules that refer to
ipset.
> > 
> > I have not checked if the behavior applies to other types of ipsets
than the
> > ipportiphash. This would be something fast for me to check if it helps
knowing
> > / confirming what you suspect.  
You can easily check with any other set type. I'd guess the result would be
the
same.
> > While I will not be able to use 2.6.34, I'll try to take a
different Linux
> > distribution just to confirm whether this performs better on newer
releases.
> > 
> > Is this spin lock acquired every time a packet is processed? 
Yes, it's acquired at processing every packet and every rule. But it's
still a
guessing why the performance drops.

In 6.x the lock was removed.
> > It's not easy to imagine how a lock related bottleneck would not
seem to have
> > almost any impact for up to 24 rules and then cause a very significant
> > performance impact lightly above that # of rules threshold.
> 
> One more related question if you can advise. Since I don't have the
option to
> upgrade the OS, I may be able to limit my rules to not exceed the threshold
> that causes performance issues. 
> To do so, I think I need to have an ipsets with more than 64k entries (the
> current max). Is changing this max as simple as doubling MAX_RANGE in
ip_set.h
> and recompiling ipset? or are there other dependency? Not I only use
> ipportiphash in case this makes the workaround any simpler.  
> 
> /* General limit for the elements in a set */
> #define MAX_RANGE 0x0000FFFF
You can hack ipset 4.x that way, the MAX_RANGE value was just a safety belt in
order to prevent exhausting the memory accidentally. But if you know what you
are doing, then you can remove that limit.

Best regards,
Jozsef

-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching all bug changes.