LARTC - May 2004 - MARK causes high CPU usage / using tc in conjunction with MASQ

Using below configuration multiplied by 3000+ nodes to control bandwidth
causes very high kernel cpu usage (99.5%) narrowed it down to the mangle
table. Any ideas to do this more efficiently would be appreciated.

The mangle table entry (indicated by ***) is sucking all the cpu. I am
running RH7.3 kernel  2.4.18-3 and iptables 1.2.5
This setup has worked well for more than 1000 devices but as the network
has grown to 3000+ devices the CPU is not keeping up. I have thought to
use IPMARK instead of MARK. Or, to possibly use CLASSIFY. Since this is
hard to recreate in the lab I was looking for some experienced advice on
the matter.

### root ###
tc qdisc add dev eth0 root handle 1: cbq bandwidth 100Mbit avpkt 1000 cell 8

tc qdisc add dev eth1 root handle 1: cbq bandwidth 100Mbit avpkt 1000 cell 8


### Classful qdisc upload/download rate for a group of IP address ###
tc class add dev eth0 parent 1:0 classid 1:11 cbq bandwidth 100Mbit rate
100Mbit weight 54Kbit prio 8 allot 1514 cell 8 maxburst 20 av pkt 1000

tc qdisc add dev eth0 parent 1:11 tbf rate 2048Kbit buffer 10Kb/8 limit
15Kb mtu 1500

tc class add dev eth1 parent 1:0 classid 1:11 cbq bandwidth 100Mbit rate
100Mbit weight 54Kbit prio 8 allot 1514 cell 8 maxburst 20 av pkt 1000

tc qdisc add dev eth1 parent 1:11 tbf rate 2048Kbit buffer 10Kb/8 limit
15Kb mtu 1500


### A single IP address and it''s own upload/download rate ###
tc class add dev eth0 parent 1:11 classid 1:2115 cbq bandwidth 100Mbit
rate 100Mbit weight 54Kbit prio 8 allot 1514 cell 8 maxburst 20 avpkt 1000

tc qdisc add dev eth0 parent 1:2115 tbf rate 2048Kbit buffer 10Kb/8
limit 15Kb mtu 1500

*** eth0 is MASQUERADE''d so I mark the packet on eth1 ***
*** I have narrowed it down to this one entry sucking all the CPU ***
iptables -t mangle -A PREROUTING -s 10.10.6.20 -i eth1 -j MARK
--set-mark 0x843

tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 2115 fw
classid 1:2115

tc class add dev eth1 parent 1:11 classid 1:2115 cbq bandwidth 100Mbit
rate 100Mbit weight 54Kbit prio 8 allot 1514 cell 8 maxburst 20 avpkt 1000

tc qdisc add dev eth1 parent 1:2115 tbf rate 2048Kbit buffer 10Kb/8
limit 15Kb mtu 1500

tc filter add dev eth1 parent 1:0 protocol ip prio 1 u32 match ip dst
10.10.6.20 flowid 1:2115






_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Am Saturday 22 May 2004 23:25 schrieb Paul Lambert:
> This setup has worked well for more than 1000 devices but as the network
> has grown to 3000+ devices the CPU is not keeping up.
I guess you mean 3000+ clients, not actual network devices on one machine.
> *** eth0 is MASQUERADE''d so I mark the packet on eth1 ***
> *** I have narrowed it down to this one entry sucking all the CPU ***
> iptables -t mangle -A PREROUTING -s 10.10.6.20 -i eth1 -j MARK
> --set-mark 0x843
Well, if you have 3000+ rules like that, it will certainly slow you down.
You should use some kind of hashing. How that is done for tc filters,
is described here: http://www.lartc.org/lartc.html#LARTC.ADV-FILTER.HASHING

Apply the same (or a similar) mechanism to your iptables ruleset and 
you should get improved speeds.

HTH
Andreas
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

>Well, if you have 3000+ rules like that, it will certainly slow you down.
>You should use some kind of hashing. How that is done for tc filters,
>is described here: http://www.lartc.org/lartc.html#LARTC.ADV-FILTER.HASHING
>
>Apply the same (or a similar) mechanism to your iptables ruleset and 
>you should get improved speeds.
>  
>
If he wanted to keep the system of using iptables to classify and tc to 
filter, then couldn''t he look at using seperate filter chains to 
decrease the search space?

Also, what about using return rules to speedup the search times in a 
given filter chain?

I think his point was actually that it was not a CPU issue without 
adding that one particular rule.  But perhaps you will have more success 
asking on the iptables list?

Good luck

Ed W
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Am Monday 24 May 2004 00:30 schrieb Ed Wildgoose:
> If he wanted to keep the system of using iptables to classify and tc to
> filter, then couldn''t he look at using seperate filter chains to
> decrease the search space?
Is there any other way to implement iptables hashing than using new chains?
I can''t think of any right now.
> Also, what about using return rules to speedup the search times in a
> given filter chain?
If you mean adding one return rule after every match test, then only
if that doesn''t require testing the same thing twice. Since otherwise
you''ll double the number of tests for all packets that don''t
match at
all or match to the last rule.
> I think his point was actually that it was not a CPU issue without
> adding that one particular rule.  But perhaps you will have more success
> asking on the iptables list?
If it is one *single* rule, then I''d consider this a bug.
If it''s the same rule 3000+ times, then it''s probably normal,
depending on the cost of that particular test.

Andreas
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Andreas Klauer wrote:
> Am Saturday 22 May 2004 23:25 schrieb Paul Lambert:
> 
>>This setup has worked well for more than 1000 devices but as the network
>>has grown to 3000+ devices the CPU is not keeping up.
> 
> 
> I guess you mean 3000+ clients, not actual network devices on one machine.
> 
Yes 3000+ client devices.
> 
>>*** eth0 is MASQUERADE''d so I mark the packet on eth1 ***
>>*** I have narrowed it down to this one entry sucking all the CPU ***
>>iptables -t mangle -A PREROUTING -s 10.10.6.20 -i eth1 -j MARK
>>--set-mark 0x843
> 
> 
> Well, if you have 3000+ rules like that, it will certainly slow you down.
> You should use some kind of hashing. How that is done for tc filters,
> is described here: http://www.lartc.org/lartc.html#LARTC.ADV-FILTER.HASHING
tc is keeping up well for now. However, I think you''re right and this
is
worth implementing.
> 
> Apply the same (or a similar) mechanism to your iptables ruleset and 
> you should get improved speeds.
I like this idea. I never thought about using a hash filter in iptables. 
I could have two sections. I could match the subnet and then jump to 
look up the node address. I think this would lessen the load 
considerably as long as it is the lookup that is taking the most cpu 
cycles and not the actual MARK routine having to execute on every packet.
--
Thanks
> 
> HTH
> Andreas
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> > Apply the same (or a similar) mechanism to your iptables ruleset and 
> > you should get improved speeds.
> 
> I like this idea. I never thought about using a hash filter in iptables. 
> I could have two sections. I could match the subnet and then jump to 
> look up the node address. I think this would lessen the load 
> considerably as long as it is the lookup that is taking the most cpu 
> cycles and not the actual MARK routine having to execute on every packet.
Well, I don''t know your ruleset and setup but maybe you could use the
connection tracking system to improve this. Put the mark value into the
conntack table using CONNMARK extension. 

For every packet you receive you have to restore this value but there''s
no
need to fall through the rules anymore if it is already marked. Lookups in
the CONNTRACK table use hashing so shall be much faster. A huge ruleset
decreases iptables performance pretty bad.

Regards

-- 
"Sie haben neue Mails!" - Die GMX Toolbar informiert Sie beim Surfen!
Jetzt aktivieren unter http://www.gmx.net/info

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

I''m actually using the WISP Billing System from PowerNOC. I talked with
them about your idea and they implemented it for me already. It is 
working great. Thanks everyone for your help!

Andreas Klauer wrote:
> Am Saturday 22 May 2004 23:25 schrieb Paul Lambert:
> 
>>This setup has worked well for more than 1000 devices but as the network
>>has grown to 3000+ devices the CPU is not keeping up.
> 
> 
> I guess you mean 3000+ clients, not actual network devices on one machine.
> 
> 
>>*** eth0 is MASQUERADE''d so I mark the packet on eth1 ***
>>*** I have narrowed it down to this one entry sucking all the CPU ***
>>iptables -t mangle -A PREROUTING -s 10.10.6.20 -i eth1 -j MARK
>>--set-mark 0x843
> 
> 
> Well, if you have 3000+ rules like that, it will certainly slow you down.
> You should use some kind of hashing. How that is done for tc filters,
> is described here: http://www.lartc.org/lartc.html#LARTC.ADV-FILTER.HASHING
> 
> Apply the same (or a similar) mechanism to your iptables ruleset and 
> you should get improved speeds.
> 
> HTH
> Andreas
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/