hi i''m using htb + u32 filters, and i was wondering if there is something one can optimize at this stage. i have a lot of filters (~ 50.000 / interface, and there are two interfaces), and around 4500 classes / interface. the traffic going through this machine is something around 210-230mbit/s at 50kpps. as you can imagine, the load is pretty high. in fact (as it''s a dual xeon at 2.4ghz), one CPU is always at 100% when the traffic increases. i did some tests with esfq (that brought down the classes to around 150), but the filters remained, and the load was still 100%. and i get some packet loss because of that. not much, around 1-2%, but it''s enough :) is there something i could do to bring the load down? short of replacing the whole system? i didn''t find anything performance-related on the net, or in any documentation. thanks.
On Wed, Feb 08, 2006 at 07:29:57PM +0200, Imre Gergely wrote:> i did some tests with esfq (that brought down the classes to around 150), but > the filters remained, and the load was still 100%. and i get some packet loss > because of that. not much, around 1-2%, but it''s enough :) > > is there something i could do to bring the load down?Are the filters already hashed? If not, that''s the first thing I''d try. There was a section on that on www.lartc.org. (Hmmm, seems to be down.). http://www.linux.org/docs/ldp/howto/Adv-Routing-HOWTO/lartc.adv-filter.hashing.html HTH Andreas Klauer
yepp, hashing is done, for every type C class (/24), there are around 300 of these, and all are redirected to a more specific table, according to the documentation. now i have a question about this, too. to me it''s not clear how these filters are looked up. at first, there is that default table 800::, where i create these ''hashing filters''. if i have 300 of them, how are they processed? if a packet comes in, what happens? are they looked up in the same order i created them? like in iptables? then, if say, one filter matched, the more specific filter table is looked up, the key being the last octet of the ip address (specified by the mask 0x000000ff). it looks up the right entry in the table, and it know in which flow (in which class) it should put the packet in. right? now what if i have to filters? one with, say, a source port of 25 specified, the other one with port 80. these are some ''subquestions'' :) the main question is the optimisation of course :) i was just wondering how things are done. Andreas Klauer wrote:> On Wed, Feb 08, 2006 at 07:29:57PM +0200, Imre Gergely wrote: >> i did some tests with esfq (that brought down the classes to around 150), but >> the filters remained, and the load was still 100%. and i get some packet loss >> because of that. not much, around 1-2%, but it''s enough :) >> >> is there something i could do to bring the load down? > > Are the filters already hashed? If not, that''s the first thing I''d try. > There was a section on that on www.lartc.org. (Hmmm, seems to be down.). > > http://www.linux.org/docs/ldp/howto/Adv-Routing-HOWTO/lartc.adv-filter.hashing.html > > HTH > Andreas Klauer > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >
On Wed, Feb 08, 2006 at 07:58:48PM +0200, Imre Gergely wrote:> at first, there is that default table 800::, where i create these ''hashing > filters''. if i have 300 of them, how are they processed? if a packet comes in, > what happens? are they looked up in the same order i created them? like in > iptables?Yes, if i understood you correctly. You can create a hash to match a network (hashkey mask 0x0000ff00), then match an address (in another hash table). Or even match with hashkey mask 0x000ff000 (or another bits), then... etc. You can cascade them, in other words.> then, if say, one filter matched, the more specific filter table is looked up, > the key being the last octet of the ip address (specified by the mask > 0x000000ff). it looks up the right entry in the table, and it know in which > flow (in which class) it should put the packet in. right? > > now what if i have to filters? one with, say, a source port of 25 specified, > the other one with port 80.Mmm... If I understood you... :-) Sorry, that''s my english. Every hash table cell may contain many filters, they will be processed in sequence. And if the packet will not match any of them, it''ll be dropped to default class.> > these are some ''subquestions'' :) the main question is the optimisation of > course :) i was just wondering how things are done.-- _,-=._ /|_/| `-.} `=._,.-=-._., @ @._, `._ _,-. ) _,.-'' ` G.m-"^m`m'' Dmytro O. Redchuk
On Wed, Feb 08, 2006 at 07:58:48PM +0200, Imre Gergely wrote:> yepp, hashing is done, for every type C class (/24), there are around 300 of > these, and all are redirected to a more specific table, according to the > documentation.That''s weird, then - with proper hashing, the total number of filter rules should not affect CPU load too much, since only very few of the filters actually have to be traversed. Maybe it''s caused by something else, or the hashing does not work as expected.> now i have a question about this, too. to me it''s not clear how these filters > are looked up.Good question. Actually I can''t answer it properly. For my filters, the order either did not really matter or I had few enough of them to use the priority parameter to order them properly. Regards Andreas Klauer
Dnia środa, 8 lutego 2006 18:29, Imre Gergely napisał:> hi > > i''m using htb + u32 filters, and i was wondering if there is something one > can optimize at this stage. i have a lot of filters (~ 50.000 / interface, > and there are two interfaces), and around 4500 classes / interface. the > traffic going through this machine is something around 210-230mbit/s at > 50kpps. as you can imagine, the load is pretty high. in fact (as it''s a > dual xeon at 2.4ghz), one CPU is always at 100% when the traffic increases. > > i did some tests with esfq (that brought down the classes to around 150), > but the filters remained, and the load was still 100%. and i get some > packet loss because of that. not much, around 1-2%, but it''s enough :) > > is there something i could do to bring the load down? short of replacing > the whole system? i didn''t find anything performance-related on the net, or > in any documentation. > > thanks.Show your dmesg, cat /proc/interrupts ( or use itop to determine which card/interface is hogging ), lsmod and .config from kernel compilation Also show us ip -s link What ethcards do you have, is NAPI enabled on them ? You could also disable connection tracking if that''s not done already. And finally, are you using any libpcap based application ? -- Jakub Wartak -vnull FreeBSD/OpenBSD/Linux/Solaris/Network Administrator http://vnull.pcnet.com.pl/
Jakub Wartak wrote:> Dnia środa, 8 lutego 2006 18:29, Imre Gergely napisał: >> hi >> >> i''m using htb + u32 filters, and i was wondering if there is something one >> can optimize at this stage. i have a lot of filters (~ 50.000 / interface, >> and there are two interfaces), and around 4500 classes / interface. the >> traffic going through this machine is something around 210-230mbit/s at >> 50kpps. as you can imagine, the load is pretty high. in fact (as it''s a >> dual xeon at 2.4ghz), one CPU is always at 100% when the traffic increases. >> >> i did some tests with esfq (that brought down the classes to around 150), >> but the filters remained, and the load was still 100%. and i get some >> packet loss because of that. not much, around 1-2%, but it''s enough :) >> >> is there something i could do to bring the load down? short of replacing >> the whole system? i didn''t find anything performance-related on the net, or >> in any documentation. >> >> thanks. > > Show your dmesg, cat /proc/interrupts ( or use itop to determine which > card/interface is hogging ), lsmod and .config from kernel compilation > Also show us ip -s link[root@btv root]# cat /proc/interrupts CPU0 CPU1 0: 55921457 383025821 IO-APIC-edge timer 1: 342 259 IO-APIC-edge i8042 2: 0 0 XT-PIC cascade 8: 0 0 IO-APIC-edge rtc 14: 1 13 IO-APIC-edge ide0 24: 2326117989 1473249 IO-APIC-level ioc0, eth1 25: 305396 1034030719 IO-APIC-level ioc1, eth2 28: 62532 2546645 IO-APIC-level eth0 NMI: 111277 253384 LOC: 438830354 438830358 ERR: 0 MIS: 0 (eth1 is the download interface. eth2 is the upload, on which currently is no htb) dmesg attached. [root@btv root]# lsmod Module Size Used by bcm5700 132208 0 e100 34304 0 mii 5440 1 e100 .config, ip -s link output attached.> What ethcards do you have, is NAPI enabled on them ?02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03)> You could also disable connection tracking if that''s not done already.iptables is used only on INPUT, for firewall.> And finally, are you using any libpcap based application ?only occasionaly, for a couple of seconds. note: the initial system as of starting the thread was replaced with this one. _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc