Hello, I''ve just compiled the kernel 2.6.17 and noticed an odd HTB behaviour. For instance: tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 750Kbit burst 15k tc filter add dev eth0 parent 1:0 protocol ip prio 15 u32 match ip dst 192.168.5.1 classid 1:30 The filter works ok and the traffic to 192.168.5.1 flows through the class 1:30, but the rate gets much higher than 750 Kbps. As far as I realized, the faster the processor is, the higher the traffic gets above the ceiling. I''ve seen that same behaviour in several machines with this kernel, both with Intel and AMD processors. With a dual Xeon 3 GHz, when my ceiling is 750 Kbps, I get the traffic about 5 Mbps. It seems to be something related to clock. http://mailman.ds9a.nl/pipermail/lartc/2005q3/016981.html In that post, there is another guy with the same problem, but with 2.6.11. Any clues? -- Marlon
What kind of network cards do you have? What are the tc stats that you get on the class (anything about giants?)? There''s something funny with the intel gig cards. Even if the mtu on them is set to 1500 there''s something that breaks / confuses htb. If these are the cards you are using you need to explicitly set the mtu for your classes to 16500.>Hello, > >I''ve just compiled the kernel 2.6.17 and noticed an odd HTB behaviour. > >For instance: > >tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil >750Kbit burst 15k >tc filter add dev eth0 parent 1:0 protocol ip prio 15 u32 match ip dst >192.168.5.1 classid 1:30 > >The filter works ok and the traffic to 192.168.5.1 flows through the >class 1:30, but the rate gets much higher than 750 Kbps. > >As far as I realized, the faster the processor is, the higher the >traffic gets above the ceiling. > >I''ve seen that same behaviour in several machines with this kernel, >both with Intel and AMD processors. With a dual Xeon 3 GHz, when my >ceiling is 750 Kbps, I get the traffic about 5 Mbps. > >It seems to be something related to clock. > >http://mailman.ds9a.nl/pipermail/lartc/2005q3/016981.html > >In that post, there is another guy with the same problem, but with 2.6.11. > >Any clues?_________________________________________________________________ Express yourself instantly with MSN Messenger! Download today - it''s FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
On 27-07-2006 19:45, Marlon Dutra wrote:> Hello, >...> > http://mailman.ds9a.nl/pipermail/lartc/2005q3/016981.html > > In that post, there is another guy with the same problem, but with 2.6.11.I can''t see any problem there! If this guy expected lower rate for 1:39 class, he should have lower the ceil. This class had much higher traffic then the other class and probably was borrowing just when this stats were prepared. Jarek P.
On 7/27/06, Jake Altchill <altchill@hotmail.com> wrote:> What kind of network cards do you have?When I noticed the problem, I was using a Dell server with an Intel e1000. But I reproduced the same problem in my desktop, that''s an AMD Semprom with an nVidia ethernet card. So, different processor, chipset and nic.> What are the tc stats that you get on the class (anything about > giants?)?class htb 1:31 parent 1:1 leaf 31: prio 0 rate 1000bit ceil 750000bit burst 15Kb cburst 1974b Sent 11678670 bytes 2711 pkts (dropped 0, overlimits 0) rate 1725Kbit 43pps backlog 13p As you can see, I have rate in 1725Kbit when my ceiling is 750Kbit. That difference is low, since I''m testing on my Semprom. If I do the same on the Dell server, the difference is much higher. Some times over ten times the ceiling. With a 2.4 kernel, it works perfectly in both hardwares.> If these are the cards you are using you need to explicitly set the > mtu for your classes to 16500.Sorry, I didn''t get it. How should I do that? -- Marlon
On 7/28/06, Jarek Poplawski <jarkap@poczta.onet.pl> wrote:> I can''t see any problem there! If this guy expected lower rate for > 1:39 class, he should have lower the ceil. This class had much higher > traffic then the other class and probably was borrowing just when this > stats were prepared.Indeed. In that case, there is no any problem. But I still have a problem here, as you can see in my last post. Regards. -- Marlon
Marlon Dutra wrote: (anything about>> giants?)? > > > class htb 1:31 parent 1:1 leaf 31: prio 0 rate 1000bit ceil 750000bit > burst 15Kb cburst 1974b > Sent 11678670 bytes 2711 pkts (dropped 0, overlimits 0) > rate 1725Kbit 43pps backlog 13p > > As you can see,I see you snipped giants even though you were asked about them. Regardless 11678670/2711 > 1500 so specify your mtu next to every rate and ceil. Andy.
On 7/28/06, Andy Furniss <lists@andyfurniss.entadsl.com> wrote:> I see you snipped giants even though you were asked about them.Oh sorry. Here it goes: class htb 1:31 parent 1:1 leaf 31: prio 0 rate 1000bit ceil 750000bit burst 15Kb cburst 1875b Sent 23991629 bytes 5227 pkts (dropped 0, overlimits 0) rate 1567Kbit 44pps lended: 150 borrowed: 5077 giants: 3986 tokens: -14728330 ctokens: -21365> Regardless 11678670/2711 > 1500 so specify your mtu next to every rate > and ceil.I created the class with "mtu 1500" and the result is above, same behaviour. Afaik, the default mtu is 1600. By the way, my ethernet MTU is 1500. How can I prevent those giants? -- Marlon
Marlon Dutra wrote:> On 7/28/06, Andy Furniss <lists@andyfurniss.entadsl.com> wrote: > >> I see you snipped giants even though you were asked about them. > > > Oh sorry. Here it goes: > > class htb 1:31 parent 1:1 leaf 31: prio 0 rate 1000bit ceil 750000bit > burst 15Kb cburst 1875b > Sent 23991629 bytes 5227 pkts (dropped 0, overlimits 0) > rate 1567Kbit 44pps > lended: 150 borrowed: 5077 giants: 3986 > tokens: -14728330 ctokens: -21365 > >> Regardless 11678670/2711 > 1500 so specify your mtu next to every rate >> and ceil. > > > I created the class with "mtu 1500" and the result is above, same > behaviour. Afaik, the default mtu is 1600. > > By the way, my ethernet MTU is 1500. > > How can I prevent those giants?It seems there is a problem with some nics not obeying mtu then (I mean ifconfig/ip mtu) maybe it''s to do with tso or something. Normally you shouldn''t have to specify mtu to tc if you are running 1500 - you only need to if you are bigger, which it looks like you are whatever ifconfig/ip says. I would find/lookup what you are really using and tell htb that and then check that the giants count is 0. Andy.>
I believe there are some unresolved memory management issues with HTB. It looks like deleting the qdisc is causing some use after free or memory corruption problems. See: http://bugzilla.kernel.org/show_bug.cgi?id=6681 I don''t use HTB so I have no idea if this a new or old problem.
Stephen Hemminger wrote:> I believe there are some unresolved memory management issues with HTB. > It looks like deleting the qdisc is causing some use after free or > memory corruption problems. > > See: > http://bugzilla.kernel.org/show_bug.cgi?id=6681 > > I don''t use HTB so I have no idea if this a new or old problem. >Hmm he''s doing crash (and oops) in command like tc filter del dev ethX protocol ip parent 1:0 prio 5 handle X:XX:XX u32 is that valid - I''ve always reccomended to del root qdisc and start again to change something. I don''t know if it''s documented anywhere - none of the man pages in iproute2 even mention del (OK remove is there) and tc-filters.8 is referenced but not there. It does say in tc.8 - Filters have a three part ID, which is only needed when using a hashed filter hierarchy, for which see tc-filters(8) so maybe it''s just tc/kernel failing to give rtnetlink error - which is what I''ve seen before when trying to delete specific parts of trees. man tc-htb doesn''t even mention tc filter anything. Andy.
Just append "mtu 16500" at the end of your qdisc and class (especially class) statements. Like this: tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 750Kbit burst 15k mtu 16500 I had a similar problem on Dell 2850s with intel e1000 cards. The mtu on the e1000 interfaces was set to the default 1500, but the cards didn''t obey it all the time. I suspect that nVidia ethernet driver has similar problems. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today - it''s FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
Andy Furniss wrote:> Marlon Dutra wrote: > >> lended: 150 borrowed: 5077 giants: 3986 >> tokens: -14728330 ctokens: -21365 >> >>> Regardless 11678670/2711 > 1500 so specify your mtu next to every rate >>> and ceil. >> >> I created the class with "mtu 1500" and the result is above, same >> behaviour. Afaik, the default mtu is 1600. >> >> By the way, my ethernet MTU is 1500. >> >> How can I prevent those giants? > > > It seems there is a problem with some nics not obeying mtu then (I mean > ifconfig/ip mtu) maybe it''s to do with tso or something.Right, the "problem" is related to TCP segmentation offloading (or GSO in current kernels). The card gets large chunks of data and is responsible for creating MTU-sized packets, which essentially means qdiscs get to see those large chunks of data. You can disable TSO using ethtool (but it will cost you performance) or configure your qdisc appropriately.
Hello, On 7/28/06, Patrick McHardy <kaber@trash.net> wrote:> Right, the "problem" is related to TCP segmentation offloading (or GSO > in current kernels). The card gets large chunks of data and is > responsible for creating MTU-sized packets, which essentially means > qdiscs get to see those large chunks of data. You can disable TSO > using ethtool (but it will cost you performance) or configure your > qdisc appropriately.Thanks a lot. It was exactly that. I turned the tso off and HTB is working properly. How can I know the largest chunk of data the kernel sends to the card, so that I can configure qdisc appropriately? In the last post, Jake Altchill recommended using mtu 16500 in the qdiscs, but I''m not sure whether that''s a real or just a big number. Regards. -- Marlon