Hi Here is something that worked with with 2.6.10-1.771_FC2smp and stopped working when I upgraded to 2.6.16-1.2069_FC4smp. These are fedora kernels and the network controller is an Intel Gbit (e1000) running a 100 Mbps Full Duplex. Don''t know how or if this matters but the 2.6.10 kernel has CONFIG_X86_HZ=1000 and the 2.6.16 has CONFIG_HZ=250 The idea is to just shape to , say 2Mbit, a certain kind of traffic everything else should goes unshaped. # tc qdisc add dev eth0 root handle 1: htb default 20 # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit # tc qdisc add dev eth0 parent 1:2 sfq perturb 10 # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2 This was working as expected with 2.6.10 I''ve tried creating a proper 1:1 100Mbit parent to be the default but it didn''t help. And it was working fine without it on 2.6.10 With the 2.6.16 kernel I get results like # tc -s -d class show dev eth0 class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0 Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) rate 3484Kbit 45pps backlog 0b 0p requeues 0 lended: 20627 borrowed: 0 giants: 30926 tokens: -9768 ctokens: -9768 As you can see the correctly calculated rate is way above the ceiling and I''ve seen it go as fast as this certain type of traffic can go (something like 8-10Mbits), i.e. the class doesn''t seem to shape at all. No drops or overlimits. FWIW the cbq in 2.6.10 kernel was working fine as well until the controller changed from a 100Mbit it a Gbit one and thats why I switched to htb. The cbq in 2.6.16 is still similarly broken (in a different than htb way) Thanks in advance for any insight on this. Yanko
Yanko Kaneti wrote:> Hi > > Here is something that worked with with 2.6.10-1.771_FC2smp and stopped > working when I upgraded to 2.6.16-1.2069_FC4smp. > These are fedora kernels and the network controller is an Intel Gbit > (e1000) running a 100 Mbps Full Duplex. > Don''t know how or if this matters but the 2.6.10 kernel has > CONFIG_X86_HZ=1000 and the 2.6.16 has CONFIG_HZ=250 > > The idea is to just shape to , say 2Mbit, a certain kind of traffic > everything else should goes unshaped. > > # tc qdisc add dev eth0 root handle 1: htb default 20Why default 20 - if you don''t have 1:20 it would be better to use default 0 which is unshaped and is the default - so ommitting default is the same - unclassifed traffic goes through unshaped.> # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit > # tc qdisc add dev eth0 parent 1:2 sfq perturb 10 > # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2 > > This was working as expected with 2.6.10 > I''ve tried creating a proper 1:1 100Mbit parent to be the default but it > didn''t help. And it was working fine without it on 2.6.10 > > With the 2.6.16 kernel I get results like > > # tc -s -d class show dev eth0 > class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0 > Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) > rate 3484Kbit 45pps backlog 0b 0p requeues 0 > lended: 20627 borrowed: 0 giants: 30926The giants are the problem - if you specify mtu XXXXX on 1:2 class it should work. Or you could consider setting mtu on nic to 1500 if that is practical for you ie. this traffic is headed somewhere that is going to frag it down to 1500 anyway. Andy.
On Sun, 2006-04-16 at 00:13 +0100, Andy Furniss wrote:> Yanko Kaneti wrote: > > Hi > > > > Here is something that worked with with 2.6.10-1.771_FC2smp and stopped > > working when I upgraded to 2.6.16-1.2069_FC4smp. > > These are fedora kernels and the network controller is an Intel Gbit > > (e1000) running a 100 Mbps Full Duplex. > > Don''t know how or if this matters but the 2.6.10 kernel has > > CONFIG_X86_HZ=1000 and the 2.6.16 has CONFIG_HZ=250 > > > > The idea is to just shape to , say 2Mbit, a certain kind of traffic > > everything else should goes unshaped. > > > > # tc qdisc add dev eth0 root handle 1: htb default 20 > > Why default 20 - if you don''t have 1:20 it would be better to use > default 0 which is unshaped and is the default - so ommitting default is > the same - unclassifed traffic goes through unshaped.No reason. I obviously missed the explanation for the 0 class. Will omit default in the future.> > # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit > > # tc qdisc add dev eth0 parent 1:2 sfq perturb 10 > > # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2 > > > > This was working as expected with 2.6.10 > > I''ve tried creating a proper 1:1 100Mbit parent to be the default but it > > didn''t help. And it was working fine without it on 2.6.10 > > > > With the 2.6.16 kernel I get results like > > > > # tc -s -d class show dev eth0 > > class htb 1:2 root leaf 800b: prio 0 quantum 25000 rate 2000Kbit ceil 2000Kbit burst 2600b/8 mpu 0b overhead 0b cburst 2600b/8 mpu 0b overhead 0b level 0 > > Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) > > rate 3484Kbit 45pps backlog 0b 0p requeues 0 > > lended: 20627 borrowed: 0 giants: 30926 > > The giants are the problem - if you specify mtu XXXXX on 1:2 class it > should work.> Or you could consider setting mtu on nic to 1500 if that is practical > for you ie. this traffic is headed somewhere that is going to frag it > down to 1500 anyway.Setting mtu 16500 for the class fixed it. But I wonder where did these giants come from in the first place? The mtu of the interface is and was 1500. Or so ifconfig and ip link tell me. Or this is some other mtu we are talking about... Thanks Yanko
Yanko Kaneti wrote:> Setting mtu 16500 for the class fixed it. But I wonder where did these > giants come from in the first place? The mtu of the interface is and was > 1500. Or so ifconfig and ip link tell me. Or this is some other mtu we > are talking about...Hmm I didn''t expect that - maybe there is some problem with the nic drivers not obeying kernel - is there any tso offload etc. at work here ? Andy.
On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote:> Yanko Kaneti wrote: > > > Setting mtu 16500 for the class fixed it. But I wonder where did these > > giants come from in the first place? The mtu of the interface is and was > > 1500. Or so ifconfig and ip link tell me. Or this is some other mtu we > > are talking about... > > Hmm I didn''t expect that - maybe there is some problem with the nic > drivers not obeying kernel - is there any tso offload etc. at work here ?Yes and its on by default. The interface mtu still says 1500. I''ve tried deleting and attaching the qdisc+class (without explicit large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it doesnt seem to matter - giants appear in both cases. With large mtu for the class no giants with both tso on and off. Yanko
Yanko Kaneti wrote:> On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: > >>Yanko Kaneti wrote: >> >> >>>Setting mtu 16500 for the class fixed it. But I wonder where did these >>>giants come from in the first place? The mtu of the interface is and was >>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we >>>are talking about... >> >>Hmm I didn''t expect that - maybe there is some problem with the nic >>drivers not obeying kernel - is there any tso offload etc. at work here ? > > > Yes and its on by default. The interface mtu still says 1500. > I''ve tried deleting and attaching the qdisc+class (without explicit > large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it > doesnt seem to matter - giants appear in both cases. > With large mtu for the class no giants with both tso on and off. > > Yanko > >I think you need to ask fedora or intel driver maintainer about this. AIUI tso is not in vanilla kernels and the patches are quite invasive. Andy.
On Sun, 2006-04-16 at 19:40 +0100, Andy Furniss wrote:> Yanko Kaneti wrote: > > On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: > > > >>Yanko Kaneti wrote: > >> > >> > >>>Setting mtu 16500 for the class fixed it. But I wonder where did these > >>>giants come from in the first place? The mtu of the interface is and was > >>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we > >>>are talking about... > >> > >>Hmm I didn''t expect that - maybe there is some problem with the nic > >>drivers not obeying kernel - is there any tso offload etc. at work here ? > > > > > > Yes and its on by default. The interface mtu still says 1500. > > I''ve tried deleting and attaching the qdisc+class (without explicit > > large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it > > doesnt seem to matter - giants appear in both cases. > > With large mtu for the class no giants with both tso on and off. > > > > I think you need to ask fedora or intel driver maintainer about this. > AIUI tso is not in vanilla kernels and the patches are quite invasive.Well, as much as google tells me TSO has been in the kernel and enabled since 2.5.33 and e1000 was the first driver to support it. The FC4 2.6.16 kernel doesn''t have any tso related patches as can be seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/ Since my immediate problem was solved with the mtu param I plan on forgetting about htb and traffic control in general for the time being :) Thanks again. Cheers Yanko
Yanko Kaneti wrote:>>I think you need to ask fedora or intel driver maintainer about this. >>AIUI tso is not in vanilla kernels and the patches are quite invasive. > > > Well, as much as google tells me TSO has been in the kernel and enabled > since 2.5.33 and e1000 was the first driver to support it. > The FC4 2.6.16 kernel doesn''t have any tso related patches as can be > seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/Ahh OK I must have been thinking of another card. Andy.
Yanko Kaneti wrote:> On Sun, 2006-04-16 at 19:40 +0100, Andy Furniss wrote: > >>Yanko Kaneti wrote: >> >>>On Sun, 2006-04-16 at 03:03 +0100, Andy Furniss wrote: >>> >>> >>>>Yanko Kaneti wrote: >>>> >>>> >>>> >>>>>Setting mtu 16500 for the class fixed it. But I wonder where did these >>>>>giants come from in the first place? The mtu of the interface is and was >>>>>1500. Or so ifconfig and ip link tell me. Or this is some other mtu we >>>>>are talking about... >>>> >>>>Hmm I didn''t expect that - maybe there is some problem with the nic >>>>drivers not obeying kernel - is there any tso offload etc. at work here ? >>> >>> >>>Yes and its on by default. The interface mtu still says 1500. >>>I''ve tried deleting and attaching the qdisc+class (without explicit >>>large mtu) with both tso on (ethtool -K eth0 tso on) and tso off , it >>>doesnt seem to matter - giants appear in both cases. >>>With large mtu for the class no giants with both tso on and off. >>> >> >>I think you need to ask fedora or intel driver maintainer about this. >>AIUI tso is not in vanilla kernels and the patches are quite invasive. > > > Well, as much as google tells me TSO has been in the kernel and enabled > since 2.5.33 and e1000 was the first driver to support it. > The FC4 2.6.16 kernel doesn''t have any tso related patches as can be > seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/ > > Since my immediate problem was solved with the mtu param I plan on > forgetting about htb and traffic control in general for the time > being :) Thanks again.One more thing I just thought - sfq sets its quantum from the dev mtu. While I always thought that the "must be >=mtu" comment in the source was a bit OTT, it still "should" be >= mtu for the drr to be 0(1) for cpu work. You can set it explicitly by adding quantum=X on the sfq line. For ethernet X is mtu + 14. Andy.
Andy Furniss wrote:>> Well, as much as google tells me TSO has been in the kernel and enabled >> since 2.5.33 and e1000 was the first driver to support it. The FC4 >> 2.6.16 kernel doesn''t have any tso related patches as can be >> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/ >> >> Since my immediate problem was solved with the mtu param I plan on >> forgetting about htb and traffic control in general for the time >> being :) Thanks again. > > > One more thing I just thought - sfq sets its quantum from the dev mtu.One more possibility: current kernels support UDP fragmentation offload (UFO), which has similar effects as TSO. The in-tree e1000 driver doesn''t support it, but maybe the fedora one does. Changes in the fragmentation behaviour of conntrack in 2.6.16 could also be responsible (if you''re using it). Can you please post your NAT and marking rules, routing rules etc?
On Mon, 2006-04-17 at 21:32 +0200, Patrick McHardy wrote:> Andy Furniss wrote: > >> Well, as much as google tells me TSO has been in the kernel and enabled > >> since 2.5.33 and e1000 was the first driver to support it. The FC4 > >> 2.6.16 kernel doesn''t have any tso related patches as can be > >> seen here http://cvs.fedora.redhat.com/viewcvs/rpms/kernel/FC-4/ > >> > >> Since my immediate problem was solved with the mtu param I plan on > >> forgetting about htb and traffic control in general for the time > >> being :) Thanks again. > > > > > > One more thing I just thought - sfq sets its quantum from the dev mtu.Riiight. I should have tried without the sfq earlier. Without it this works as expected without explicit mtu setting for the htb class. And no giants. # tc qdisc add dev eth0 root handle 1: htb # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2> One more possibility: current kernels support UDP fragmentation offload > (UFO), which has similar effects as TSO. The in-tree e1000 driver > doesn''t support it, but maybe the fedora one does.No mention of ufo or e1000 in any of the patches that can be found in the url above.> Changes in the fragmentation behaviour of conntrack in 2.6.16 could also > be responsible (if you''re using it). Can you please post your NAT and > marking rules, routing rules etc?Here are the "interesting" rules where the packets in question pass. Have no rules other than the ones in the mangle table mangle OUTPUT -m owner --uid-owner shaped -j userchain userchain .... -m length --length 512:65535 -j MARK --set-mark 0x32 -> the fw filter
Yanko Kaneti wrote:>>>One more thing I just thought - sfq sets its quantum from the dev mtu. > > > Riiight. I should have tried without the sfq earlier. Without it this > works as expected without explicit mtu setting for the htb class. And no > giants. > > # tc qdisc add dev eth0 root handle 1: htb > # tc class add dev eth0 parent 1: classid 1:2 htb rate 2Mbit > # tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 50 fw flowid 1:2I wouldn''t have expected that to make any difference to the giants. Looking again at your stats - Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) rate 3484Kbit 45pps backlog 0b 0p requeues 0 lended: 20627 borrowed: 0 giants: 30926 tokens: -9768 ctokens: -9768 The giants count is higher than the packet count so now I am really confused. Andy.
Andy Furniss wrote:> Looking again at your stats - > > Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) > rate 3484Kbit 45pps backlog 0b 0p requeues 0 > lended: 20627 borrowed: 0 giants: 30926 > tokens: -9768 ctokens: -9768 > > The giants count is higher than the packet count so now I am really > confused.Doh - I suppose thats just the way HTB counts so you add them together. Andy.
Andy Furniss wrote:> Andy Furniss wrote: > >> Looking again at your stats - >> >> Sent 189796883 bytes 20626 pkt (dropped 0, overlimits 0 requeues 0) >> rate 3484Kbit 45pps backlog 0b 0p requeues 0 >> lended: 20627 borrowed: 0 giants: 30926 >> tokens: -9768 ctokens: -9768 >> >> The giants count is higher than the packet count so now I am really >> confused. > > > Doh - I suppose thats just the way HTB counts so you add them together.LOL - Third try, testing on lo which is confusing and maybe misleading but it looks like the giants count gets doubled up but the packet count doesn''t. Andy.
Possibly Parallel Threads
- tc shown rate larger than ceil (was "Weird rate in HTB")
- Weird HTB behaviour in 2.6.17
- [Bug 101418] New: Build failure in GNOME Continuous
- TC (HTB) doesn''t work well when network is congested?
- Re: Fwd: Does Set Block IO tune take effect in the same session.