Dear all,
I''m having real problems getting tc to do anything useful at all.
I''m
also under pressure to get this fixed before the students start arriving
later this month (I work in a university).
In short, I want each IP address to be hard limited to 128kbit down,
64kbit up, never to be allowed more bandwidth than this. It is also
important that the latency remains reasonably low - maybe this implies a
need to apply some sort of traffic filtering and classifying. I did
manage to get a script semi-working but as soon as any decent bandwidth
started flowing on the connection, the latency jumped up to >4000ms.
I tried to change my script to make it more classful and intelligent but
I ended up breaking it and now it doesn''t work at all. (Upon execution,
I get ''172.19.123.254 Illegal "match"'') I''m
inexperienced with tc so I
don''t really know the best way to design such a system as this. I also
struggle with the tc syntax. I only know what I need the end result to be.
I''d be very grateful if anyone could lend a hand to help me get this
working in time for the start of term. I''ve attached my script at the
end of this email.
Cheers,
Jonathan
#!/bin/sh
# Interfaces
LAN=eth0
WAN=eth1
# Maximum global uplink and downlink in mbit/s
GLOBAL_DOWN=100
GLOBAL_UP=100
# Maximum per-user download & upload speed in kbit/s
DOWNLINK=128
UPLINK=64
UPLINK=$((UPLINK/4)) # required because the old rate wasn''t
accurate
# IP range in each subnet
LOW_IP=2
HIGH_IP=254
#-----------------Don''t mess with stuff below---------------|
#-----------------this line or you''ll break it--------------|
# Flush existing rules
tc qdisc del dev $LAN root
# tc qdisc del dev $WAN root
# Create root class for 100mbit interface - total traffic can''t exceed
this
tc qdisc add dev $LAN root handle 1: cbq avpkt 1000 bandwidth
${GLOBAL_DOWN}mbit
tc qdisc add dev $LAN ingress handle ffff:
# Set useful counter
total=0
# Apply rules for all included subnets
for i in `seq $LOW_IP $HIGH_IP`
do
total=$((total+1))
echo 172.19.123.$i
tc class add dev $LAN parent 1: classid 1:$total cbq rate
${DOWNLINK}kbit allot 1600 prio 1 bounded isolated
tc filter add dev $LAN parent 1: protocol ip prio 1 u32 match ip dst
172.19.123.$i flowid 1:$total
tc filter add dev $LAN parent ffff: protocol ip prio 50 u32 match ip
src 172.19.123.$i police rate ${UPLINK}kbit burst 10k drop flowid :1
tc filter add dev $LAN parent 1: protocol ip prio 11 u32 match ip
protocol 1 0xff flowid 1:$total match ip protocol 6 0xff match u8 0x05
0x0f at 0 match u16 0x0000 0xffc0 at 2 match u8 0x10 0xff at 33 flowid
1:$total
done
------------------------
Jonathan Gazeley
ResNet | Wireless & VPN Team
Information Systems & Computing
University of Bristol
------------------------
Jonathan Gazeley wrote:> Dear all, > > I''m having real problems getting tc to do anything useful at all. I''m > also under pressure to get this fixed before the students start arriving > later this month (I work in a university). > > In short, I want each IP address to be hard limited to 128kbit down, > 64kbit up, never to be allowed more bandwidth than this. It is also > important that the latency remains reasonably low - maybe this implies a > need to apply some sort of traffic filtering and classifying. I did > manage to get a script semi-working but as soon as any decent bandwidth > started flowing on the connection, the latency jumped up to >4000ms. > > I tried to change my script to make it more classful and intelligent but > I ended up breaking it and now it doesn''t work at all. (Upon execution, > I get ''172.19.123.254 Illegal "match"'') I''m inexperienced with tc so I > don''t really know the best way to design such a system as this. I also > struggle with the tc syntax. I only know what I need the end result to be. > > I''d be very grateful if anyone could lend a hand to help me get this > working in time for the start of term. I''ve attached my script at the > end of this email.I''ve never used cbq so don''t know how well it will do this, maybe htb would be better. maybe hfsc better still. I don''t know why you need /4 for the policers, perhaps if you tested on a lan with a short buffer like 10k you were seeing the rate before the drops or something. TCP doesn''t much like policers with LAN latency and short burst - it''s not so bad with WAN latency, but if you have a 100meg to JANET maybe your WAN latency can be quite low aswell. If you have a multicore/smp CPU you shouldn''t use CPU as a clocksource. You need a - tc qdisc del dev $LAN ingress> # Create root class for 100mbit interface - total traffic can''t exceed thisIf you have a 100meg nic then 100mbit here is a bit high as the 100mbit line rate includes overheads not seen by TC.> tc filter add dev $LAN parent 1: protocol ip prio 11 u32 match ip > protocol 1 0xff flowid 1:$total match ip protocol 6 0xff match u8 0x05 > 0x0f at 0 match u16 0x0000 0xffc0 at 2 match u8 0x10 0xff at 33 flowid > 1:$totalThis is the problem giving the error it looks like two rules but one got partially deleted and lost the newline. tc filter add dev $LAN parent 1: protocol ip prio 11 u32 match ip protocol 1 0xff flowid 1:$total tc filter add dev $LAN parent 1: protocol ip prio 11 u32 match ip protocol 6 0xff match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 match u8 0x10 0xff at 33 flowid 1:$total They look redundant to me in this setup, though as you already filter by ip address to 1:$total, so to do anything useful ICMP and and small tcp+whatever the otherbits match would need to go to different classes. I would include UDP as interactive aswell, though there are exeptions, but it shouldn''t matter if you keep it within each users. If I have time later I''ll test how I would do it. Andy.
Andy Furniss wrote:> If I have time later I''ll test how I would do it.I managed to have a play - CBQ doesn''t seem too accurate it let netperf get throughput of about 180kbit. HTB was OK so I used that. Below is what I tested - I wouldn''t consider it finished because it would probably be nicer to have SFQs on the bulk classes and something shorter on the interactives. I don''t know how much memory this does/could use, if you don''t specify child qdiscs htb uses pfifos with a length taken from txqueuelen (1000 on eth) so that adds up to quite a bit. With window scaling on and a netperf running for each IP I managed to backlog >200 packets on each. Rather than police you could, if using recentish 2.6 use ifb and have the same setup on ingress eth0. Or if you don''t do nat on the same box on the wan. If you do do nat and don''t have ifb then you need to use netfilter to mark by ip and match the marks. If the hosts are wireless, then there may be other ways to make things better - not that I have wireless myself, but if there is much packet loss I always thought it would be better to proxy wan and have different MTU/MSS for the wlan - maybe also use one of the tcp congestion controls that''s less sensitive to random loss. It would be more elegant to use tc''s hashing but I''ve not done that before. The filters are nested so only the IP matches see upto all the traffic. I just matched tcp length <128 / not tcp for interactive. If you want counters for filter hits tc -s filter ls dev eth0 for top level tc -s filter ls dev eth0 parent 1:1 for the children tc -s class ls dev eth0 for loads of htb data - beware the rates use a long average, it takes 100sec for them to be right for me. Andy !/bin/sh #set -x # Interfaces LAN=eth0 DOWNLINK=128 # IP range in each subnet LOW_IP=2 HIGH_IP=254 # Flush existing rules tc qdisc del dev $LAN root tc qdisc add dev $LAN root handle 1: htb # Set useful counter total=0 # Apply rules for all included subnets for i in `seq $LOW_IP $HIGH_IP` do total=$((total+1)) echo 172.19.123.$i tc class add dev $LAN parent 1: classid 1:$total htb rate ${DOWNLINK}kbit tc class add dev $LAN parent 1:$total classid 1:a$total htb rate 100kbit ceil ${DOWNLINK}kbit prio 0 tc class add dev $LAN parent 1:$total classid 1:b$total htb rate 28kbit ceil ${DOWNLINK}kbit prio 1 tc filter add dev $LAN parent 1: protocol ip prio 1 u32 match ip src 172.19.123.$i flowid 1:$total tc filter add dev $LAN parent 1:$total protocol ip prio 2 u32 match ip protocol 6 0xff match u16 0x0000 0xff80 at 2 flowid 1:a$total tc filter add dev $LAN parent 1:$total protocol ip prio 3 u32 match ip protocol 6 0xff flowid 1:b$total tc filter add dev $LAN parent 1:$total protocol ip prio 4 u32 match u32 0 0 flowid 1:a$total done
Hi Andy, Thanks a bunch for your help - really good of you to put time into helping a newbie. Andy Furniss wrote:> I managed to have a play - CBQ doesn''t seem too accurate it let > netperf get throughput of about 180kbit. HTB was OK so I used that.I was only using CBQ because that''s what was being used in the tutorials and howtos I looked at - there doesn''t seem to be a massive amount of documentation out there. My script was basically a copy & paste job... If you recommend HTB then I''ll give it a try.> If you have a multicore/smp CPU you shouldn''t use CPU as a > clocksource. I don''t know how much memory this does/could use, if you > don''t specify child qdiscs htb uses pfifos with a length taken from > txqueuelen (1000 on eth) so that adds up to quite a bit. With window > scaling on and a netperf running for each IP I managed to backlog >200 > packets on each.It runs on relatively sporty hardware and doesn''t do anything other than NAT and shaping, so I don''t think memory usage is really a problem. It has dual processors so I guess that means I shouldnt use CPU as a clocksource.> Rather than police you could, if using recentish 2.6 use ifb and have > the same setup on ingress eth0. Or if you don''t do nat on the same box > on the wan. If you do do nat and don''t have ifb then you need to use > netfilter to mark by ip and match the marks.This box is also a NAT box, so I''ll do marking with iptables to sort the incoming traffic. I read about it somewhere on Google so I''m sure I can manage! If I''m marking packets with iptables, would it be better to shape them as they leave on the internal interface, rather than doing something with ingress on the external interface? What is ifb? For once, Google doesn''t seem to turn up much. Cheers, Jonathan ------------------------ Jonathan Gazeley ResNet | Wireless & VPN Team Information Systems & Computing University of Bristol ------------------------
Hi Jonathan, On 31 Aug 2007, jonathan.gazeley@bristol.ac.uk wrote:> > Hi Andy, > > Thanks a bunch for your help - really good of you to put time into > helping a newbie. > > Andy Furniss wrote: >> Rather than police you could, if using recentish 2.6 use ifb and >> have the same setup on ingress eth0. Or if you don''t do nat on the >> same box on the wan. If you do do nat and don''t have ifb then you >> need to use netfilter to mark by ip and match the marks. > > This box is also a NAT box, so I''ll do marking with iptables to sort > the incoming traffic. I read about it somewhere on Google so I''m sure > I can manage! If I''m marking packets with iptables, would it be better > to shape them as they leave on the internal interface, rather than > doing something with ingress on the external interface?As long as you do not want to shape traffic from/to the box itself, the easiest solution is to shape on egress (at least if you only have two interfaces). If you want to also shape traffic from/to the box itself and don''t do NAT, you can use ifb. ifb is a pseudo device you can redirect incoming traffic to, using tc. then you can attach egress qdisc to that pseudo device. If you want to also shape traffic from/to the box itself and do NAT, you still have to use IMQ (http://www.linuximq.net/) - I think. The latest kernel I have good results with IMQ ist 2.6.18. Greetings Jens
Jonathan Gazeley wrote:> Hi Andy, > > Thanks a bunch for your help - really good of you to put time into > helping a newbie. > > Andy Furniss wrote: >> I managed to have a play - CBQ doesn''t seem too accurate it let >> netperf get throughput of about 180kbit. HTB was OK so I used that. > I was only using CBQ because that''s what was being used in the tutorials > and howtos I looked at - there doesn''t seem to be a massive amount of > documentation out there. My script was basically a copy & paste job... > If you recommend HTB then I''ll give it a try.OK - I just noticed I forgot to change ... match ip src 172.19.123.$i ... back to match dst so if you copy & paste mine you''ll need to.>> If you have a multicore/smp CPU you shouldn''t use CPU as a >> clocksource. I don''t know how much memory this does/could use, if you >> don''t specify child qdiscs htb uses pfifos with a length taken from >> txqueuelen (1000 on eth) so that adds up to quite a bit. With window >> scaling on and a netperf running for each IP I managed to backlog >200 >> packets on each. > It runs on relatively sporty hardware and doesn''t do anything other than > NAT and shaping, so I don''t think memory usage is really a problem. It > has dual processors so I guess that means I shouldnt use CPU as a > clocksource.Yes I guess so - I havent got anything other than uniprocessors so don''t know what''s best. Whatever you choose I would still use Hz 1000 as HTB is a bit more accurate that way. I also haven''t tried no Hz yet so don''t know what that''s like.>> Rather than police you could, if using recentish 2.6 use ifb and have >> the same setup on ingress eth0. Or if you don''t do nat on the same box >> on the wan. If you do do nat and don''t have ifb then you need to use >> netfilter to mark by ip and match the marks. > This box is also a NAT box, so I''ll do marking with iptables to sort the > incoming traffic. I read about it somewhere on Google so I''m sure I can > manage! If I''m marking packets with iptables, would it be better to > shape them as they leave on the internal interface, rather than doing > something with ingress on the external interface? > > What is ifb? For once, Google doesn''t seem to turn up much.You have it already if modprobe ifb ip link set up dev ifb0 works for you, if not it''s called intermediate functional block under netdevices in kernel config. If you had ifb then you could just use the same script with the different rates and tc src match. No need to use netfilter as you could direct the incoming traffic on eth0 to the ifb0 device with a tc filter and it would be shaped before any NAT happened. tc filter add dev eth0 ingress tc filter add dev eth0 parent ffff: protocol ip u32 match ip src 172.17.123.0/24 flowid 1:0 action mirred egress redirect dev ifb0 then just change the script so LAN=ifb0, the rates and ... match ip src ... But as you say you can also easily shape on wan and use netfilter marks. I assume your box only forwards traffic for 172.17.123.0/24 if there is traffic to and from the box its self then you need to decide whether it needs shaping aswell and make a filter to exempt it, or if it is to be shaped then you need to think about whether the nic, if it''s gig eht does tcp segmentation offload. If it does you can turn it off with ethtool -k. Andy.> > Cheers, > Jonathan > > ------------------------ > Jonathan Gazeley > ResNet | Wireless & VPN Team > Information Systems & Computing > University of Bristol > ------------------------ > >
Andy, Thanks for your script. I''ve been looking at it a lot but still can''t get it to work in the way I need it to. While the script runs without errors echoed to ssh, it doesn''t have the desired effect. The clients have a downlink shaped to 128kbit. However, each of my two test clients will happily download at rates much higher than 128kbit. Client 1 gets a rate of 160kB/s and Client 2 gets a rate of 250kB/s (yes, kilobytes, not kilobit). I have no idea why this is and it''s starting to get confusing. I know you mentioned that CPU timing wouldn''t be reliable on dual-core CPUs (I have dual core) but you''d think the rate would be out by a factor of 2. Here, I''m getting a factor of 10 for one client and a factor of 16 for the other. When I actually went to the console of the box earlier, there was loads of output from your script echoed to the console, each one saying that the quantum was too small and I should consider adjusting r2q. I don''t really understand what this means. Any advice gratefully received (from anyone else on the list as well as Andy!) Cheers, Jonathan Andy Furniss wrote:> I managed to have a play - CBQ doesn''t seem too accurate it let > netperf get throughput of about 180kbit. HTB was OK so I used that. > > Below is what I tested - I wouldn''t consider it finished because it > would probably be nicer to have SFQs on the bulk classes and something > shorter on the interactives. > > I don''t know how much memory this does/could use, if you don''t specify > child qdiscs htb uses pfifos with a length taken from txqueuelen (1000 > on eth) so that adds up to quite a bit. With window scaling on and a > netperf running for each IP I managed to backlog >200 packets on each. > > Rather than police you could, if using recentish 2.6 use ifb and have > the same setup on ingress eth0. Or if you don''t do nat on the same box > on the wan. If you do do nat and don''t have ifb then you need to use > netfilter to mark by ip and match the marks. > > If the hosts are wireless, then there may be other ways to make things > better - not that I have wireless myself, but if there is much packet > loss I always thought it would be better to proxy wan and have > different MTU/MSS for the wlan - maybe also use one of the tcp > congestion controls that''s less sensitive to random loss. > > It would be more elegant to use tc''s hashing but I''ve not done that > before. The filters are nested so only the IP matches see upto all the > traffic. I just matched tcp length <128 / not tcp for interactive. > > If you want counters for filter hits > > tc -s filter ls dev eth0 > for top level > > tc -s filter ls dev eth0 parent 1:1 > for the children > > tc -s class ls dev eth0 > for loads of htb data - beware the rates use a long average, it takes > 100sec for them to be right for me. > > Andy > > !/bin/sh > #set -x > > # Interfaces > LAN=eth0 > DOWNLINK=128 > > # IP range in each subnet > LOW_IP=2 > HIGH_IP=254 > > # Flush existing rules > tc qdisc del dev $LAN root > > tc qdisc add dev $LAN root handle 1: htb > > # Set useful counter > total=0 > > # Apply rules for all included subnets > for i in `seq $LOW_IP $HIGH_IP` > do > total=$((total+1)) > echo 172.19.123.$i > tc class add dev $LAN parent 1: classid 1:$total htb rate > ${DOWNLINK}kbit > tc class add dev $LAN parent 1:$total classid 1:a$total htb rate > 100kbit ceil ${DOWNLINK}kbit prio 0 > tc class add dev $LAN parent 1:$total classid 1:b$total htb rate > 28kbit ceil ${DOWNLINK}kbit prio 1 > tc filter add dev $LAN parent 1: protocol ip prio 1 u32 match ip src > 172.19.123.$i flowid 1:$total > tc filter add dev $LAN parent 1:$total protocol ip prio 2 u32 match > ip protocol 6 0xff match u16 0x0000 0xff80 at 2 flowid 1:a$total > tc filter add dev $LAN parent 1:$total protocol ip prio 3 u32 match > ip protocol 6 0xff flowid 1:b$total > tc filter add dev $LAN parent 1:$total protocol ip prio 4 u32 match > u32 0 0 flowid 1:a$total > done > > > >-- ------------------------ Jonathan Gazeley ResNet | Wireless & VPN Team Information Systems & Computing University of Bristol ------------------------
Jonathan Gazeley wrote:> Andy, > > Thanks for your script. I''ve been looking at it a lot but still can''t > get it to work in the way I need it to. While the script runs without > errors echoed to ssh, it doesn''t have the desired effect. The clients > have a downlink shaped to 128kbit. However, each of my two test clients > will happily download at rates much higher than 128kbit. Client 1 gets a > rate of 160kB/s and Client 2 gets a rate of 250kB/s (yes, kilobytes, not > kilobit). I have no idea why this is and it''s starting to get confusing. > I know you mentioned that CPU timing wouldn''t be reliable on dual-core > CPUs (I have dual core) but you''d think the rate would be out by a > factor of 2. Here, I''m getting a factor of 10 for one client and a > factor of 16 for the other.I don''t know how much the CPU timing would affect things. I assume you changed src to dst in the match. Are you testing downloads from the wan or hosted on the box doing the shaping? After an overrate download can you post the output of - tc -s class ls dev eth0 | grep "parent 1:X" -A 4 changing the X in "parent 1:X" to one less than the last part of the IP address of the test machine.> > When I actually went to the console of the box earlier, there was loads > of output from your script echoed to the console, each one saying that > the quantum was too small and I should consider adjusting r2q. I don''t > really understand what this means.That shouldn''t really hurt too much, to fix it you could ether add quantum 1514 to each of the two child classes or r2q 2 to the root class. FWIW I tried browsing with this setup and downloading aswell and it''s not very nice with such a long queue. An ISP/teleco would never have that long a fifo on a 128kbit line. It would be better to add an sfq to the bulk class or limit the length of the fifo. Andy.
Andy Furniss wrote:> I assume you changed src to dst in the match.Yes I did this some time ago.> Are you testing downloads from the wan or hosted on the box doing the > shaping?All downloads are coming from the WAN via eth1 - nothing else runs on this box except NAT and shaping.> After an overrate download can you post the output of - > > tc -s class ls dev eth0 | grep "parent 1:X" -A 4 > > changing the X in "parent 1:X" to one less than the last part of the > IP address of the test machine.Results appended. Sadly everything seems to show zero bytes. I checked that everything is flowing through the correct interface, and yes it appears to do so. This box has 3 interfaces. eth0 is the one facing the NAT clients, eth1 is the public internet connection and eth2 is a management interface. Only ssh traffic to my workstation is flowing on eth2. I see two-way FTP traffic flowing on eth1 (I am downloading large files from an FTP server for testing) but on eth0, i only see traffic LEAVING the NAT clients and entering my shaping box. I have no idea why I don''t see traffic with destination of the NAT clients. There is no way that the traffic could be "leaking" around my shaping box - eth0 is physically connected to a 5 port switch with my test clients on.>> When I actually went to the console of the box earlier, there was >> loads of output from your script echoed to the console, each one >> saying that the quantum was too small and I should consider adjusting >> r2q. I don''t really understand what this means. > That shouldn''t really hurt too much, to fix it you could ether add > quantum 1514 to each of the two child classes or r2q 2 to the root class.Adding r2q seemed to work.> FWIW I tried browsing with this setup and downloading aswell and it''s > not very nice with such a long queue. An ISP/teleco would never have > that long a fifo on a 128kbit line. It would be better to add an sfq > to the bulk class or limit the length of the fifo.Sorry for such a simplistic question - how might this be done? However I''m not overly worried about making the quality of service perfect - I just need it to be limited bandwidth. The reason why I want such a restricted service is because the shaped NAT clients of this box are going to be the "naughty" users on the university network who will have their 100Mbit service reduced to 128kbit for a week as punishment for rulebreaking etc. So if people complain about the occasional dropped packet - they shouldn''t have been downloading copyright material in the first place! ------------------------ Jonathan Gazeley ResNet | Wireless & VPN Team Information Systems & Computing University of Bristol ------------------------ [root@pokey ~]# tc -s class ls dev eth0 | grep "parent 1:14" -A 4 class htb 1:b148 parent 1:148 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:a149 parent 1:149 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a148 parent 1:148 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:b149 parent 1:149 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:b146 parent 1:146 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:a147 parent 1:147 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a146 parent 1:146 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:b147 parent 1:147 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:b144 parent 1:144 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:a145 parent 1:145 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a144 parent 1:144 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:b145 parent 1:145 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:b142 parent 1:142 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:a143 parent 1:143 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a142 parent 1:142 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:b143 parent 1:143 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:b14 parent 1:14 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:b140 parent 1:140 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000 -- class htb 1:a141 parent 1:141 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a14 parent 1:14 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:a140 parent 1:140 prio 0 rate 100000bit ceil 128000bit burst 1612b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 128960 ctokens: 101000 -- class htb 1:b141 parent 1:141 prio 1 rate 28000bit ceil 128000bit burst 1603b cburst 1616b Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 458000 ctokens: 101000
Jonathan Gazeley wrote:> Results appended. Sadly everything seems to show zero bytes. I checked > that everything is flowing through the correct interface, and yes it > appears to do so. This box has 3 interfaces. eth0 is the one facing the > NAT clients, eth1 is the public internet connection and eth2 is a > management interface. Only ssh traffic to my workstation is flowing on > eth2. I see two-way FTP traffic flowing on eth1 (I am downloading large > files from an FTP server for testing) but on eth0, i only see traffic > LEAVING the NAT clients and entering my shaping box. I have no idea why > I don''t see traffic with destination of the NAT clients. There is no way > that the traffic could be "leaking" around my shaping box - eth0 is > physically connected to a 5 port switch with my test clients on.That''s strange - so there are no vlans involved and tcpdump -nnei eth0 only sees traffic coming in? If the nic is a gig eth, I wonder if its drivers are doing something strange to do with segmentation offload. You should be able to turn it off with ethtool. ethtool -k eth0 to see if it is on and turn it off with ethtool -k eth0 tso off>> FWIW I tried browsing with this setup and downloading aswell and it''s >> not very nice with such a long queue. An ISP/teleco would never have >> that long a fifo on a 128kbit line. It would be better to add an sfq >> to the bulk class or limit the length of the fifo. > Sorry for such a simplistic question - how might this be done? However > I''m not overly worried about making the quality of service perfect - I > just need it to be limited bandwidth. The reason why I want such a > restricted service is because the shaped NAT clients of this box are > going to be the "naughty" users on the university network who will have > their 100Mbit service reduced to 128kbit for a week as punishment for > rulebreaking etc. So if people complain about the occasional dropped > packet - they shouldn''t have been downloading copyright material in the > first place!Ahh I see - actually packet loss could make things nicer by getting the tcp senders congestion control to back off. The command for sfq would be tc qdisc add dev $LAN parent 1:b$total handle b$total: sfq limit 30 in the loop under the bulk class. You could just have pfifo instead of sfq if you wanted, and the limit of 30 can be changed. sfq has a parameter called perturb, which can make it fairer at the expense of packet reordering, but it can also make single downloads back off a bit too much, so I didn''t use it. The script is not the best example bash wise as I just adapted it from the one you started with. It could be done better - eg my mixing of hex and decimal for class ids is not ideal and total is not needed as such, but it should work. Andy.
Andy Furniss wrote:> That''s strange - so there are no vlans involved and tcpdump -nnei eth0 > only sees traffic coming in?Nope, no vlans. And yes, both tcpdump and tethereal see only on the outgoing (i.e. clients to internet) ack packets, but not the actual incoming data packets. Like I said, it can''t be a routing problem because the only physical route those packets can take between the NAT clients and the internet is via eth0 on this box. All four interfaces on the box I''m using are Intel ones, so not just some cheap rubbish! Probably unlikely to be drivers too?> > If the nic is a gig eth, I wonder if its drivers are doing something > strange to do with segmentation offload. You should be able to turn it > off with ethtool. > > ethtool -k eth0 to see if it is on and turn it off with > > ethtool -k eth0 tso offAll the interfaces are gigabit. Segmentation offload was enabled on eth0 - now it''s off but appears to have had no effect on the bandwidth :(>>> FWIW I tried browsing with this setup and downloading aswell and >>> it''s not very nice with such a long queue. An ISP/teleco would never >>> have that long a fifo on a 128kbit line. It would be better to add >>> an sfq to the bulk class or limit the length of the fifo. >> Sorry for such a simplistic question - how might this be done? >> However I''m not overly worried about making the quality of service >> perfect - I just need it to be limited bandwidth. The reason why I >> want such a restricted service is because the shaped NAT clients of >> this box are going to be the "naughty" users on the university >> network who will have their 100Mbit service reduced to 128kbit for a >> week as punishment for rulebreaking etc. So if people complain about >> the occasional dropped packet - they shouldn''t have been downloading >> copyright material in the first place! > > Ahh I see - actually packet loss could make things nicer by getting > the tcp senders congestion control to back off. The command for sfq > would be > > tc qdisc add dev $LAN parent 1:b$total handle b$total: sfq limit 30 > > in the loop under the bulk class.Yep - just tried this too. Again, this appears to have zero effect on the bandwidth. Client 1 still downloads flatly at 166kb/s and Client 2 at 250kb/s. Something pretty weird is going on. Actually a colleague here has offered to set up a Cisco router to do the shaping and let my box just do the NAT. Start of term is imminent and this MUST be live within a week! Thanks for all your help though, you''ve been excellent and I''ve learned loads of stuff. As a relative beginner with Linux, this is yet another experience where something didn''t work as it should, despite the advice of documentation, colleagues, lists, websites, etc... Cheers, Jonathan ------------------------ Jonathan Gazeley ResNet | Wireless & VPN Team Information Systems & Computing University of Bristol ------------------------
Jonathan Gazeley wrote:> Andy Furniss wrote: >> That''s strange - so there are no vlans involved and tcpdump -nnei eth0 >> only sees traffic coming in? > Nope, no vlans. And yes, both tcpdump and tethereal see only on the > outgoing (i.e. clients to internet) ack packets, but not the actual > incoming data packets. Like I said, it can''t be a routing problem > because the only physical route those packets can take between the NAT > clients and the internet is via eth0 on this box. All four interfaces on > the box I''m using are Intel ones, so not just some cheap rubbish! > Probably unlikely to be drivers too?That''s very strange - If the drivers are in the kernel and the kernel is recent then you would hope it''s not them, but it''s something to suspect.> Yep - just tried this too. Again, this appears to have zero effect on > the bandwidth. Client 1 still downloads flatly at 166kb/s and Client 2 > at 250kb/s. Something pretty weird is going on. Actually a colleague > here has offered to set up a Cisco router to do the shaping and let my > box just do the NAT. Start of term is imminent and this MUST be live > within a week! Thanks for all your help though, you''ve been excellent > and I''ve learned loads of stuff. As a relative beginner with Linux, this > is yet another experience where something didn''t work as it should, > despite the advice of documentation, colleagues, lists, websites, etc...It seems you''ve been unlucky, I can''t really explain what you see. Hope the Cisco goes OK for you. Andy.