I''m trying to setup traffic shaping on my linux gateway/router. The system has 3 interfaces: eth0 - My LAN - with IP address 192.168.0.254 eth1 - The ethernet connection to which my ADSL modem is connected. This has a 10.25.x.x IP, more on this later. The ADSL link has an upstream of ~1.2mbit. ppp0 - The PPP connection which is my WAN connection, with a real world IP. The system acts a router, performing NAT for my LAN. This works perfectly, as does traffic shaping on ppp0 - I get very good results. The trouble is that my ISP allows me to user another service over my ADSL line, as a bonus. Basically the modem has 2 virtual circuits, one being for my WAN connection, and the other being a private network between other users of the same ISP, on the same telephone exchange - this is where the 10.25.x.x IP on eth1 comes from. To make things clear, low latency on the eth1 interface is not important, this interface is only used for file sharing and such. Latency on ppp0 is obviously important, being my WAN connection. My IPTables rules provide NAT for both connections, the only thing I cannot get working correctly is traffic shaping. So far, I have experimented with wondershaper, shaping on the ppp0 interface. This works well to keep latency down when traffic is on the ppp0 interface. If there is traffic on eth1 (the ''private'' network of 10.25.x.x), with no traffic on the ppp0 interface, latency on ppp0 remains low, regardless of whether traffic shaping is active. I believe this has something to do with the way my ISP has configured priorities at the telephone exchange. I begin to run in to trouble when I am uploading heavily on eth1 & ppp0 simultaneously. Once this happens, ping times over ppp0 rise dramatically, to well over 1200ms (normal is around 7ms). I have tried shaping on eth1 instead of ppp0 (as eth1 should contain all the packets for ppp0, I believe), but this does not yield lower latency, though I did note that it did limit the speed of the connection if I set the upstream and downstream values absurdly low. I think what I need to do is somehow setup a script where traffic directed to 10.25.0.0 on eth1 is somehow counted against the bandwidth specified for ppp0, but I''m really not sure. Could someone offer some advice? Thanks. I believe this is due to the fact that as outbound traffic is occurring on both interfaces, tc does not worry about queues etc, as it does not look as though ppp0 is being maxed out (ppp0 would only be using roughly half of 1.2mbit), when the ADSL link itself is being maxed out.
Terry Baume wrote:> I''m trying to setup traffic shaping on my linux gateway/router. > > The system has 3 interfaces: > eth0 - My LAN - with IP address 192.168.0.254 > eth1 - The ethernet connection to which my ADSL modem is connected. > This has a 10.25.x.x IP, more on this later. The ADSL link has an > upstream of ~1.2mbit. > ppp0 - The PPP connection which is my WAN connection, with a real > world IP. > > The system acts a router, performing NAT for my LAN. This works > perfectly, as does traffic shaping on ppp0 - I get very good results. > > The trouble is that my ISP allows me to user another service over my > ADSL line, as a bonus. Basically the modem has 2 virtual circuits, one > being for my WAN connection, and the other being a private network > between other users of the same ISP, on the same telephone exchange - > this is where the 10.25.x.x IP on eth1 comes from. To make things > clear, low latency on the eth1 interface is not important, this > interface is only used for file sharing and such. Latency on ppp0 is > obviously important, being my WAN connection. > > My IPTables rules provide NAT for both connections, the only thing I > cannot get working correctly is traffic shaping. > > So far, I have experimented with wondershaper, shaping on the ppp0 > interface. This works well to keep latency down when traffic is on the > ppp0 interface. If there is traffic on eth1 (the ''private'' network of > 10.25.x.x), with no traffic on the ppp0 interface, latency on ppp0 > remains low, regardless of whether traffic shaping is active. I > believe this has something to do with the way my ISP has configured > priorities at the telephone exchange. I begin to run in to trouble > when I am uploading heavily on eth1 & ppp0 simultaneously. Once this > happens, ping times over ppp0 rise dramatically, to well over 1200ms > (normal is around 7ms). I have tried shaping on eth1 instead of ppp0 > (as eth1 should contain all the packets for ppp0, I believe), but this > does not yield lower latency, though I did note that it did limit the > speed of the connection if I set the upstream and downstream values > absurdly low. > > I think what I need to do is somehow setup a script where traffic > directed to 10.25.0.0 on eth1 is somehow counted against the bandwidth > specified for ppp0, but I''m really not sure. Could someone offer some > advice? > > Thanks. > > I believe this is due to the fact that as outbound traffic is > occurring on both interfaces, tc does not worry about queues etc, as > it does not look as though ppp0 is being maxed out (ppp0 would only be > using roughly half of 1.2mbit), when the ADSL link itself is being > maxed out. > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartcTry using IMQ, from www.linuximq.net
Terry Baume wrote:> I''m trying to setup traffic shaping on my linux gateway/router. > > The system has 3 interfaces: > eth0 - My LAN - with IP address 192.168.0.254 > eth1 - The ethernet connection to which my ADSL modem is connected. This > has a 10.25.x.x IP, more on this later. The ADSL link has an upstream of > ~1.2mbit. > ppp0 - The PPP connection which is my WAN connection, with a real world IP. > > The system acts a router, performing NAT for my LAN. This works > perfectly, as does traffic shaping on ppp0 - I get very good results. > > The trouble is that my ISP allows me to user another service over my > ADSL line, as a bonus. Basically the modem has 2 virtual circuits, one > being for my WAN connection, and the other being a private network > between other users of the same ISP, on the same telephone exchange - > this is where the 10.25.x.x IP on eth1 comes from. To make things clear, > low latency on the eth1 interface is not important, this interface is > only used for file sharing and such. Latency on ppp0 is obviously > important, being my WAN connection. > > My IPTables rules provide NAT for both connections, the only thing I > cannot get working correctly is traffic shaping. > > So far, I have experimented with wondershaper, shaping on the ppp0 > interface. This works well to keep latency down when traffic is on the > ppp0 interface.Wondershaper is slightly flawed, depending on how it''s setup. You need to make sure the rateds of children don''t add up to more that the parent class. Unless you patch for atm overheads (which is going to be tricky for your case) make sure you back off say 20% from the line rate - but then this bit works for you anyway - I just say because it can appear to be OK testing with bulk traffic, but then fail when you have a lot of small packets going out. If there is traffic on eth1 (the ''private'' network of> 10.25.x.x), with no traffic on the ppp0 interface, latency on ppp0 > remains low, regardless of whether traffic shaping is active. I believe > this has something to do with the way my ISP has configured priorities > at the telephone exchange. I begin to run in to trouble when I am > uploading heavily on eth1 & ppp0 simultaneously. Once this happens, ping > times over ppp0 rise dramatically, to well over 1200ms (normal is around > 7ms). I have tried shaping on eth1 instead of ppp0 (as eth1 should > contain all the packets for ppp0, I believe), but this does not yield > lower latency, though I did note that it did limit the speed of the > connection if I set the upstream and downstream values absurdly low.You could in theory do it all on eth - but you would have to use the right tc filter ethertypes to get the pppoe and ip.> I think what I need to do is somehow setup a script where traffic > directed to 10.25.0.0 on eth1 is somehow counted against the bandwidth > specified for ppp0, but I''m really not sure. Could someone offer some > advice?I would use ifb it''s been in kernel for a while so you don''t need to patch as you would with imq. You can redirect all ip traffic going out on ppp0/eth1 to ifb0 and add your htb rules to that. Something like - tc qdisc add dev eth1 handle 1:0 root prio tc qdisc add dev ppp0 handle 1:0 root prio modprobe ifb ip link set up dev ifb0 tc filter add dev ppp0 parent 1:0 protocol ip prio 1 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0 tc filter add dev eth1 parent 1:0 protocol ip prio 1 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0 then add your htb rules on dev ifb0 Andy.
Terry Baume wrote:> Hi Andy, > I had a chance to play around with ifb and the wondershaper script so > far, I''ve come to realise a few things, one being related to what you > previously mentioned about wondershaper being somewhat flawed in > particular setups. I''ve included my entire modified wondershaper script > at the end of this mail so that you can see my modifications. > > Your suggestion of redirecting traffic to ifb0 works wonderfully, keeps > latency down very low even with 2 concurrent FTP transfers - one over > ppp0 and one over eth1. > > I start to notice problems however when I have a large amount of traffic > in the lowest priority group (1:30) as well as traffic in (1:20) > competing for bandwidth - Is this what you meant by wondershaper being > slightly flawed? If I classify all traffic directed to 10.25.0.0/25 as > the lowest priority, by adding it to the ''NOPRIOHOSTDST'' option, I > notice that if there is an FTP connection from 10.25.0.0/25, as well as > one coming over ppp0, latency rises very high again. I presume this is > related to the following few lines in the script: > > # bulk & default class 1:20 - gets slightly less traffic, > # and a lower priority: > > tc class add dev ifb0 parent 1:1 classid 1:20 htb rate > $[9*$UPLINK/10]kbit \ > burst 6k prio 2 > > tc class add dev ifb0 parent 1:1 classid 1:30 htb rate > $[8*$UPLINK/10]kbit \ > burst 6k prio 2 > > I''m not sure if I am reading into this correctly, but it seems to > suggest that combined, these 2 classes have more bandwidth than the link > itself?Yes you are right htb rates on leafs are not restricted by the parent rate. I guess when wondershaper was written htb was new and not in kernel. It may work with the cbq version - not that I have ever tested cbq.> > Do you have any suggestions as to how I can modify the script to work > around these problems - ie so that a stream of bulk (1:30) and a stream > of regular (1:20) traffic will not cause high latency?You need to make the rates of the child classes add up to the parents rate. To let them borrow spare bandwidth you can use the ceil parameter. If you still get high latency reduce UPLINK a bit more - dsl has quite high overheads.> tc class add dev ifb0 parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \ > burst 6k prio 1... rate $[6*$UPLINK/10]kbit ceil ${UPLINK}kbit ...> > # bulk & default class 1:20 - gets slightly less traffic, > # and a lower priority: > > tc class add dev ifb0 parent 1:1 classid 1:20 htb rate > $[9*$UPLINK/10]kbit \ > burst 6k prio 2... rate $[2*$UPLINK/10]kbit ceil ${UPLINK}kbit ...> > tc class add dev ifb0 parent 1:1 classid 1:30 htb rate > $[8*$UPLINK/10]kbit \ > burst 6k prio 2... rate $[2*$UPLINK/10]kbit ceil ${UPLINK}kbit ...> ########## downlink ############# > # slow downloads down to somewhat less than the real speed to prevent > # queuing at our ISP. Tune to see how high you can set it. > # ISPs tend to have *huge* queues to make sure big downloads are fast > # > # attach ingress policer: > > tc qdisc add dev ifb0 handle ffff: ingress > > # filter *everything* to it (0.0.0.0/0), drop everything that''s > # coming in too fast: > > tc filter add dev ifb0 parent ffff: protocol ip prio 50 u32 match ip src \ > 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1This won''t work on ifb0. You could put it on eth1 and change the "protocol ip" to "protocol all" and change the match to "match u32 0 0" . In theory that may catch a few arp so I suppose you could add another rule to exempt them. tc filter add dev eth1 parent ffff: protocol arp prio 1 u32 match u32 0 0 flowid :2 I would also make the burst bigger since you have quite high ingress bandwidth. Andy.
Andy Furniss wrote:> This won''t work on ifb0. You could put it on eth1 and change the > "protocol ip" to "protocol all" and change the match to "match u32 0 > 0" . In theory that may catch a few arp so I suppose you could add > another rule to exempt them. > > tc filter add dev eth1 parent ffff: protocol arp prio 1 u32 match u32 > 0 0 flowid :2 > > I would also make the burst bigger since you have quite high ingress > bandwidth. > > Andy.Thanks for the suggestions Andy, I''ve put them all into place, and everything seems to be working nicely. I just had a question regarding the rule to catch the ARP''s - I get this message when I add the rule: RTNETLINK answers: File exists I guess this because the parent qdisc for that interface has already been defined - does this mean that the rule won''t work? Or is it just a debugging notice that can be safely ignored? I''ve also raised my burst to 50k. Implementing the downstream shaping techniques you suggested, I am seeing good results. I had a question regarding the ifb device - does it behave as a normal interface when doing masquerading etc? It seems that if I place destination IP''s in the NOPRIOHOSTDST field, these get marked as low priority, and the same when I put source ports in NOPRIOPORTSRC. When I put source IP''s in the NOPRIOHOSTSRC field, these do not seem to get marked as low priority - I tried with 192.168.0.1 as an example (an FTP server on the network). Could this be related to the fact that I''m using the IFB device? Thanks, Terry.
Terry Baume wrote:> Andy Furniss wrote: >> This won''t work on ifb0. You could put it on eth1 and change the >> "protocol ip" to "protocol all" and change the match to "match u32 0 >> 0" . In theory that may catch a few arp so I suppose you could add >> another rule to exempt them. >> >> tc filter add dev eth1 parent ffff: protocol arp prio 1 u32 match u32 >> 0 0 flowid :2 >> >> I would also make the burst bigger since you have quite high ingress >> bandwidth. >> >> Andy. > Thanks for the suggestions Andy, I''ve put them all into place, and > everything seems to be working nicely. I just had a question regarding > the rule to catch the ARP''s - I get this message when I add the rule: > > RTNETLINK answers: File exists > > I guess this because the parent qdisc for that interface has already > been defined - does this mean that the rule won''t work? Or is it just a > debugging notice that can be safely ignored?You shouldn''t get an error for that - as long as you change the download part of the script to something like - tc qdisc del dev eth1 ingress &>/dev/null tc qdisc add dev eth1 handle ffff: ingress tc filter add dev eth1 parent ffff: protocol arp prio 1 u32 match u32 0 0 flowid :2 tc filter add dev eth1 parent ffff: protocol all prio 50 u32 match u32 0 0 police rate ${DOWNLINK}kbit burst 50k drop flowid :1> > I''ve also raised my burst to 50k. Implementing the downstream shaping > techniques you suggested, I am seeing good results.Good - I guess 50k is OK as it''s a wan with wan latency and you are shaping ingress from a bitrate limited line, so the virtual buffer only fills slowly. If you really wanted to have more control than just policing the whole link, you could use another ifb and shape the traffic as you do on egress - it won''t be quite the same, though, because you will be shaping from the wrong end of the bottleneck. FWIW if it were a lan then 50k @ 17mbit totally borks a single tcp connection so you''ll get nowhere near the set rate. It''s because the latencies are so low I guess - netem 10ms delay on and all is well again - or make the burst bigger.> > I had a question regarding the ifb device - does it behave as a normal > interface when doing masquerading etc?I don''t think iptables will see it as a real device.> It seems that if I place > destination IP''s in the NOPRIOHOSTDST field, these get marked as low > priority, and the same when I put source ports in NOPRIOPORTSRC. When I > put source IP''s in the NOPRIOHOSTSRC field, these do not seem to get > marked as low priority - I tried with 192.168.0.1 as an example (an FTP > server on the network). Could this be related to the fact that I''m using > the IFB device?No it''s because if you do SNAT/MASQUERADE the the addresses have already been changed - it would still happen if you shaped directly on ppp0. To workaround you need to use iptables rules to mark the traffic and then make tc filter rules to match the marks eg. iptables -t mangle -A POSTROUTING --src 192.168.0.1 -j MARK --set-mark 35 tc filter add dev ifb0 parent 1:0 prio 27 protocol ip handle 35 fw flowid 1:X depending on what other rules are used you''ll need to change the prio to something unused. Andy.