Hi,
Firstly I appologise for the cross-posting but as I got help from all
over the place I thought I had better feed back the results now I have
this working.
The Problem:
I run a Speedtouch ADSL modem on my router and host a number of services
behind my link (web server mainly) so people can access my files.
Unfortunately some of the files on my web-server are quite big > 100Mb
and while I''m happy for people to download them it kinda kills
interactivity when I am at home surfing.
The Solution:
Line rate control! I was going through various cookbook approaches (e.g.
the wondershapper) but decided to role my own because:
a) I''ve just grokked iptables and I didn''t want to learn yet
another
packet matching syntax
b) cbq solutions looked to hard to understand
c) its the only way to learn
What this script does is mark upstream packets using a bunch of iptables
matches. The marks correspond to the priority I want to assign my
traffic (remember I can only do this for outgoing packets, shaping
incoming data on my LAN wouldn''t achieve much).
The script then create a bunch of htb shapers, one for each traffic type
giving a controlled rate of output. The filters are then setup to direct
packets to each traffic class based on the iptables matches done
earlier.
For more info read the script
Caveats:
This works for me, YMMV. I''ve done limited testing and for me I can
surf at my normal high speeds while large downloads happen from my
server. I expect it can be tuned further with experimentation and
would welcome any feedback on the script. I have a moderate number of
services on my link, I expect most people can simplify the priorities
to traffic originated by me and incomming connections. The script is
part of a larger firewall script that can be found on my websites CVS
pages (under software) but its not fully integrated yet.
Enjoy,
Alex.
function setup_shaping ()
{
# Setup POSTROUTING marking on dsl output
# needed for QoS type hacks
# 1 - outgoing interactive (ssh)
# 2 - outgoing file stuff (www)
# 3 - incomming interactive (ssh)
# 4 - incomming personal use (https, http-tunnel)
# 5 - incomming web
# 6 - incomming mail
# 7 - everything else
# create the to-dsl table (we can only shape outgoing traffic)
/sbin/iptables -t mangle -N to-dsl
# For outgoing packets we need to mark stuff
/sbin/iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK
--set-mark 1
/sbin/iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK
--set-mark 2
/sbin/iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK
--set-mark 3
/sbin/iptables -t mangle -A to-dsl -p tcp --sport 443 -j MARK
--set-mark 4
/sbin/iptables -t mangle -A to-dsl -p tcp --sport 8890 -j MARK
--set-mark 4
/sbin/iptables -t mangle -A to-dsl -p tcp --sport 80 -j MARK
--set-mark 5
/sbin/iptables -t mangle -A to-dsl -p tcp --sport 25 -j MARK
--set-mark 6
# enable the marking on all outgoing packets
/sbin/iptables -t mangle -A POSTROUTING -o $EXTIF -j to-dsl
# and the qdisc''s
# Base htb class
/sbin/tc qdisc add dev ppp0 root handle 1: htb default 60
# add a rate limiting class underneath - this ensure we don''t send
# packets to the dsl modem faster than its going to send them
/sbin/tc class add dev ppp0 parent 1: classid 1:1 htb rate 250kbit
burst 6k
#sub classes for each traffic type
/sbin/tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 250kbit
burst 15k
/sbin/tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 250kbit
burst 15k
/sbin/tc class add dev ppp0 parent 1:1 classid 1:30 htb rate 250kbit
burst 15k
/sbin/tc class add dev ppp0 parent 1:1 classid 1:40 htb rate 250kbit
burst 15k
/sbin/tc class add dev ppp0 parent 1:1 classid 1:50 htb rate 128kbit
burst 50k
/sbin/tc class add dev ppp0 parent 1:1 classid 1:60 htb rate 100kbit
burst 15k
#note to self: to show class stats
#tc -s -d class show dev ppp0 parent 1:
# don''t use prio anymore
#tc qdisc add dev ppp0 parent 1:1 handle 2: prio bands 6 priomap 0 1
2 3 4 5
# create sfq''s under each traffic class to share it all out
/sbin/tc qdisc add dev ppp0 parent 1:10 handle 10: sfq
/sbin/tc qdisc add dev ppp0 parent 1:20 handle 20: sfq
/sbin/tc qdisc add dev ppp0 parent 1:30 handle 30: sfq
/sbin/tc qdisc add dev ppp0 parent 1:40 handle 40: sfq
/sbin/tc qdisc add dev ppp0 parent 1:50 handle 50: sfq
/sbin/tc qdisc add dev ppp0 parent 1:60 handle 60: sfq
# note to self: delete with
# tc qdisc del dev ppp0 parent 1:0 handle 10:
# are flowid and classid interchangable?
# create filters from the root to sort the traffic
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1
fw classid 1:10
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 2 handle 2
fw classid 1:20
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 3 handle 3
fw classid 1:30
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 4 handle 4
fw classid 1:40
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 5 handle 5
fw classid 1:50
/sbin/tc filter add dev ppp0 parent 1: protocol ip prio 6 handle 6
fw classid 1:60
#look at with
#tc filter show dev ppp0 parent 1:
#delete with
#tc filter del dev ppp0 parent 1: prio 1 etc..
}
--
Alex@Bennee.com
http://www.bennee.com/~alex/
Hi, I''m happy that HTB gained so much popularity ;) Only one hint for you - you can completely avoid all these tc filter add .... fw ... You can use only one tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw and set classid directly in iptables like: iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 and so on .. devik On 17 Apr 2002, alex wrote:> /sbin/iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK > --set-mark 1 > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK > --set-mark 2 > > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 > fw classid 1:10 > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 2 handle 2 > fw classid 1:20
Martin Devera said:> Hi, > I''m happy that HTB gained so much popularity ;) Only > one hint for you - you can completely avoid all these > tc filter add .... fw ... > You can use only one > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > and set classid directly in iptables like: > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > and so on .. > devikThanks for that it should make my script a bit less cumbersome. I have also realised that at the moment the bandwidth is being shared out in proportion to allocated bandwidths which is not quite what I was after. Having re-read your manual pages I''ve now added "prio" statements to each htb class so that if I''m downloading from inside I get all the bandwidth I need at the expense of the uploads, rather tha a 2:1 split. I got it the second time, the first time I wasn''t sure if prio 0 was the highest or lowest priority. The other thing that is current sub-optimal is the division of long uploads vs short uploads. I''ve attempted to ensure that normal webpages are downloaded as fast as possible with the burst parameter but if someone is downloading a large file from my website all other web users suffer. I''ve got to do some more reading but my current plan involves the iptable connection tracking. I''m not sure if iptables does this already but if I can match and tag a packet based on the time of the connection I can still allow new connections to get priority of long lived downloads. This may involve writting a new kernel module as a netfilter extension but it would be the iceing on the cake to my setup :-) Alex www.bennee.com/~alex/
> The other thing that is current sub-optimal is the division of long uploads > vs short uploads. I''ve attempted to ensure that normal webpages are > downloaded as fast as possible with the burst parameter but if someone is > downloading a large file from my website all other web users suffer. I''ve > got to do some more reading but my current plan involves the iptable > connection tracking. > > I''m not sure if iptables does this already but if I can match and tag a > packet based on the time of the connection I can still allow new > connections to get priority of long lived downloads. This may involve > writting a new kernel module as a netfilter extension but it would be theHi I was already planing it ;) To add netfilter match to STATUS module which would allow --conn-traffic from:to to select only connection whose transfered bytes value is in given range. Then you can have different classes for long and short downloads .. devik
On Wednesday 17 April 2002 10:34, Martin Devera wrote:> Hi, > I''m happy that HTB gained so much popularity ;) Only > one hint for you - you can completely avoid all these > tc filter add .... fw ... > You can use only one > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > and set classid directly in iptables like: > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > and so on .. > devikThx. Even I ''m learning from this list :) But how do you translate xx:xx to HEX ? Stef> > On 17 Apr 2002, alex wrote: > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK > > --set-mark 1 > > > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK > > --set-mark 2 > > > > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 > > fw classid 1:10 > > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 2 handle 2 > > fw classid 1:20 > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/-- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.openprojects.net
> > You can use only one > > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > > > and set classid directly in iptables like: > > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > > > and so on .. > > devik > Thx. > Even I ''m learning from this list :) > But how do you translate xx:xx to HEX ?handle numbers in tc ARE in hex, so that: ... classid a23f:334d can be written as --set-mark 0xa23f334d devik
> handle numbers in tc ARE in hex, so that: > > ... classid a23f:334d > > can be written as --set-mark 0xa23f334dAnd handle 11:111 ? Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.openprojects.net
Stef Coene said:>> handle numbers in tc ARE in hex, so that: >> >> ... classid a23f:334d >> >> can be written as --set-mark 0xa23f334d > And handle 11:111 ?would become --set-mark 0x00110111...> Stef > > -- > > stef.coene@docum.org > "Using Linux as bandwidth manager" > http://www.docum.org/ > #lartc @ irc.openprojects.net > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/Alex www.bennee.com/~alex/
0x110111 Last 4 digits (16bits) is class, firct 16bit is qdisc. On Wed, 17 Apr 2002, Stef Coene wrote:> > handle numbers in tc ARE in hex, so that: > > > > ... classid a23f:334d > > > > can be written as --set-mark 0xa23f334d > And handle 11:111 ? >
> > And handle 11:111 ? > > would become --set-mark 0x00110111...Or 0x011111? mhh. So the hex number is splitted in 2 equal halves and the ":" is putted in the middle ? Stef
> From: Martin Devera <devik@cdi.cz>> I''m happy that HTB gained so much popularity ;) Only > one hint for you - you can completely avoid all these > tc filter add .... fw ... > You can use only one > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > and set classid directly in iptables like: > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > and so on .. > devik > > > On 17 Apr 2002, alex wrote: > > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK > > --set-mark 1 > > > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK > > --set-mark 2 > > > > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 > > fw classid 1:10 > > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 2 handle 2 > > fw classid 1:20 Is this documented anywhere? How/why does it work? I gather that the mark is interpreted as 16 bits of parent and 16 bits of class. But you couldn''t you also have said something like this? iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 10 fw classid 1:20 ** **** Would that just not work?
exactly On Wed, 17 Apr 2002, Stef Coene wrote:> > > And handle 11:111 ? > > > > would become --set-mark 0x00110111... > Or 0x011111? > > mhh. So the hex number is splitted in 2 equal halves and the ":" is putted > in the middle ?
> Is this documented anywhere? How/why does it work?I don''t think so (docs). But yes it is fw''s feature. I found it in cls_fw.c code - it is commented/described here.> I gather that the mark is interpreted as 16 bits of parent and 16 bits > of class. But you couldn''t you also have said something like this? > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > /sbin/tc filter add dev ppp0 parent 1: protocol ip prio 1 > handle 10 fw classid 1:20 > ** **** > Would that just not work?No. The trick above works ONLY when mark == qdisc:classid AND fw filter has NO children (no classid terms). devik
----- Original Message ----- From: "alex" <alex@bennee.com>> # For outgoing packets we need to mark stuff > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK > --set-mark 1 > > /sbin/iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK > --set-mark 2I''d also do like this: iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK --set-mark 1 iptables -t mangle -A to-dsl -p tcp --dport 22 -j RETURN iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 2 iptables -t mangle -A to-dsl -p tcp --dport 80 -j RETURN etc... Otherwise iptables will do the whole "to-dsl" list for every packet. In your case ot wouldn''t matter except for some extra CPU usage. But if you would like to mark port 80 as bulk-traffic and ACK''s as interactive traffic, then those port 80 ACK''s could be marked as bulk which you wouldn''t want it to. Which brings me to another subject :) If your DSL-connection have different bandwidth like 1mbit/128kbit then your download speed could be destroyed by huge queues in your uplink. I''d guess this would do the trick. # Set ACK as prioritized traffic (ACK''s are less than 100 bytes) $IPTABLES -t mangle -A MANGLE_MARK -p tcp -m length --length :100 -j MARK --set-mark 1 $IPTABLES -t mangle -A MANGLE_MARK -p tcp -m length --length :100 -j RETURN (You could probably mark ACK''s with --tcp-flags SYN,FIN,RST ACK. But I have not tested that yet.) They also mention this here: http://lartc.org/wondershaper/ /Jonas
Hello there!> I''d also do like this: > > iptables -t mangle -A to-dsl -p tcp --dport 22 -j MARK --set-mark 1 > iptables -t mangle -A to-dsl -p tcp --dport 22 -j RETURN > > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 2 > iptables -t mangle -A to-dsl -p tcp --dport 80 -j RETURN > > etc... > > Otherwise iptables will do the whole "to-dsl" list for every packet. In > your case ot wouldn''t matter except for some extra CPU usage. But if you > would like to mark port 80 as bulk-traffic and ACK''s as interactive > traffic, then those port 80 ACK''s could be marked as bulk which you > wouldn''t want it to.Does this behavior also occure when using ipchains? Greetings Nils
> > Otherwise iptables will do the whole "to-dsl" list for every packet. In > > your case ot wouldn''t matter except for some extra CPU usage. But if you > > would like to mark port 80 as bulk-traffic and ACK''s as interactive > > traffic, then those port 80 ACK''s could be marked as bulk which you > > wouldn''t want it to. > > Does this behavior also occure when using ipchains?yes
Hi there!> I''m happy that HTB gained so much popularity ;)Oh yes, its realy popular already, not only among people who now a lot about linux, they just don''t know they are using it. -> www.fli4l.de> Only one hint for you - you can completely avoid all these > tc filter add .... fw ... > You can use only one > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > and set classid directly in iptables like: > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020Is this also possible with the -m option in ipchains? Greetings Nils
Hello again!> Only one hint for you - you can completely avoid all these > tc filter add .... fw ... > You can use only one > tc filter add dev ppp0 parent 1: protocol ip prio 1 handle 1 fw > > and set classid directly in iptables like: > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020Oh an and I forgot to ask: Is there any other improvement exept for having a shorter script? Greetings Nils
> > and set classid directly in iptables like: > > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > Oh an and I forgot to ask: Is there any other improvement exept for having a > shorter script?It is a bit faster and simpler to maintain. devik
Hello Devik!> > > and set classid directly in iptables like: > > > iptables -t mangle -A to-dsl -p tcp --dport 80 -j MARK --set-mark 0x10010 > > > iptables -t mangle -A to-dsl -p tcp --sport 24 -j MARK --set-mark 0x10020 > > > > Oh an and I forgot to ask: Is there any other improvement exept for having a > > shorter script? > > It is a bit faster and simpler to maintain.We are using Ipchains. I guess that makes no difference. At the moment we do the following: For Example (all rules for the same device): 1. filter ACKs by using u32 2. filter a specific IP by using ipchains -m (because we masquarade) with its own tc fw 3. filter ToS by using u32 4. filter by another IP with ipchains -m with its own tc fw With only one tc fw per device (and using 8bit values with ipchains -m to specifie the targetclass) i guess the order like shown above could not be maintained. It would look like 2. 4. 1. 3. or 1. 2. 4. 3. depending on the positon of the tc fw filter. The same order like in the example can not be achieved. Am I right with that assumption? Is it possible to have more than one of these "global" tc fw filters for one device? Greetings Nils
> We are using Ipchains. I guess that makes no difference. At the moment we do the > following: > > For Example (all rules for the same device): > 1. filter ACKs by using u32 > 2. filter a specific IP by using ipchains -m (because we masquarade) with its > own tc fw > 3. filter ToS by using u32 > 4. filter by another IP with ipchains -m with its own tc fw > > With only one tc fw per device (and using 8bit values with ipchains -m to > specifie the targetclass) i guess the order like shown above could not be > maintained. It would look like > > 2. 4. 1. 3. or 1. 2. 4. 3. depending on the positon of the tc fw filter. The > same order like in the example can not be achieved. Am I right with that > assumption?unfortunately, you are right> Is it possible to have more than one of these "global" tc fw filters for one > device?probapbly yes but the first one will match all. But you can filter acks with ipchains too (-y). devik
Hello Devik!> But you can filter acks > with ipchains too (-y).Uh can I? I thought -y is for matching SYN-Packets. Greetings Nils
Errr my mistake ... Yes SYN not ACK, sorry. On Sat, 27 Apr 2002, Nils Lichtenfeld wrote:> Hello Devik! > > > But you can filter acks > > with ipchains too (-y). > > Uh can I? I thought -y is for matching SYN-Packets. > > Greetings Nils > > >