LARTC - Apr 2005 - Qos with 2 internet connections problems

Hi all, ive got 2 internet connections set up via the nano howto (which
are working great)  and we are running NAT.
Was looking in to qos mainly to stop large http downloads/ftp downloads
from hogging the line so that browsing for other users doesnt slow to a
crawl, but if the line is free and no one is doing anything then for it
to use the available bandwith. The wondershaper sounded exactly what i
wanted, i obviously had to customize this slightly to get my other
connection to be shaped as well.
my connections are 2 adsl connections running at 512kbit:256kbit, i
compiled the kernel with all the tos + iptables stuff (eventually) so
that the script ran without any errors, once it had i did a download on
each pipe and then tried to ping jolt.co.uk and google.co.uk, jolt.co.uk
normally is about 15ms, so i was hoping that it would maybe be 200ms
with my new shaping, but it was coming at about 900ms+ google was the
same. Browsing other website was also to a crawl (what i was trying to
avoid) but i did notice that when traffic came from our mail server
through my gateway it was only receiving at about 25k/sec instead of
2mb/sec etc as its on a 100mbit switch. So something was obviously
shaping, so i suppose i will now need to look in to specifying some of
the hosts which are local to me but are on the outside interfaces and
for them not to be shaped.. But still no better off :(

Ive been reading various howtos like the adsl bandwith management howto,
but these havnt been updated in about 3 years and they mention various
techniques which were being tried back then to help with these kind of
situations, so what i really want is a howto which shows all these new
features? or is this adsl bandwith management still current ? :)


my network diagram

Lan machines -> Linux Router -> Alcatel Router -> ADSL
Lan Machine -> linux router -> alcatel router2 -> ADSL

Alcatel router -> Linux router
Alcatel router -> mail server etc




my modified wondershaper script is below: if anyone could point out some
errors and or maybe point me to somewhere where i could learn how to do
what i want better please let me know.



#!/bin/bash -x
# Wonder Shaper
# please read the README before filling out these values
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits. Also set the device that is to be shaped.

DOWNLINK=512
UPLINK=256
DEV=eth1
DEV2=eth2

DOWNLINKlan=512
UPLINKlan=256
DEVlan=eth1




# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC
# low priority destination netmasks
NOPRIOHOSTDST
# low priority source ports
NOPRIOPORTSRC
# low priority destination ports
NOPRIOPORTDST

# Now remove the following two lines :-)

#echo Please read the documentation in ''README'' first
#exit


#pipe1

if [ "$1" = "status" ]
then
      tc -s qdisc ls dev $DEV
      tc -s class ls dev $DEV
      exit
fi

# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null


###### uplink

# install root HTB, point default traffic to 1:20:

tc qdisc add dev $DEV root handle 1: htb default 20

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k

# high prio class 1:10:

tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
   burst 6k prio 1

# bulk & default class 1:20 - gets slightly less traffic,
# and a lower priority:

tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
   burst 6k prio 2

tc class add dev $DEV parent 1:1 classid 1:30 htb rate $[8*$UPLINK/10]kbit \
   burst 6k prio 2

# all get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10

# TOS Minimum Delay (ssh, NOT scp) in 1:10:

tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
  match ip tos 0x10 0xff  flowid 1:10

# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
        match ip protocol 1 0xff flowid 1:10

# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:

tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8 0x10 0xff at 33 \
   flowid 1:10

# rest is ''non-interactive'' ie ''bulk'' and
ends up in 1:20

# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
        tc filter add dev $DEV parent 1: protocol ip prio 14 u32 \
           match ip dport $a 0xffff flowid 1:30
done

for a in $NOPRIOPORTSRC
do
        tc filter add dev $DEV parent 1: protocol ip prio 15 u32 \
           match ip sport $a 0xffff flowid 1:30
done

for a in $NOPRIOHOSTSRC
do
        tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
           match ip src $a flowid 1:30
done

for a in $NOPRIOHOSTDST
do
        tc filter add dev $DEV parent 1: protocol ip prio 17 u32 \
           match ip dst $a flowid 1:30
done

# rest is ''non-interactive'' ie ''bulk'' and
ends up in 1:20

tc filter add dev $DEV parent 1: protocol ip prio 18 u32 \
   match ip dst 0.0.0.0/0 flowid 1:20


########## downlink #############
# slow downloads down to somewhat less than the real speed  to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:

tc qdisc add dev $DEV handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that''s
# coming in too fast:

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1


###########################################################################
#pipe2
#########################################################################
if [ "$1" = "status" ]
then
        tc -s qdisc ls dev $DEV2
        tc -s class ls dev $DEV2
        exit
fi





# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV2 root    2> /dev/null > /dev/null
tc qdisc del dev $DEV2 ingress 2> /dev/null > /dev/null

if [ "$1" = "stop" ]
then
      exit
fi


###### uplink

# install root HTB, point default traffic to 1:20:

tc qdisc add dev $DEV2 root handle 2: htb default 20

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:

tc class add dev $DEV2 parent 2: classid 2:1 htb rate ${UPLINK}kbit burst 6k

# high prio class 1:10:

tc class add dev $DEV2 parent 2:1 classid 2:10 htb rate ${UPLINK}kbit \
   burst 6k prio 1

# bulk & default class 1:20 - gets slightly less traffic,
# and a lower priority:

tc class add dev $DEV2 parent 2:1 classid 2:20 htb rate
$[9*$UPLINK/10]kbit \
   burst 6k prio 2

tc class add dev $DEV2 parent 2:1 classid 2:30 htb rate
$[8*$UPLINK/10]kbit \
   burst 6k prio 2

# all get Stochastic Fairness:
tc qdisc add dev $DEV2 parent 2:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV2 parent 2:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV2 parent 2:30 handle 30: sfq perturb 10

# TOS Minimum Delay (ssh, NOT scp) in 1:10:

tc filter add dev $DEV2 parent 2:0 protocol ip prio 10 u32 \
      match ip tos 0x10 0xff  flowid 2:10

# ICMP (ip protocol 1) in the interactive class 2:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV2 parent 2:0 protocol ip prio 10 u32 \
        match ip protocol 1 0xff flowid 2:10

# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:

tc filter add dev $DEV2 parent 2: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8 0x10 0xff at 33 \
   flowid 2:10

# rest is ''non-interactive'' ie ''bulk'' and
ends up in 1:20

# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
      tc filter add dev $DEV2 parent 2: protocol ip prio 14 u32 \
         match ip dport $a 0xffff flowid 2:30
done

for a in $NOPRIOPORTSRC
do
      tc filter add dev $DEV2 parent 2: protocol ip prio 15 u32 \
         match ip sport $a 0xffff flowid 2:30
done

for a in $NOPRIOHOSTSRC
do
      tc filter add dev $DEV2 parent 2: protocol ip prio 16 u32 \
         match ip src $a flowid 2:30
done

for a in $NOPRIOHOSTDST
do
      tc filter add dev $DEV2 parent 2: protocol ip prio 17 u32 \
         match ip dst $a flowid 2:30
done

# rest is ''non-interactive'' ie ''bulk'' and
ends up in 2:20

tc filter add dev $DEV2 parent 2: protocol ip prio 18 u32 \
   match ip dst 0.0.0.0/0 flowid 2:20


########## downlink #############
# slow downloads down to somewhat less than the real speed  to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:

tc qdisc add dev $DEV2 handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that''s
# coming in too fast:

tc filter add dev $DEV2 parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1



cheers

anthony

Anthony Letchet wrote:
> Hi all, ive got 2 internet connections set up via the nano howto (which
> are working great)  and we are running NAT.
> Was looking in to qos mainly to stop large http downloads/ftp downloads
> from hogging the line so that browsing for other users doesnt slow to a
> crawl, but if the line is free and no one is doing anything then for it
> to use the available bandwith. The wondershaper sounded exactly what i
> wanted, i obviously had to customize this slightly to get my other
> connection to be shaped as well.
> my connections are 2 adsl connections running at 512kbit:256kbit, i
> compiled the kernel with all the tos + iptables stuff (eventually) so
> that the script ran without any errors, once it had i did a download on
> each pipe and then tried to ping jolt.co.uk and google.co.uk, jolt.co.uk
> normally is about 15ms, so i was hoping that it would maybe be 200ms
> with my new shaping, but it was coming at about 900ms+ google was the
> same. Browsing other website was also to a crawl (what i was trying to
> avoid) but i did notice that when traffic came from our mail server
> through my gateway it was only receiving at about 25k/sec instead of
> 2mb/sec etc as its on a 100mbit switch. So something was obviously
> shaping, so i suppose i will now need to look in to specifying some of
> the hosts which are local to me but are on the outside interfaces and
> for them not to be shaped.. But still no better off :(
> 
> Ive been reading various howtos like the adsl bandwith management howto,
> but these havnt been updated in about 3 years and they mention various
> techniques which were being tried back then to help with these kind of
> situations, so what i really want is a howto which shows all these new
> features? or is this adsl bandwith management still current ? :)
> 
I would recommend reading the masters thesis of Jesper Dangaard Brouer 
at http://www.adsl-optimizer.dk/thesis/
Altough he didnt release software yet (there is some code & patches in 
the pdf file) which could lead to the best solution, you could estimate 
the adsl overhead with real knowledge, and not like it says in some 
scripts (put x kbit less than the link bandwith, or 5% less,...)
> my network diagram
> 
> Lan machines -> Linux Router -> Alcatel Router -> ADSL
> Lan Machine -> linux router -> alcatel router2 -> ADSL
> 
> Alcatel router -> Linux router
> Alcatel router -> mail server etc
> 
> 
> 
> 
> my modified wondershaper script is below: if anyone could point out some
> errors and or maybe point me to somewhere where i could learn how to do
> what i want better please let me know.
> 
> 
> 
> #!/bin/bash -x
> # Wonder Shaper
> # please read the README before filling out these values
> #
> # Set the following values to somewhat less than your actual download
> # and uplink speed. In kilobits. Also set the device that is to be shaped.
> 
> DOWNLINK=512
> UPLINK=256
Too much. Because ATM overhead (5/53), 463/231 is the actual maximum 
bandwith.
And you have yet to consider the per packet overhead. Read the thesis I 
mentioned.
> DEV=eth1
> DEV2=eth2
> 
> DOWNLINKlan=512
> UPLINKlan=256
> DEVlan=eth1
> 
Did you use the DEVlan variable?
eth1 is LAN and external at the same time?

Have you looked at http://routeskeeper.sourceforge.net/
or http://selab.edu.ms/twiki/bin/view/Networking/RoutesKeeperProject

Regards,
Francisco.

On Tuesday 05 April 2005 18:36, Francisco Pereira wrote:
<snip>
> I would recommend reading the masters thesis of Jesper Dangaard Brouer
> at http://www.adsl-optimizer.dk/thesis/
> Altough he didnt release software yet (there is some code & patches in
> the pdf file) which could lead to the best solution, you could estimate
> the adsl overhead with real knowledge, and not like it says in some
> scripts (put x kbit less than the link bandwith, or 5% less,...)
I have been eagerly awaiting the release of that software myself.

-- 

Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

Jason Boxman wrote:
> On Tuesday 05 April 2005 18:36, Francisco Pereira wrote:
> <snip>
> 
>>I would recommend reading the masters thesis of Jesper Dangaard Brouer
>>at http://www.adsl-optimizer.dk/thesis/
>>Altough he didnt release software yet (there is some code & patches
in
>>the pdf file) which could lead to the best solution, you could estimate
>>the adsl overhead with real knowledge, and not like it says in some
>>scripts (put x kbit less than the link bandwith, or 5% less,...)
> 
> 
> I have been eagerly awaiting the release of that software myself.
> 
I am testing a version at the moment based on Ed Wildgoose''s - if that 
didn''t work for you though, I can''t see why this one should.

Mine is for UK pppoatm but if you know your overheads it''s easy to 
modify - Do you? there are tables in the doc linked to.

Andy.

Anthony Letchet wrote:
> 
> Lan machines -> Linux Router -> Alcatel Router -> ADSL
> Lan Machine -> linux router -> alcatel router2 -> ADSL
> 
> Alcatel router -> Linux router
> Alcatel router -> mail server etc
So not all traffic goes through Linux router?

As Francisco says you need to back off from link rates - even if you do 
perfect calculation you can only max egress - you won''t build up a
queue
if you are upto the limit for ingress.


Andy.

On Tue, 5 Apr 2005, Jason Boxman wrote:
> On Tuesday 05 April 2005 18:36, Francisco Pereira wrote:
> <snip>
>> I would recommend reading the masters thesis of Jesper Dangaard Brouer
>> at http://www.adsl-optimizer.dk/thesis/
>> Altough he didnt release software yet (there is some code & patches
in
>> the pdf file) which could lead to the best solution, you could estimate
>> the adsl overhead with real knowledge, and not like it says in some
>> scripts (put x kbit less than the link bandwith, or 5% less,...)
>
> I have been eagerly awaiting the release of that software myself.
Sorry, I have not released the software yet... I have been delayed by the 
birth of my little new daughter... have not gotten much sleep lately.
(http://www.trykdenaf.dk/gallery/silke_fodsel)

I can release the patches and a "beta" version of the scripts, if
people
will give me some feedback on the tar.gz distribution file and can live 
with too much debug information/output from the graph-module.

One of my friends are trying out the tar.gz distribution file today. He 
will hopefully give me some positive feedback tomorrow, wether he 
succesfully can follow the install instruction and have a functional 
system.

Hilsen
   Jesper Brouer

--
-------------------------------------------------------------------
Research Assistant and Network Administrator
Dept. of Computer Science, University of Copenhagen
E-mail: hawk@diku.dk, Direct Tel.: 353 21438
-------------------------------------------------------------------

On Wed, 6 Apr 2005, Andy Furniss wrote:
> Jason Boxman wrote:
>> On Tuesday 05 April 2005 18:36, Francisco Pereira wrote:
>> <snip>
>> 
>>> I would recommend reading the masters thesis of Jesper Dangaard
Brouer
>>> at http://www.adsl-optimizer.dk/thesis/
>>> Altough he didnt release software yet (there is some code &
patches in
>>> the pdf file) which could lead to the best solution, you could
estimate
>>> the adsl overhead with real knowledge, and not like it says in some
>>> scripts (put x kbit less than the link bandwith, or 5% less,...)
>> 
>> 
>> I have been eagerly awaiting the release of that software myself.
(see my other excuse mail... ;-)

> I am testing a version at the moment based on Ed Wildgoose''s - if
that didn''t
> work for you though, I can''t see why this one should.
Hmm, I just googled/looked at Ed Wildgoose''s patch, and I is not 
correct/precise.  You can read why my patch is correct/precise in Section 
6.1.2 of the Thesis (which gives a description of the patch).

> Mine is for UK pppoatm but if you know your overheads it''s easy to
modify -
> Do you? there are tables in the doc linked to.
In Chapter 5, I have tried to summarize the different types of 
encapsulation methods and their according overheads, for easy reference.

Is is still a problem figuring out, which type of encapsulation your 
specific ADSL connection is using...


Hilsen
   Jesper Brouer

--
-------------------------------------------------------------------
Research Assistant
Dept. of Computer Science, University of Copenhagen
E-mail: hawk@diku.dk, Direct Tel.: 353 21438
-------------------------------------------------------------------

Jesper Dangaard Brouer wrote:
> Hmm, I just googled/looked at Ed Wildgoose''s patch, and I is not 
> correct/precise.  You can read why my patch is correct/precise in 
> Section 6.1.2 of the Thesis (which gives a description of the patch).
Yes it was always not quite perfect - but it was safe - a cell too safe 
Ed knew this.

I just made one for UK like you do it - but nowhere near as 
comprehensivly - It doesnt do mpu/overhead as just assumes you already 
hacked whatever calls rtab to be aal5_len - 1, which as you say is just 
one line in htb.
> 
> 
>> Mine is for UK pppoatm but if you know your overheads it''s
easy to
>> modify - Do you? there are tables in the doc linked to.
> 
> 
> In Chapter 5, I have tried to summarize the different types of 
> encapsulation methods and their according overheads, for easy reference.
> 
> Is is still a problem figuring out, which type of encapsulation your 
> specific ADSL connection is using...
Yes I agree it can look confusing, life is easier if like me you have a 
modem that gives a cell count and a quiet link. It''s just a case of 
sending various sized pings then.


Andy.

At Wed, 6 Apr 2005 17:56:13 +0200 (CEST),
hawk wrote:

Hey
> >> I would recommend reading the masters thesis of Jesper Dangaard
Brouer
> >> at http://www.adsl-optimizer.dk/thesis/
> > I have been eagerly awaiting the release of that software myself.
> Sorry, I have not released the software yet... I have been delayed by the 
> birth of my little new daughter... have not gotten much sleep lately.
> (http://www.trykdenaf.dk/gallery/silke_fodsel)
Jesper have now released a version of his software. 

http://www.adsl-optimizer.dk/ADSL-optimizer/

http://www.adsl-optimizer.dk/ADSL-optimizer/download/ADSL-optimizer-0.03.tar.gz


Per Marker Mortensen <permm@diku.dk>

research assistant - distlab.dk
office N220 -- dept. of Computer Science, University of Copenhagen
direct +45 35321438 -- mobile +45 20413070 -- home +45 32592041