LARTC - Apr 2005 - HTB Weird Shaping Question(Bug?). Please Help!

Hi all,

I have a script that allocates an ADSL(1500K/256K) bandwidth to three
users.

My idea is to allow each user having a guarentee rate, while each one is
allowed to burst to the max rate while no one is using the bandwidth.

I use imq0 for uplink (for some mobility reason) and imq1 for downlink.
Everything works smoothly except for VoIP traffic.

There are three VoIP phones attached to a computer. They are set up for
different providers. It is possible that all of them are engaged at the
same time. And it does happen that all of them are using G711u codec,
which consuming around 110kbit each for each direction (i.e. the total
bandwidth consumed is larger than the physical link rate 256K). In this
situation, it slowed down the whole network affecting all classes, which
should not be happening. I have done a bit of testing, and isolated the
problem to the uplink congestion.

Please refer to the script I attached blow. I classify all the udp
traffic(generated by the VoIP applications) to 1:110. I continously
generate ping traffic on the computer and the router to a remote
computer in the ISP network. The pings from the computer are classified
as 1:105, while the pings from the router are classified as 1:800. The
max uplink speed is 220Kbit determined by observation.

In the first test, I limit the SUBCLASS_OUTRATE to 200Kbit. Both pings
are around 20ms before I start the VoIP services. However, once I start
the services, the pings jump up to 1800ms.

In the second test, I limit the SUBCLASS_OUTRATE to 180Kbit. The pings
jump up to 80ms, which is perfectly acceptable.

After a few tests, I noticed that 180Kbit is a magic number, anything
exceed that will generate 1800ms pings, and below it is 80ms.

In my senario, the weird point is that the determining factor is the
ceiling, but not the rate. That''s the "rate" for other class
doesn''t
seem to give bandwidth to packets in the corresponding class unless the
ceil for the 1:110 is low enough!

I attached my script and "tc -s class show" below. I truncated part of
the script and the results to make it short.

Please shine me a light!

Thanks heaps,
Leo

P.S. My router is Linksys WRT54G. I''ve tried top, it doesn''t
LOOK LIKE a
CPU load problem.


Script:
####################################
#!/bin/sh

#ADDRs
ROUTER=192.168.1.1
LEO_LAP_WL=192.168.1.31
LEO_LAP=192.168.1.32
LEO_DES=192.168.1.33
PONY_DES=192.168.1.34
ERIC_DES=192.168.1.35
ERIC_LAP=192.168.1.36

# Interfaces
OUTQ=imq0
INQ=imq1

# Speed Rates
ROOT_OUTRATE=200kbit
ROOT_INRATE=1310kbit

CLASS_OUTRATE=200kbit
CLASS_INRATE=1310kbit

SUBCLASS_OUTRATE=180kbit
SUBCLASS_INRATE=1310kbit

# Init Interfaces
ip link set ${INQ} up
ip link set ${OUTQ} up
iptables -t mangle -F
iptables -t mangle -A POSTROUTING -o ppp0 -j IMQ --todev 0
iptables -t mangle -A PREROUTING -i ppp0 -j IMQ --todev 1

#Outbound Traffic
tc qdisc del dev ${OUTQ} root

tc qdisc add dev ${OUTQ} root handle 1: htb default 800
tc class add dev ${OUTQ} parent 1: classid 1:1 htb rate ${ROOT_OUTRATE}

#Leo
tc class add dev ${OUTQ} parent 1:1 classid 1:100 htb rate 17kbps ceil
${CLASS_OUTRATE} prio 0
tc class add dev ${OUTQ} parent 1:100 classid 1:105 htb rate 5kbps ceil
${SUBCLASS_OUTRATE} prio 3 # Other
tc class add dev ${OUTQ} parent 1:100 classid 1:110 htb rate 4kbps ceil
${SUBCLASS_OUTRATE} prio 0 # VoIP
tc class add dev ${OUTQ} parent 1:100 classid 1:120 htb rate 1kbps ceil
${SUBCLASS_OUTRATE} prio 0 # SSH
tc class add dev ${OUTQ} parent 1:100 classid 1:130 htb rate 1kbps ceil
${SUBCLASS_OUTRATE} prio 1 # Telnet
tc class add dev ${OUTQ} parent 1:100 classid 1:140 htb rate 2kbps ceil
${SUBCLASS_OUTRATE} prio 1 # TermSrv
tc class add dev ${OUTQ} parent 1:100 classid 1:150 htb rate 4kbps ceil
${SUBCLASS_OUTRATE} prio 2 # Mail

#Pony
tc class add dev ${OUTQ} parent 1:1 classid 1:200 htb rate 5kbps ceil
${CLASS_OUTRATE} prio 4

#Eric
tc class add dev ${OUTQ} parent 1:1 classid 1:300 htb rate 5kbps ceil
${CLASS_OUTRATE} prio 4

#Other
tc class add dev ${OUTQ} parent 1:1 classid 1:800 htb rate 0.5kbps ceil
${CLASS_OUTRATE} prio 5

tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 1 fw classid 1:105
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 2 fw classid 1:200
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 3 fw classid 1:300
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 4 fw classid 1:800
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 5 fw classid 1:110
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 6 fw classid 1:120
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 7 fw classid 1:130
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 8 fw classid 1:140
tc filter add dev ${OUTQ} parent 1:0 protocol ip handle 9 fw classid 1:150

tc qdisc add dev ${OUTQ} parent 1:105 handle 105: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:200 handle 200: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:300 handle 300: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:800 handle 800: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:110 handle 110: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:120 handle 120: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:130 handle 130: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:140 handle 140: pfifo limit 5
tc qdisc add dev ${OUTQ} parent 1:150 handle 150: pfifo limit 5

iptables -t mangle -A POSTROUTING --src ${LEO_DES} -p udp -j MARK
--set-mark 5
iptables -t mangle -A POSTROUTING --src ${LEO_DES} -p udp -j RETURN
iptables -t mangle -A POSTROUTING --src ${LEO_DES} -p gre -j MARK
--set-mark 5
iptables -t mangle -A POSTROUTING --src ${LEO_DES} -p gre -j RETURN
...
...
######################################################################

########  Long delay 1800ms #######

class htb 1:110 parent 1:100 leaf 110: prio 0 rate 32Kbit ceil 200Kbit
burst 1639b cburst 1856b
 Sent 3108142 bytes 15541 pkts (dropped 2725, overlimits 0)
 rate 25459bps 127pps backlog 3p
 lended: 2522 borrowed: 13016 giants: 0
 tokens: -1918 ctokens: -2823

class htb 1:1 root rate 220Kbit ceil 220Kbit burst 1880b cburst 1880b
 Sent 3137858 bytes 15954 pkts (dropped 0, overlimits 0)
 rate 25493bps 129pps
 lended: 4990 borrowed: 0 giants: 0
 tokens: 45460 ctokens: 45460

class htb 1:100 parent 1:1 rate 136Kbit ceil 220Kbit burst 1773b cburst
1880b
 Sent 3120904 bytes 15754 pkts (dropped 0, overlimits 0)
 rate 25397bps 128pps
 lended: 8026 borrowed: 4990 giants: 0
 tokens: -6802 ctokens: 45460

##############################

#########  Short Delay 80ms #############

class htb 1:110 parent 1:100 leaf 110: prio 0 rate 32Kbit ceil 180Kbit
burst 1639b cburst 1829b
 Sent 2675988 bytes 13461 pkts (dropped 4043, overlimits 0)
 rate 22888bps 114pps backlog 5p
 lended: 2542 borrowed: 10914 giants: 0
 tokens: -4451 ctokens: -5694

class htb 1:1 root rate 220Kbit ceil 220Kbit burst 1880b cburst 1880b
 Sent 2927810 bytes 15992 pkts (dropped 0, overlimits 0)
 rate 22935bps 116pps
 lended: 3295 borrowed: 0 giants: 0
 tokens: 45460 ctokens: 45460

class htb 1:100 parent 1:1 rate 136Kbit ceil 220Kbit burst 1773b cburst
1880b
 Sent 2801155 bytes 14888 pkts (dropped 0, overlimits 0)
 rate 22834bps 115pps
 lended: 7650 borrowed: 3275 giants: 0
 tokens: -8277 ctokens: 45460

##################################

Leo wrote:
> In the first test, I limit the SUBCLASS_OUTRATE to 200Kbit. Both pings
> are around 20ms before I start the VoIP services. However, once I start
> the services, the pings jump up to 1800ms.
> 
> In the second test, I limit the SUBCLASS_OUTRATE to 180Kbit. The pings
> jump up to 80ms, which is perfectly acceptable.
> 
> After a few tests, I noticed that 180Kbit is a magic number, anything
> exceed that will generate 1800ms pings, and below it is 80ms.
> 
> In my senario, the weird point is that the determining factor is the
> ceiling, but not the rate. That''s the "rate" for other
class doesn''t
> seem to give bandwidth to packets in the corresponding class unless the
> ceil for the 1:110 is low enough!
> 
> I attached my script and "tc -s class show" below. I truncated
part of
> the script and the results to make it short.
> 
> Please shine me a light!
It''s because the link is dsl and there are lots of overheads on each 
packet (and they vary with packet size). HTB rates are based on ip 
packet length and with lots of small packets like voip the difference 
can be alot.

The 1800ms latency is not caused by a queue within htb it''s in your 
modem/router because it can''t send >180kbit ip level for voip.

You can patch HTB and TC to make things perfect - you could set a ceil 
very close to your sync rate then. You need to know exactly what type of 
dsl you are on to find your overhead though. If your modem/router gives 
ATM cell counts you can deduce it from those.

There is a very good thesis and patch info here -

http://www.adsl-optimizer.dk/

Andy.

Oops, forgot to post to the group...

---------- Forwarded message ----------
From: Leo Huang <leo.maillist@gmail.com>
Date: Apr 27, 2005 10:46 AM
Subject: Re: [LARTC] HTB Weird Shaping Question(Bug?). Please Help!
To: Andy Furniss <andy.furniss@dsl.pipex.com>


Thanks Andy,

What you have said makes absolute sense to me. However, I only
"reserved" 136Kbit for the VoIP traffic, there are 44Kbit available
even we assume the 180Kbit is the maximum. Why doesn''t HTB allocate
the 44Kbit to the class for ping traffic, which only require rate
4Kbit and 0.5 Kbit?

Thanks again,
Leo


On 4/27/05, Andy Furniss <andy.furniss@dsl.pipex.com>
wrote:
> Leo wrote:
>
> > In the first test, I limit the SUBCLASS_OUTRATE to 200Kbit. Both pings
> > are around 20ms before I start the VoIP services. However, once I
start
> > the services, the pings jump up to 1800ms.
> >
> > In the second test, I limit the SUBCLASS_OUTRATE to 180Kbit. The pings
> > jump up to 80ms, which is perfectly acceptable.
> >
> > After a few tests, I noticed that 180Kbit is a magic number, anything
> > exceed that will generate 1800ms pings, and below it is 80ms.
> >
> > In my senario, the weird point is that the determining factor is the
> > ceiling, but not the rate. That''s the "rate" for
other class doesn''t
> > seem to give bandwidth to packets in the corresponding class unless
the
> > ceil for the 1:110 is low enough!
> >
> > I attached my script and "tc -s class show" below. I
truncated part of
> > the script and the results to make it short.
> >
> > Please shine me a light!
>
> It''s because the link is dsl and there are lots of overheads on
each
> packet (and they vary with packet size). HTB rates are based on ip
> packet length and with lots of small packets like voip the difference
> can be alot.
>
> The 1800ms latency is not caused by a queue within htb it''s in
your
> modem/router because it can''t send >180kbit ip level for voip.
>
> You can patch HTB and TC to make things perfect - you could set a ceil
> very close to your sync rate then. You need to know exactly what type of
> dsl you are on to find your overhead though. If your modem/router gives
> ATM cell counts you can deduce it from those.
>
> There is a very good thesis and patch info here -
>
> http://www.adsl-optimizer.dk/
>
> Andy.
>
>

> What you have said makes absolute sense to me. However, I only
> "reserved" 136Kbit for the VoIP traffic, there are 44Kbit
available
> even we assume the 180Kbit is the maximum. Why doesn''t HTB
allocate
> the 44Kbit to the class for ping traffic, which only require rate
> 4Kbit and 0.5 Kbit?
Some of this could be do to the fact that an ICMP echo request packet is
extremely small.  It is quite likely that your ADSL connection has a raw
throughput of 256 kbps.  On top of the ADSL signal is a signaling protocol, be
it Frame Relay (older DSL circuits in my town are this) or ATM (newer DSL
circuits), each have their own protocol overhead as well as minimum packet size.
So if you are sending ICMP echo request packets that are very small, they will
have to be wrapped in the network layer (OSI layer 2) packets and transmitted on
the ADSL line (OSI layer 1) thus growing in size.  It is quite likely that the
size of the packets on physical layer are approaching 256 kbps and thus heating
the physical maximum of your circuit.  There is always the fact that ADSL is
half duplex where as SDSL is full duplex.  You would see this as a problem if
you were trying to download something and upl
 oad something at the same time.  Your circuit can only do one thing at a time
thus somethi
ng will have to wait.  You will see this if you are able to FTP a large file out
to a system on the net fast, close to your maximum, yet your VoIP (SIP?) traffic
will start having problems at less than the maximum rate that the physical link
can handle.

Any one care to support or refute this?  I''m mainly going off of what I
have read and discussed with others.  I''m presently going after CCNA
and this is the answer that I would give to a client, but if there is something
better or there is a discussion that is to be had I''m game for it. 
Someone please correct me as I want to learn more.  ;)



Grant. . . .

> There is always the fact that ADSL is half
> duplex where as SDSL is full duplex.  You would see this as a problem if
> you were trying to download something and upload something at the same
> time.  Your circuit can only do one thing at a time thus somethi
> ng will have to wait.  You will see this if you are able to FTP a large
> file out to a system on the net fast, close to your maximum, yet your VoIP
> (SIP?) traffic will start having problems at less than the maximum rate
> that the physical link can handle.
Err, since when is ADSL half duplex ? News to me :)

I think you''ll find if you read up on (A)DSL that it is in fact full
duplex. Going from memory, the frequency range from 25Khz up to 1.1Mhz is
broken up into a fairly large number of subcarriers (52 I think ?) and
some subcarriers are used for the downstream direction (the majority in
the case of ADSL, or half of them for SDSL) while the rest are used for
the upstream direction.

Because it uses frequency division muliplexing for each carrier to exist
on the line simultaneously there is no need to do any time division
multiplexing between upstream and downstream.

Besides, even a normal modem dialup connection is full duplex, it''s
unlikely ADSL would take a backwards step from that ;-)

Regards,
Simon

> Err, since when is ADSL half duplex ? News to me :)
> 
> I think you''ll find if you read up on (A)DSL that it is in fact
full
> duplex. Going from memory, the frequency range from 25Khz up to 1.1Mhz is
> broken up into a fairly large number of subcarriers (52 I think ?) and
> some subcarriers are used for the downstream direction (the majority in
> the case of ADSL, or half of them for SDSL) while the rest are used for
> the upstream direction.
*nod*  This is indeed the case.
> Because it uses frequency division muliplexing for each carrier to exist
> on the line simultaneously there is no need to do any time division
> multiplexing between upstream and downstream.
I''m not sure why ADSL is simplex / half duplex as it seems that it
should be able to be (full) duplex.  I do know that the the types of service
that have been seen here in my town are indicative of simplex traffic.  I
don''t know if this is a limit that the LEC is putting in place or not. 
Also many a provider that I''ve talked to have agreed that ADSL in
particular is simplex.  If you are talking SDSL the circuit is indeed duplex.
> Besides, even a normal modem dialup connection is full duplex,
it''s
> unlikely ADSL would take a backwards step from that ;-)
Keep in mind that 33.6 kbps modems download at 33.6 kbps and upload at 33.6 kbps
where as 56 kbps modems download at 56 kbps and upload at 28.8 kbps.  One reason
that ADSL my be simplex is that simplex technology is easier to implement and
thus has more features or speed on a given circuit.  Thus ADSL could be simplex
allowing for a wider band for download sacrificing some of the upload bandwidth
/ capabilities.  In a residential / small office environment this is typically
not a problem.

Again, I have no paperwork in front of me to back this up, just my experience
and conversations with half a dozen or more people in the ISP business.



Grant. . . .

Taylor, Grant wrote:
>> Err, since when is ADSL half duplex ? News to me :)
>>
>> I think you''ll find if you read up on (A)DSL that it is in
fact full
>> duplex. Going from memory, the frequency range from 25Khz up to 1.1Mhz
is
>> broken up into a fairly large number of subcarriers (52 I think ?) and
>> some subcarriers are used for the downstream direction (the majority in
>> the case of ADSL, or half of them for SDSL) while the rest are used for
>> the upstream direction.
> 
> 
> *nod*  This is indeed the case.
> 
>> Because it uses frequency division muliplexing for each carrier to
exist
>> on the line simultaneously there is no need to do any time division
>> multiplexing between upstream and downstream.
> 
> 
> I''m not sure why ADSL is simplex / half duplex as it seems that it
> should be able to be (full) duplex.  I do know that the the types of 
> service that have been seen here in my town are indicative of simplex 
> traffic.  I don''t know if this is a limit that the LEC is putting
in
> place or not.  Also many a provider that I''ve talked to have
agreed that
> ADSL in particular is simplex.  If you are talking SDSL the circuit is 
> indeed duplex.
> 
>> Besides, even a normal modem dialup connection is full duplex,
it''s
>> unlikely ADSL would take a backwards step from that ;-)
> 
> 
> Keep in mind that 33.6 kbps modems download at 33.6 kbps and upload at 
> 33.6 kbps where as 56 kbps modems download at 56 kbps and upload at 28.8 
> kbps.  One reason that ADSL my be simplex is that simplex technology is 
> easier to implement and thus has more features or speed on a given 
> circuit.  Thus ADSL could be simplex allowing for a wider band for 
> download sacrificing some of the upload bandwidth / capabilities.  In a 
> residential / small office environment this is typically not a problem.
> 
> Again, I have no paperwork in front of me to back this up, just my 
> experience and conversations with half a dozen or more people in the ISP 
> business.
I''ve never heard of half duplex adsl - your modem should tell you up
and
down sync rates and that''s what you''ve got.

It is very easy to make it look like it though, just start an upload 
while downloading something and watch your down rate fall apart. This is 
TCP and the fact that adsl modems tend to have huge buffers not the 
link. The first thing to do when making a shaping script is priorotise 
empty acks - then uploading doesn''t trash your download.

Andy.

On Wednesday 27 April 2005 14:56, Andy Furniss wrote:
<snip>
> It is very easy to make it look like it though, just start an upload
> while downloading something and watch your down rate fall apart. This is
> TCP and the fact that adsl modems tend to have huge buffers not the
> link. The first thing to do when making a shaping script is priorotise
> empty acks - then uploading doesn''t trash your download.
As much.

If you''re fully saturating your downstream, too, you''re
queuing on your ISP
and then bulk downloading interferes with interactive traffic.

The joys of having an asymmetrical connection, eh?

Leo Huang wrote:
> What you have said makes absolute sense to me. However, I only
> "reserved" 136Kbit for the VoIP traffic, there are 44Kbit
available
> even we assume the 180Kbit is the maximum. Why doesn''t HTB
allocate
> the 44Kbit to the class for ping traffic, which only require rate
> 4Kbit and 0.5 Kbit?
I don''t quite know what you mean here. If there was a queue of ICMP 
traffic then htb would give it 44kbit. If you want icmp times to be 
lower then there are things you can do like setting quantum size on htb 
leafs and setting a define - hysteris in the source code to 0, this 
makes it more accurate.

Andy.