LARTC - Dec 2002 - how to get the latency down on maxed out classes?

Hi!

I''m using HTB to shape traffic to/from clients, but one of the problems
I
have is that once a class utilizes its maximum potential, they latency
spirals out of control.

For example:

     .-----.
     | isp |
     `-----''
        |
   .--------.
   | shaper |
   `--------''
        |
   .--------.
   | client |
   `--------''

lets say I want to limit traffic to/from client to 64kbit. now, client opens
a tcp connection blasting away at full speed.

If client now pings isp, it gets on average around 7 seconds latency. I
tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy,
but that does not really improve the situation, only makes it much worse.
with SFQ I get anything between 250ms and 13 seconds latency.

I then tried fifos. With small packet fifos the packet loss is just
to great to be of any use and even then the latency is quite high (~200ms).
If I make the fifos big enough so that unreasonable numbers of packets
isn''t
dropped, it just doesn''t do much to the latency/or throughput. This
behaviour kind of makes sense, but doesn''t help me :P

I''m thinking of using RED, but the number of parameters is daunting and
I
have no idea how the HTB rate correlates to packet size and burst rates for
red.

Does anybody have any idea how to get the latency down and still maintain
the correct throughput?

-- 

Regards
 Abraham

Sooner or later you must pay for your sins.
(Those who have already paid may disregard this cookie).

___________________________________________________
 Abraham vd Merwe [ZR1BBQ] - Frogfoot Networks
 P.O. Box 3472, Matieland, Stellenbosch, 7602
 Cell: +27 82 565 4451 Http: http://www.frogfoot.net
 Email: abz@frogfoot.net

Hi


even i have same problem using CBQ.INIT
let me know if you get any solution
thanks
hare
----- Original Message ----- 
From: "Abraham van der Merwe" <abz@frogfoot.net>
To: "Linux Advanced Routing & Traffic Control list"
<lartc@mailman.ds9a.nl>
Sent: Saturday, December 07, 2002 6:45 PM
Subject: [LARTC] how to get the latency down on maxed out classes?



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Saturday 07 December 2002 14:15, Abraham van der Merwe
wrote:
> Hi!
>
> I''m using HTB to shape traffic to/from clients, but one of the
problems I
> have is that once a class utilizes its maximum potential, they latency
> spirals out of control.
>
> For example:
>
>      .-----.
>
>      | isp |
>
>      `-----''
>
>    .--------.
>
>    | shaper |
>
>    `--------''
>
>    .--------.
>
>    | client |
>
>    `--------''
>
> lets say I want to limit traffic to/from client to 64kbit. now, client
> opens a tcp connection blasting away at full speed.
>
> If client now pings isp, it gets on average around 7 seconds latency. I
> tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy,
> but that does not really improve the situation, only makes it much worse.
> with SFQ I get anything between 250ms and 13 seconds latency.
>
> I then tried fifos. With small packet fifos the packet loss is just
> to great to be of any use and even then the latency is quite high (~200ms).
> If I make the fifos big enough so that unreasonable numbers of packets
> isn''t dropped, it just doesn''t do much to the latency/or
throughput. This
> behaviour kind of makes sense, but doesn''t help me :P
>
> I''m thinking of using RED, but the number of parameters is
daunting and I
> have no idea how the HTB rate correlates to packet size and burst rates for
> red.
>
> Does anybody have any idea how to get the latency down and still maintain
> the correct throughput?
If you want low latency for some traffic (ping, telnet, ssh), then you can 
create a separate class for it. 
And if you have a 64kbit modem, you have to be sure you never send more data 
then the modem can handle.  If you send more data, the hugh modem
queue''s
will be filled so they create a lot of latency.  So for a 64kbit link, try to 
limit ALL traffic to 60kbit so the queue''s of the modem are never
filled.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hi Stef!
> > Does anybody have any idea how to get the latency down and still
maintain
> > the correct throughput?
> If you want low latency for some traffic (ping, telnet, ssh), then you can 
> create a separate class for it. 
> And if you have a 64kbit modem, you have to be sure you never send more
data
> then the modem can handle.  If you send more data, the hugh modem
queue''s
> will be filled so they create a lot of latency.  So for a 64kbit link, try
to
> limit ALL traffic to 60kbit so the queue''s of the modem are never
filled.
I''m doing all my tests under ideal conditions (over 100mbit lan and
shaping
the traffic to something low such as 384kbit). The problem is that the
latency becomes unnatural.

If you have a normal line, e.g. 64kbit and you saturate the line, the
latency still stays within limits, but with HTB, the latency can become very
high if you have multiple concurrent tcp sessions going at full steam in a
class. I can understand why it does this, but I need a way to get the
latency down to acceptable limits.


-- 

Regards
 Abraham

I never killed a man that didn''t deserve it.
		-- Mickey Cohen

___________________________________________________
 Abraham vd Merwe [ZR1BBQ] - Frogfoot Networks
 P.O. Box 3472, Matieland, Stellenbosch, 7602
 Cell: +27 82 565 4451 Http: http://www.frogfoot.net
 Email: abz@frogfoot.net

On Saturday 07 December 2002 17:49, Abraham van der Merwe
wrote:
> Hi Stef!
>
> > > Does anybody have any idea how to get the latency down and still
> > > maintain the correct throughput?
> >
> > If you want low latency for some traffic (ping, telnet, ssh), then you
> > can create a separate class for it.
> > And if you have a 64kbit modem, you have to be sure you never send
more
> > data then the modem can handle.  If you send more data, the hugh modem
> > queue''s will be filled so they create a lot of latency.  So
for a 64kbit
> > link, try to limit ALL traffic to 60kbit so the queue''s of
the modem are
> > never filled.
>
> I''m doing all my tests under ideal conditions (over 100mbit lan
and shaping
> the traffic to something low such as 384kbit). The problem is that the
> latency becomes unnatural.
>
> If you have a normal line, e.g. 64kbit and you saturate the line, the
> latency still stays within limits, but with HTB, the latency can become
> very high if you have multiple concurrent tcp sessions going at full steam
> in a class. I can understand why it does this, but I need a way to get the
> latency down to acceptable limits.
The only thing where I can think of, is adding a small fifo to each class.  
But you already tried and it created packets loss.  So I can''t help
you.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> If you have a normal line, e.g. 64kbit and you saturate the line, the
> latency still stays within limits, but with HTB, the latency can become
> very high if you have multiple concurrent tcp sessions going at full steam
> in a class. I can understand why it does this, but I need a way to get the
> latency down to acceptable limits.
Are you sure the latency and packet loss are not the same,
when you have the same queue size as the router has ?

Is the traffic locally generated (on the same machine as the
HTB is running) ? If so then perhaps that''s the difference.



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> lets say I want to limit traffic to/from client to 64kbit. now, client
opens
 > a tcp connection blasting away at full speed.
 > 
 > If client now pings isp, it gets on average around 7 seconds latency. I
 > tried to improve this by using SFQ on the leaf nodes of my HTB hierarchy,
 > but that does not really improve the situation, only makes it much worse.
 > with SFQ I get anything between 250ms and 13 seconds latency.

You understand what''s going on here?
As I recall, both pfifo and sfq default to queues of length 128
packets.  If you fill that with 1500 byte packets you have ~200Kbytes
which is about 1.6Mbits.  At 64Kbit/sec that would take ~30 sec to
send so your latency could be as high as 30 sec.
You can limit this latency by reducing the queue size.

On the other hand, the application that fills the queue evidently
doesn''t mind large latency.  Otherwise it wouldn''t fill the
queue.

I think I posted to this list once a description (maybe even the
code?) of another way to limit latency - drop packets that have been
in the queue for more than a timeout period (I tend to use 3 sec).

SFQ should have the desirable result that one tcp connection won''t
slow down another one or a ping.

 > I then tried fifos. With small packet fifos the packet loss is just
 > to great to be of any use and even then the latency is quite high
(~200ms).
You consider 200ms high?  One max size packet = 1500 bytes = 12Kbit
which is about 200ms on a 64Kbit link.  You can''t expect to do better.

 > I''m thinking of using RED, but the number of parameters is
daunting and I
 > have no idea how the HTB rate correlates to packet size and burst rates
for
 > red.
RED should be independent of HTB.  
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hi Don!
>  > I then tried fifos. With small packet fifos the packet loss is just
>  > to great to be of any use and even then the latency is quite high
(~200ms).
> You consider 200ms high?  One max size packet = 1500 bytes = 12Kbit
> which is about 200ms on a 64Kbit link.  You can''t expect to do
better.
The problem is that with 200ms the packet loss is so much that the link is
effectively useless (90% packet loss). As soon as I make the queue big
enough to not drop significant amounts of packets, the latency goes way up
(>3 secs).

Abraham van der Merwe writes:
 > Hi Don!
 > 
 > >  > I then tried fifos. With small packet fifos the packet loss is
just
 > >  > to great to be of any use and even then the latency is quite
high (~200ms).

A small detail: what are "small packet fifos"?  You mean fifos that
can only hold a small number of packets?  Or fifos that only hold
packets with small numbers of bytes?

 > > You consider 200ms high?  One max size packet = 1500 bytes = 12Kbit
 > > which is about 200ms on a 64Kbit link.  You can''t expect to
do better.
 > 
 > The problem is that with 200ms the packet loss is so much that the link is
 > effectively useless (90% packet loss). As soon as I make the queue big
 > enough to not drop significant amounts of packets, the latency goes way up
 > (>3 secs).

I don''t understand the connection between 200ms and packet loss.
If you make the queue small (in packet capacity) then worst case
latency decreases.  Packet loss occurs in either case whenever a
packet arrives and the queue is full.

If you try to send at a higher rate than allowed then you will fill
the queue in either case (a small queue more quickly, of course),
and from then on you will lose packets.  If you send packets at twice 
the allowed rate you lose half of them, if you send at 10 times the
allowed rate you lose 90%.

The fact that you''re losing lots of packets, though, indicates to me
that you''re acting like an attacker, and dropping most of that traffic
is therefore exactly the right thing to do.

If you were using a correctly working tcp it would not continue to
send at 10 times the allowed rate.  It would notice that packets were
being lost and would slow down until the loss rate became very small.

Similarly, I don''t understand the latency issue.  An application that
cares about latency will not create a large backlog.

What is this application that is sending faster than the link allows
and wants a low latency, and why is it misbehaving?
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> Similarly, I don''t understand the latency issue.  An application
that
> cares about latency will not create a large backlog.
I think the problem may be that a TCP flow will create a queue
of packets, behind which all other packets queue. Thus one sees
200ms latency via ''ping''. Of course this is exactly what you
would
see on a real connection of the specified throttled speed (e.g. a 
DSL line). However it seems to surprise folks that there''s a
relationship
between throughput and latency. They expect to be able to throttle
traffic to 100Kbits, but still see 5ms latency. Without special
measures that isn''t going to happen.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

I''ve been using HTB for a while now and been tweaking and experimenting
with
it a lot. I haven''t got it perfect yet, but it''s a lot better
than nothing
(both terms of bandwidth efficiency and latency under load).

Basically what I do is create a few classes with different priorities: 0, 1,
2 and 3 (lower value, higher priority). Then I add a filter rule to each
class that sends all packets with a specific iptables MARK to that class.
Then I create some iptables rules that put the traffic in the correct class,
which means that all game traffic (and for testing all ICMP packets) go into
priority 0. Ssh, telnet, ACK packets, etc. go into 1. Http, ftp go into 2
and all other packets (kazaa/direct connect and other unknown stuff) is left
with priority 3. I limit the total upstream traffic to 100kbit (of the
128kbit my ISP provides me). I''m pretty sure I could go up a little
higher
when I get the details tweaked right. I set rates and ceilings for each
class, although (in theory) the priorities should take care of it and it
should be possible to have each class use what they want.

My question here is: why does HTB not permit me to use more than 4
priorities? Some documentation I''ve seen says I should be able to go as
high
as priority 7. Maybe my HTB version is too old? Anyway, my linux box is a
mess, I''ll re-install it sometime soon and start using 2.4.20.

I use iptables rules like these for the marking:
iptables -I PREROUTING -t mangle -i eth2 --jump MARK --set-mark 1 -p ICMP
iptables -I PREROUTING -t mangle -i eth0 --jump MARK --set-mark 1 -p ICMP


Another trick I use is reducing the maximum (tcp) packet size, using (which
is good for the latency). I''ve seen other scripts that do the same
using
different MTU settings and even custom routing tables with maximum MTU size.
I haven''t really thought about the differences much, but this trick
effectively gives the same results:
iptables -I PREROUTING -t mangle -i eth2 --jump TCPMSS --set-mss 700 -p
TCP --tcp-flags SYN,RST SYN
iptables -I PREROUTING -t mangle -i eth0 --jump TCPMSS --set-mss 700 -p
TCP --tcp-flags SYN,RST SYN

And another important thing to do is change a simple kernel setting, which
improves HTB accuracy enormously, which is nice when you''re tweaking
the
settings: in /usr/src/linux/include/net/pkt_sched.h change
PSCHED_CLOCK_SOURCE to PSCHED_CPU if you have a cpu with timestamp counter
(TSC) that will give you Mhz timer granularity.

Don''t forget to change the interfacenames in the above lines (and the
script
below).

After having used a pretty complicated script, I recently started rewriting
it, so far I made this (no sfq or other stuff, just bare classes).

#!/bin/sh

INET_INTERFACE="eth0"

PRIOS=''0 1 2 3''

TC_UPLINK_RATE="100"
TC_RATE[0]=50
TC_CEIL[0]=110
TC_RATE[1]=15
TC_CEIL[1]=60
TC_RATE[2]=10
TC_CEIL[2]=60
TC_RATE[3]=5
TC_CEIL[3]=30

# Packet marks (which prioritiy is linked to which MARK value)
MARK[0]="1"
MARK[1]="2"
MARK[2]="3"
MARK[3]="4"


# Executables and stuff
IPTABLES="iptables"
TC="tc"
IP="ip"
LOG="/dev/null"

# Comment the next two lines to really run the tc commands.
TC="echo tc"
LOG="/dev/stdout"

# Find last prio, which will be the default
for PRIO in $PRIOS
do
 LAST_PRIO=$PRIO
done

# The TC part
$TC qdisc del dev $INET_INTERFACE root

$TC qdisc add dev $INET_INTERFACE root handle 1:0 htb default
$[$LAST_PRIO+10]
$TC class add dev $INET_INTERFACE parent 1:0 classid 1:1 htb rate
${TC_UPLINK_RATE}kbit

for PRIO in $PRIOS
do
 echo -e "\n***** Prio $PRIO *****" > $LOG
 # Add leaf classes for PRIO traffic
 $TC class add dev $INET_INTERFACE parent 1:1  \
   classid 1:$[$PRIO+10] htb                 \
   rate ${TC_RATE[$PRIO]}"kbit"              \
   ceil ${TC_CEIL[$PRIO]}"kbit"              \
     prio $PRIO

 # Now filter PRIO traffic to this leaf
 $TC filter add dev $INET_INTERFACE parent 1:0         \
     protocol ip prio $PRIO handle ${MARK[$PRIO]}      \
   fw flowid 1:$[$PRIO+10]
done

Anyway, hope it helps someone.

Jannes Faber

----- Original Message -----
From: "Don Cohen" <don-lartc@isis.cs3-inc.com>
To: <lartc@mailman.ds9a.nl>; <abz@frogfoot.net>
Sent: Sunday, December 08, 2002 6:30 PM
Subject: [LARTC] how to get the latency down on maxed out classes?

> > lets say I want to limit traffic to/from client to 64kbit. now, client
opens
>  > a tcp connection blasting away at full speed.
>  >
>  > If client now pings isp, it gets on average around 7 seconds latency.
I
>  > tried to improve this by using SFQ on the leaf nodes of my HTB
hierarchy,
>  > but that does not really improve the situation, only makes it much
worse.
>  > with SFQ I get anything between 250ms and 13 seconds latency.
>
> You understand what''s going on here?
> As I recall, both pfifo and sfq default to queues of length 128
> packets.  If you fill that with 1500 byte packets you have ~200Kbytes
> which is about 1.6Mbits.  At 64Kbit/sec that would take ~30 sec to
> send so your latency could be as high as 30 sec.
> You can limit this latency by reducing the queue size.
>
> On the other hand, the application that fills the queue evidently
> doesn''t mind large latency.  Otherwise it wouldn''t fill
the queue.
>
> I think I posted to this list once a description (maybe even the
> code?) of another way to limit latency - drop packets that have been
> in the queue for more than a timeout period (I tend to use 3 sec).
>
> SFQ should have the desirable result that one tcp connection won''t
> slow down another one or a ping.
>
>  > I then tried fifos. With small packet fifos the packet loss is just
>  > to great to be of any use and even then the latency is quite high
(~200ms).
> You consider 200ms high?  One max size packet = 1500 bytes = 12Kbit
> which is about 200ms on a 64Kbit link.  You can''t expect to do
better.
>
>  > I''m thinking of using RED, but the number of parameters is
daunting and
I
>  > have no idea how the HTB rate correlates to packet size and burst
rates
for
>  > red.
> RED should be independent of HTB.
>
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> TC_UPLINK_RATE="100"
> TC_RATE[0]=50
> TC_CEIL[0]=110
So this class has ceil = 110kbit, but the parent class has ceil = 100kbit.  
The parent ceil is not respected so this class can use 110kbit if it wants.  
I don''t think this is what you want.
> TC_RATE[1]=15
> TC_CEIL[1]=60
> TC_RATE[2]=10
> TC_CEIL[2]=60
> TC_RATE[3]=5
> TC_CEIL[3]=30
There is also a drawbeck if you use prio to improve latency.  Devik did some 
testing.  A lower prio is good for delays IF the class with the lower prio 
never sends more then it''s rate (so it''s never overlimited). 
If it do so,
other classes are served first and the delays will be very bad.  I
haven''t
test it myself, but you can find it in the user guide of htb chapter 7 
(http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio).
You can force a class to never exceeds its rate if you use a policer in the 
filter so packets that exceeds the rate are dropped by the filter.  But 
again, I haven''t test it.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> > TC_UPLINK_RATE="100"
> > TC_RATE[0]=50
> > TC_CEIL[0]=110
> So this class has ceil = 110kbit, but the parent class has ceil = 100kbit.
> The parent ceil is not respected so this class can use 110kbit if it
wants.
> I don''t think this is what you want.

Oops, you''re right. I had been testing it with parent rate at 110, but
obviously forgot to put it back down. Also, I presumed the parent ceil would
be respected, so thanks for the tip.
> > TC_RATE[1]=15
> > TC_CEIL[1]=60
> > TC_RATE[2]=10
> > TC_CEIL[2]=60
> > TC_RATE[3]=5
> > TC_CEIL[3]=30
> There is also a drawbeck if you use prio to improve latency.  Devik did
some
> testing.  A lower prio is good for delays IF the class with the lower prio
> never sends more then it''s rate (so it''s never
overlimited).  If it do so,
> other classes are served first and the delays will be very bad.  I
haven''t
> test it myself, but you can find it in the user guide of htb chapter 7
> (http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio).
> You can force a class to never exceeds its rate if you use a policer in
the
> filter so packets that exceeds the rate are dropped by the filter.  But
> again, I haven''t test it.
I need the low-latency class for games (CounterStrike), so I can''t
afford to
drop any packets (I guess). On the other hand it typically doesn''t use
too
much bandwidth, so it can lend the rest.

I had read devik''s manual (I''ve read a lot on htb stuff, but I
can''t claim
to remember it all, let alone put it in practise) but I don''t see you
statement "A lower prio is good for delays IF the class with the lower prio
never sends more then it''s rate (so it''s never
overlimited)." in it. In
fact, it says that the higher prio class gets "excess bandwidth
first",
which is what I want.

But you are probably right that my solution is way too crude for practical
purposes. Maybe I should only be using prio for the game-packets and put all
other classes in the same prio (which is also mentioned at devik''s
site).

Like I suspected, my current script is very basic, I need quite some more
work on it.

Anyway, thanks again Stef, you''re doing a great job!

Jannes Faber

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> I need the low-latency class for games (CounterStrike), so I can''t
afford
> to drop any packets (I guess). On the other hand it typically
doesn''t use
> too much bandwidth, so it can lend the rest.
>
> I had read devik''s manual (I''ve read a lot on htb stuff,
but I can''t claim
> to remember it all, let alone put it in practise) but I don''t see
you
> statement "A lower prio is good for delays IF the class with the lower
prio
> never sends more then it''s rate (so it''s never
overlimited)." in it. In
> fact, it says that the higher prio class gets "excess bandwidth
first",
> which is what I want.
http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio
At the end of the paragraph, Devik did some tests.  He gave www lower prio and 
yes, the delays are very low.  But www is never sending more then the rate. 
At point 7, he starts sending more data to the www class then the configured 
rate.  And see what happens, the www delay increases a lot (blue line) and 
the delay of the other classes drops nearly to zero (yellow line).
> But you are probably right that my solution is way too crude for practical
> purposes. Maybe I should only be using prio for the game-packets and put
> all other classes in the same prio (which is also mentioned at
devik''s
> site).
If it works, why changing ?? 
> Like I suspected, my current script is very basic, I need quite some more
> work on it.
>
> Anyway, thanks again Stef, you''re doing a great job!
No problem.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/