LARTC - Nov 2005 - Borrowing between HTB classes not working as expectd.

I''m using a fairly large number of classes, andf borrowing is not
working as expected... I''ve called this setting it up on an IMQ device
with speed 1200/256 on a 1536/384 line. I''m then throwing a UDP data
transfer at it that gets tossed in one of the class under parent 1:6.
The classification is working fine, but when I try to ping out, ping
times are in the 900ms range, even though pings are classified in the
classes under 1:5, and speeds in classes under 1:4 are well under the
60% they should be getting. Any idea what I''ve done wrong here? Perl
code to set everything up below, if you comment out the system call,
it''ll list off all the tc commands executed. How I wanted this to work
is higher priority classes to get a shot at bandwidth first in excess
of the base rates on all the other classes.

#!/usr/bin/perl -w
use strict;

my $dev = $ARGV[0];
my $speed_down = $ARGV[1];
my $speed_up = $ARGV[2];

my $tc_bin = "/sbin/tc";

tc("qdisc del dev $dev root");

tc("qdisc add dev $dev root handle 1:0 htb r2q 1 default 4");
tc("class add dev $dev parent 1:0 classid 1:2 htb ".
                      "rate " . int($speed_up * 0.85) . "kbit
".
                      "ceil " . int($speed_up * 0.85) . "kbit
".
                      "prio 1 burst 64k");

tc("qdisc add dev $dev parent 1:2 handle 2: sfq perturb 10");

# Critical class
tc("class add dev $dev parent 1:2 classid 1:3 htb ".
                      "rate " . int($speed_down * 0.05 + 64) .
"kbit ".
                      "ceil " . int($speed_down * 0.05 + 64 +
($speed_up * 0.10)) . "kbit ".
                      "prio 2 burst 64k");

tc("qdisc add dev $dev parent 1:3 handle 3: sfq perturb 10");

# High class
tc("class add dev $dev parent 1:2 classid 1:4 htb ".
                      "rate " . int($speed_up * 0.50) . "kbit
".
                      "ceil " . int($speed_up * 0.95) . "kbit
".
                      "prio 3 burst 64k");

tc("qdisc add dev $dev parent 1:4 handle 4: sfq perturb 10");

# Med class
tc("class add dev $dev parent 1:2 classid 1:5 htb ".
                      "rate " . int($speed_up * 0.40) . "kbit
".
                      "ceil " . int($speed_up * 0.95) . "kbit
".
                      "prio 5 burst 64k");

tc("qdisc add dev $dev parent 1:5 handle 5: sfq perturb 10");

# Low class
tc("class add dev $dev parent 1:2 classid 1:6 htb ".
                      "rate " . int($speed_up * 0.10) . "kbit
".
                      "ceil " . int($speed_up * 0.80) . "kbit
".
                      "prio 7 burst 64k");

tc("qdisc add dev $dev parent 1:6 handle 6: sfq perturb 10");

for (my $i=100;$i <= 110;$i++)
{
  # High class
  tc("class add dev $dev parent 1:4 classid 1:" . ($i + 256) . "
htb ".
                        "rate " . int($speed_up * 0.50) . "kbit
".
                        "ceil " . int($speed_up * 0.95) . "kbit
".
                        "prio 4 burst 15k");

  tc("qdisc add dev $dev parent 1:" . ($i + 256) . " handle
" . ($i +
256) . ": sfq perturb 10");
  tc("filter add dev $dev protocol ip parent 1: prio 5 handle " .
tohex($i + 256) . " fw flowid 1:" . ($i + 256));

  # Med class
  tc("class add dev $dev parent 1:5 classid 1:" . ($i + 512) . "
htb ".
                        "rate " . int($speed_up * 0.40) . "kbit
".
                        "ceil " . int($speed_up * 0.95) . "kbit
".
                        "prio 6 burst 15k");

  tc("qdisc add dev $dev parent 1:" . ($i + 512) . " handle
" . ($i +
512) . ": sfq perturb 10");
  tc("filter add dev $dev protocol ip parent 1: prio 5 handle " .
tohex($i + 512) . " fw flowid 1:" . ($i + 512));

  # Low class
  tc("class add dev $dev parent 1:6 classid 1:" . ($i + 768) . "
htb ".
                        "rate " . int($speed_up * 0.10) . "kbit
".
                        "ceil " . int($speed_up * 0.80) . "kbit
".
                        "prio 8 burst 15k");

  tc("qdisc add dev $dev parent 1:" . ($i + 768) . " handle
" . ($i +
768) . ": sfq perturb 10");
  tc("filter add dev $dev protocol ip parent 1: prio 5 handle " .
tohex($i + 768) . " fw flowid 1:" . ($i + 768));
}

tc("filter add dev $dev protocol ip parent 1: prio 5 handle 0x3 fw flowid
1:3");

sub tc
{
  my $arg = shift;
  print "$tc_bin $arg\n";
  system($tc_bin,split(/ /,$arg));
}
sub tohex
{
  return ''0x'' . sprintf("%2.2x",shift);
}
--
Ryan Castellucci http://ryanc.org/

Quoting Ryan Castellucci <ryan.castellucci@gmail.com>:
> I''ve called this setting it up on an IMQ device with speed
1200/256
> on a 1536/384 line.
> Perl code to set everything up below, if you comment out the system call,
> it''ll list off all the tc commands executed.
Hmmm, the output it produces for me is really weird. Could you post the output
of ''tc -d qdisc show dev <device>'' and ''tc -d
class show dev <device>''
somewhere, it''s easier to understand and maybe that will make more 
sense to me.

The main problems in the output I get is that rates do not add up; children
class rates added up together is higher than parent class rate, which 
means the
children take more bandwidth for guaranteed than the parent can offer. 
Sometimes
child class ceil rates is greater than parent class ceil rate, which 
should not
be exceeded in any case (at least to my understanding of HTB).

By looking at your Perl code, you seem to be mixing up $speed_up and 
$speed_down
quite a lot; at least to me it makes no sense to have a download class 
as child
of an upload class, especially not with that much difference in available
download/upload rate. But I''m not really a Perl programmer, so maybe I
misunderstood this part.

HTH
Andreas

I did not mix these up. I''m using the 1:2 class for TCP and ICMP
control packets, such as TCP acks which need an amount of bandwidth
proportinat to the makimum download rate.


On 11/12/05, andreas.klauer@metamorpher.de
<andreas.klauer@metamorpher.de> wrote:
> Quoting Ryan Castellucci <ryan.castellucci@gmail.com>:
>
> > I''ve called this setting it up on an IMQ device with speed
1200/256
> > on a 1536/384 line.
>
> > Perl code to set everything up below, if you comment out the system
call,
> > it''ll list off all the tc commands executed.
>
> Hmmm, the output it produces for me is really weird. Could you post the
output
> of ''tc -d qdisc show dev <device>'' and ''tc
-d class show dev <device>''
> somewhere, it''s easier to understand and maybe that will make more
> sense to me.
>
> The main problems in the output I get is that rates do not add up; children
> class rates added up together is higher than parent class rate, which
> means the
> children take more bandwidth for guaranteed than the parent can offer.
> Sometimes
> child class ceil rates is greater than parent class ceil rate, which
> should not
> be exceeded in any case (at least to my understanding of HTB).
>
> By looking at your Perl code, you seem to be mixing up $speed_up and
> $speed_down
> quite a lot; at least to me it makes no sense to have a download class
> as child
> of an upload class, especially not with that much difference in available
> download/upload rate. But I''m not really a Perl programmer, so
maybe I
> misunderstood this part.
I did not mix these up. I''m using the 1:2 class for TCP and ICMP
control packets, such as TCP acks which need an amount of bandwidth
proportinate to the maximum download rate.

The tc show output is attached.

--
Ryan Castellucci http://ryanc.org/


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Quoting Ryan Castellucci
<ryan.castellucci@gmail.com>:
> I did not mix these up. I''m using the 1:2 class for TCP and ICMP
> control packets, such as TCP acks which need an amount of bandwidth
> proportinate to the maximum download rate.
There seems to be a misunderstanding of some kind. You say you''re using
the 1:2
class for control packets; but in the output you''ve sent, the 1:2 class
is the
root HTB class, so it should be (indirectly) used for everything. The only
classes you can use directly (that means classify packets to) are the leaf
classes (HTB classes which don''t have any more children), in your setup
that
would be one of the 1:3,356-361,612-617,869-873 leaf classes.

Class 1:2 has a rate/ceil of 217kbit. Children of this class are 1:3 
(124/149),
1:4 (128/243), 1:5 (102/243), and 1:6 (25/204). As I said before, the problem
is that the rates of these classes don''t add up. These child classes
added
together for example use 124+128+102+25=379kbit, although the parent provides
only 217kbit. Classes 1:4 and 1:5 in particular can borrow up to 243kbit each,
although the parent class can provide only 217kbit in total. So how exactly do
you expect the borrowing to work? Unless you have an understanding of 
the inner
workings of HTB in great detail, the results of this setup are pretty much
unpredictable.

The same problem can be found further down the tree; for example, the 
class 1:4
has a rate of 128kbit. Children of this class are 1:356-361, with a rate of
128kbit each. Added together, they require a rate of 768kbit, but the parent
class only provides 128kbit (or it would if the parent class of this parent
class could provide as much).

Same story with 1:5 and 1:6.

The first thing you have to do is calculate the class rates so they add up
properly. Otherwise you will never get anywhere near a predictable borrowing
behaviour.

HTH
Andreas Klauer

On 11/13/05, andreas.klauer@metamorpher.de
<andreas.klauer@metamorpher.de> wrote:
> Quoting Ryan Castellucci <ryan.castellucci@gmail.com>:
> > I did not mix these up. I''m using the 1:2 class for TCP and
ICMP
> > control packets, such as TCP acks which need an amount of bandwidth
> > proportinate to the maximum download rate.
>
> There seems to be a misunderstanding of some kind. You say you''re
using
> the 1:2
> class for control packets; but in the output you''ve sent, the 1:2
class is the
> root HTB class, so it should be (indirectly) used for everything.
Erp, I ment 1:3.....
> The only
> classes you can use directly (that means classify packets to) are the leaf
> classes (HTB classes which don''t have any more children), in your
setup that
> would be one of the 1:3,356-361,612-617,869-873 leaf classes.
>
> Class 1:2 has a rate/ceil of 217kbit. Children of this class are 1:3
> (124/149),
> 1:4 (128/243), 1:5 (102/243), and 1:6 (25/204). As I said before, the
problem
> is that the rates of these classes don''t add up. These child
classes added
> together for example use 124+128+102+25=379kbit, although the parent
provides
> only 217kbit. Classes 1:4 and 1:5 in particular can borrow up to 243kbit
each,
> although the parent class can provide only 217kbit in total. So how exactly
do
> you expect the borrowing to work? Unless you have an understanding of
> the inner
> workings of HTB in great detail, the results of this setup are pretty much
> unpredictable.
>
> The same problem can be found further down the tree; for example, the
> class 1:4
> has a rate of 128kbit. Children of this class are 1:356-361, with a rate of
> 128kbit each. Added together, they require a rate of 768kbit, but the
parent
> class only provides 128kbit (or it would if the parent class of this parent
> class could provide as much).
>
> Same story with 1:5 and 1:6.
>
> The first thing you have to do is calculate the class rates so they add up
> properly. Otherwise you will never get anywhere near a predictable
borrowing
> behaviour.
I''ll go though and make sure everything adds up, and try it again.

--
Ryan Castellucci http://ryanc.org/