ISC Robert Krycza³o wrote:

>Hello,
>I have a question regarding htb and priorities.
>
>I want to limit my customers to some rate (let's say 512kbit/sek) and to
>guarantee them for example 64kbit/sek on my links to the internet. I want
>to divide it further in order to decrease latency, speed up interactive
>activities, but allow them to do bulk downloads. I created htb tree
>looking like that. B and C are customers. Customer classes are divided
>into low latency class(DNS, small SSH and telnet packets, fast FPP games),
>http traffic class and other traffic class.
>
>Situation looks like this: (in reality we have several uplinks and also
>customer sites are connected by backbone, which also have limited
>bandwidth capacity - tree is simplified)
>
>       A
>     /   \
>    B     C
>   /|\   /|\
>  D E F G H I
>
>Class B and C have prio=3,
>D and G have prio=1 (rate and ceiling are the same)
>E and H have prio=2 (rate and ceiling are the same)
>F and I have prio=3 (rate and ceiling are the same)
>
>I have following questions:
>
>Remaining bandwidth inside class B is distributed first to class D, then E
>and then F and is limited by ceiling parameter . Right???
>
Hi Robert,

      yes, what you have said is right.

>Class A has available bandwidth. Rules for guaranteed rates for classes
>D,E,F,G,H,I are fulfilled. So available bandwidth has to
>be distributed between class B and C equaly (assuming B and C has the same
>rate and prio). Is remaining bandwidth distributed to classes D and G, and
>then to classes E and H and at the end to classes F and I???
>
>
  I remember having read something about the "rate" parameter of a
parent HTB class.  I think it was that the "rate" parameter isn't
used,
only the "ceil" parameter (of a parent HTB class) is important.  Check
the list archive and the HTB home page because I'm not sure.

  If what I have written is true, there is a possibility that bandwidth
is not distributed equally between classes B and C.

>What if C and B have different rates?
>
>Is prio parameter taken into account when htb tries to meet guaranteed
>rate rules?
>
   I think the "prio" parameter is only used after all classes have
reached their guaranteed minimum rate, to allow the user to create
classes that will borrow bandwidth over other classes.

> And when packets are send?
>
>

   It has nothing to do with the prio parameter.

>What happens when sum of guaranteed rates of children class is bigger than
>guaranteed rate of parent (rate parameter is overbooked) and all of
>classes are requesting maximum bandwidth? Are classes with lower prio
>given bandwidth first?
>
>
    There are rules that you should respect when creating classes.
 Check the FAQ on the HTB home site.

>Are packets classfied to class D and G sent first?
>
>
   No, unless classes D and G haven't reached their guaranteed minimum rate.

>What will happen if prio of class B is 0 and class C is 3? I assume
>remaining bandwidth is first distributed to class B and to its children.
>Right???
>
>
>
    Same answer regarding parent HTB classes.  I'm not sure.

>Thx in advance for your answers
>
>Robert Kryczalo
>
>
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>
>
>

> >Remaining bandwidth inside class B is distributed first to class D, th=
en E
> >and then F and is limited by ceiling parameter . Right???
>       yes, what you have said is right.
Confirmed.  Lowest prio classes are allowed to send first.

> >Class A has available bandwidth. Rules for guaranteed rates for classe=
s
> >D,E,F,G,H,I are fulfilled. So available bandwidth has to
> >be distributed between class B and C equaly (assuming B and C has the =
same
> >rate and prio). Is remaining bandwidth distributed to classes D and G,=
 and
> >then to classes E and H and at the end to classes F and I???
Yes.

>   I remember having read something about the "rate" parameter of
a
> parent HTB class.  I think it was that the "rate" parameter isn't
used,
> only the "ceil" parameter (of a parent HTB class) is important. 
Check
> the list archive and the HTB home page because I'm not sure.
Nor the rate, nor the ceil are respected if the child class can send.  So=
 if B=20
end C can send the remaining bandwidth, they will.  Even if the parent ce=
il=20
is not permitting it.

>   If what I have written is true, there is a possibility that bandwidth
> is not distributed equally between classes B and C.
Indeed.  This can be true IF class B and C have different rates.  But I d=
id=20
some tests and it seems to be that remaining bandwidth is splitted 50-50 =
and=20
according to the rates.  Strange.  I will test it further tomorrow.  But =
the=20
prio of the parent is respected.  So the parent with the lowest prio get =
all=20
remaining bandwidth.

> >What if C and B have different rates?
> >
> >Is prio parameter taken into account when htb tries to meet guaranteed
> >rate rules?
>
>    I think the "prio" parameter is only used after all classes
have
> reached their guaranteed minimum rate, to allow the user to create
> classes that will borrow bandwidth over other classes.
Yep.

> >What happens when sum of guaranteed rates of children class is bigger =
than
> >guaranteed rate of parent (rate parameter is overbooked) and all of
> >classes are requesting maximum bandwidth? Are classes with lower prio
> >given bandwidth first?
>
>     There are rules that you should respect when creating classes.
>  Check the FAQ on the HTB home site.
And I have some more on the faq page on www.docum.org

> >Are packets classfied to class D and G sent first?
>
>    No, unless classes D and G haven't reached their guaranteed minimum
> rate.
>
> >What will happen if prio of class B is 0 and class C is 3? I assume
> >remaining bandwidth is first distributed to class B and to its childre=
n.
> >Right???
>
>     Same answer regarding parent HTB classes.  I'm not sure.
All remaining bandwidth goes to B.

Stef

--=20

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

Hello,

Thank you for your replies. Let me summarize your opinions....

Let's assume I have 512 kbit/s synchronous link to my upstream provider.
I have two customers named "B" and "C". I want to be sure
that they get at
last 8kbit/s. I want to limit maximum traffic generated per each customer
to 128 kbit/s.

A tree that follows describes what I did.

  A
 / \
B   C

 Class A rate and ceiling is set to something like 490kbit/sek (or lower)
to move queues to my linux traffic shaper. I set equal prios, rate for B
and C equals 8 kbit/s, ceiling for B and C equals 128 kbit/s. This setup
is supposed to fulfill my needs. It should work as expected, right?

Now I try to improve my setup. What I require is to divide customer B and
C traffic into 3 classes - interactive(D,G) , www (E,H) and other
(F,I)respectively. I dont want to allow customer B or C traffic to go over
128kbit/s.I expanded last tree and created following tree.
     A
   /   \
  B     C
 /|\   /|\
D E F G H I

D,G rate=4/ceil=64
E,H rate=3/ceil=128
F,I rate=1/ceil=32

Priorities for all classes are the same.

Lets assume all classes try to send at their maximum speed trying to
saturate the link. According to what you have written class D will get
64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The sum is
224kbit/s if I am correct. Am I right?

I dont want it to happen since customers have paid for 128kbit/s with
guaranteed rato of 8kbit/s. Is there a way to acomplish my task???. Can it
be done using HTB only?


>> >Remaining bandwidth inside class B is distributed first to class D,
>> then E and then F and is limited by ceiling parameter . Right???
>>       yes, what you have said is right.
> Confirmed.  Lowest prio classes are allowed to send first.
It is intuitive. Thanks for confirmation. Anyway, classes D,E,F are
limited only by their ceil, not by ceil of class B. Hm?. Performance
reasons, right?

>> >Class A has available bandwidth. Rules for guaranteed rates for
>> classes D,E,F,G,H,I are fulfilled. So available bandwidth has to
>> >be distributed between class B and C equaly (assuming B and C has
the
>> same rate and prio). Is remaining bandwidth distributed to classes D
>> and G, and then to classes E and H and at the end to classes F and
>> I???
> Yes.
It is also rather intuitive.

>
>>   I remember having read something about the "rate" parameter
of a
>> parent HTB class.  I think it was that the "rate" parameter
isn't
>> used, only the "ceil" parameter (of a parent HTB class) is
important.
>> Check the list archive and the HTB home page because I'm not sure.
> Nor the rate, nor the ceil are respected if the child class can send.
> So if B  end C can send the remaining bandwidth, they will.  Even if the
> parent ceil  is not permitting it.
Well... I hoped for opposite. If the ceiling of parent class is not
respected, then setting htb up the way I require is rather impossible.?
Right:( ?

>
>>   If what I have written is true, there is a possibility that
>> bandwidth
>> is not distributed equally between classes B and C.
> Indeed.  This can be true IF class B and C have different rates.  But I
> did  some tests and it seems to be that remaining bandwidth is splitted
> 50-50 and  according to the rates.  Strange.  I will test it further
> tomorrow.  But the  prio of the parent is respected.  So the parent with
> the lowest prio get all  remaining bandwidth.
I would like to see the results, if you were so kind...

>
>> >What if C and B have different rates?
>> >
>> >Is prio parameter taken into account when htb tries to meet
>> guaranteed rate rules?
>>
>>    I think the "prio" parameter is only used after all
classes have
>> reached their guaranteed minimum rate, to allow the user to create
>> classes that will borrow bandwidth over other classes.
> Yep.
OK. You have cleared thing up:)

>
>> >What happens when sum of guaranteed rates of children class is
bigger
>> than guaranteed rate of parent (rate parameter is overbooked) and all
>> of classes are requesting maximum bandwidth? Are classes with lower
>> prio given bandwidth first?
>>
>>     There are rules that you should respect when creating classes.
>>  Check the FAQ on the HTB home site.
> And I have some more on the faq page on www.docum.org
This FAQ is a nice piece of work Stef:) Well done:)

>
>> >Are packets classfied to class D and G sent first?
>>
>>    No, unless classes D and G haven't reached their guaranteed minimum
>> rate.
>>
>> >What will happen if prio of class B is 0 and class C is 3? I assume
>> remaining bandwidth is first distributed to class B and to its
>> children. Right???
>>
>>     Same answer regarding parent HTB classes.  I'm not sure.
> All remaining bandwidth goes to B.
It goes to B and then to its children, right? I mean, there are no
classifiers (filters) pointing to class B directly, only to its children.

Robert Krycza³o

On Monday 30 December 2002 23:35, ISC Robert Krycza=B3o wrote:
> Hello,
>
> Thank you for your replies. Let me summarize your opinions....
>
> Let's assume I have 512 kbit/s synchronous link to my upstream provider=
=2E
> I have two customers named "B" and "C". I want to be
sure that they get=
 at
> last 8kbit/s. I want to limit maximum traffic generated per each custom=
er
> to 128 kbit/s.
>
> A tree that follows describes what I did.
>
>   A
>  / \
> B   C
>
>  Class A rate and ceiling is set to something like 490kbit/sek (or lowe=
r)
> to move queues to my linux traffic shaper. I set equal prios, rate for =
B
> and C equals 8 kbit/s, ceiling for B and C equals 128 kbit/s. This setu=
p
> is supposed to fulfill my needs. It should work as expected, right?
No.  Why don't you take rate =3D ceil =3D 128kbit/s ??  If A uses all it =
can, 128,=20
there is still 512-128 left for the other as his mimum bandwidth.  Or do =
you=20
want so say 8kilobyte ?

> Now I try to improve my setup. What I require is to divide customer B a=
nd
> C traffic into 3 classes - interactive(D,G) , www (E,H) and other
> (F,I)respectively. I dont want to allow customer B or C traffic to go o=
ver
> 128kbit/s.I expanded last tree and created following tree.
>      A
>    /   \
>   B     C
>  /|\   /|\
> D E F G H I
>
> D,G rate=3D4/ceil=3D64
> E,H rate=3D3/ceil=3D128
> F,I rate=3D1/ceil=3D32
>
> Priorities for all classes are the same.
>
> Lets assume all classes try to send at their maximum speed trying to
> saturate the link. According to what you have written class D will get
> 64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The sum is
> 224kbit/s if I am correct. Am I right?
Yes.  So the rate of the parent B must also be at least 224kbit/s.  And n=
ot=20
8kbis/s like you wrote before.

> I dont want it to happen since customers have paid for 128kbit/s with
> guaranteed rato of 8kbit/s. Is there a way to acomplish my task???. Can=
 it
> be done using HTB only?
Yes, make the sum of D,E and F =3D 128kbit/s.

> >> >Remaining bandwidth inside class B is distributed first to
class D,
> >>
> >> then E and then F and is limited by ceiling parameter . Right???
> >>       yes, what you have said is right.
> >
> > Confirmed.  Lowest prio classes are allowed to send first.
>
> It is intuitive. Thanks for confirmation. Anyway, classes D,E,F are
> limited only by their ceil, not by ceil of class B. Hm?. Performance
> reasons, right?
I have some rules on the faq page on www.docum.org regarding the rate and=
 ceil=20
of parent and child classes.  If you follow these rules, it will be easie=
r to=20
understand how each class will behave.

> >>   I remember having read something about the "rate"
parameter of a
> >> parent HTB class.  I think it was that the "rate"
parameter isn't
> >> used, only the "ceil" parameter (of a parent HTB class)
is important=
=2E
> >> Check the list archive and the HTB home page because I'm not sure.
> >
> > Nor the rate, nor the ceil are respected if the child class can send.
> > So if B  end C can send the remaining bandwidth, they will.  Even if =
the
> > parent ceil  is not permitting it.
>
> Well... I hoped for opposite. If the ceiling of parent class is not
> respected, then setting htb up the way I require is rather impossible.?
> Right:( ?
No.  You have to see the rate as a minimum bandwidth and also like a=20
"proportion".  I mean if you have a class with child rate =3D 10 and
an o=
ther=20
child class with rate =3D 30, the first class will get 25% of the bandwid=
th. =20
So the real value of rate doesn't mather.

> >>   If what I have written is true, there is a possibility that
> >> bandwidth
> >> is not distributed equally between classes B and C.
> >
> > Indeed.  This can be true IF class B and C have different rates.  But=
 I
> > did  some tests and it seems to be that remaining bandwidth is splitt=
ed
> > 50-50 and  according to the rates.  Strange.  I will test it further
> > tomorrow.  But the  prio of the parent is respected.  So the parent w=
ith
> > the lowest prio get all  remaining bandwidth.
>
> I would like to see the results, if you were so kind...
I didn't had the time to write it down, but it seems to me that each pare=
nt=20
class can use it's rate like it should be, but the remaining bandwidth is=
=20
splitted 50-50 and according to the quantums.

> >> >What if C and B have different rates?
> >> >Is prio parameter taken into account when htb tries to meet
> >> guaranteed rate rules?
> >>
> >>    I think the "prio" parameter is only used after all
classes have
> >> reached their guaranteed minimum rate, to allow the user to create
> >> classes that will borrow bandwidth over other classes.
> >
> > Yep.
>
> OK. You have cleared thing up:)
I think there is a page on the htb homepage that state that only prio of =
leaf=20
classes are used.  In fact if you have a htb setup, and you asks the setu=
p,=20
the prio of non-leaf classes isn't even shown.  On the other hand, I did =
some=20
small tests and the prio parameter seems to be important.  I really have =
to=20
do some tests .....

> This FAQ is a nice piece of work Stef:) Well done:)
It's a nice of work that needs a rewrite :)
Its more a bunch of questions and answers :)

> It goes to B and then to its children, right? I mean, there are no
> classifiers (filters) pointing to class B directly, only to its childre=
n.
The filters doesn't mather.  As long as the packets end in a leaf class.

Stef

--=20

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

--2)/.U16_4e'yM=.9
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Tue, 31 Dec 2002 00:37:44 +0100 Stef Coene <stef.coene@docum.org>
wrote....

> > Lets assume all classes try to send at their maximum speed trying to
> > saturate the link. According to what you have written class D will get
> > 64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The sum is
> > 224kbit/s if I am correct. Am I right?
> Yes.  So the rate of the parent B must also be at least 224kbit/s.  And
> not 8kbis/s like you wrote before.
>
> > I dont want it to happen since customers have paid for 128kbit/s with
> > guaranteed rato of 8kbit/s. Is there a way to acomplish my task???.
> > Can it be done using HTB only?
> Yes, make the sum of D,E and F = 128kbit/s.

	This is the same kind of setup I've been shooting for. When you say the
sum of D, E and F, are you talking rate or ceil? I would imagine ceil.
That makes it hard to please the customer who is paying for 128kbit, yet
only sees it in 3 42kbit streams, or some other breakdown of the 128k. If
I understand you right, there is no way to run three seperate children for
prioritizing packets, and still give full bandwidth of parent to any of
the three, without allowing the use of all three at the same time to
exceed the parent?

---
Homer Parker                  /"\ ASCII Ribbon Campaign
                              \ / No HTML/RTF in email
http://www.homershut.net       x  No Word docs in email
telnet://bbs.homershut.net    / \ Respect for open standards

This e-mail message is 100% Microsoft free!

WARNING: THIS ACCOUNT BELONGS TO A RABID
ANTI-SPAMMER NET-NAZI DOT-COMMUNIST!!


--2)/.U16_4e'yM=.9
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE+FGgvrgrN227HZ+8RApJvAJ4m2g9Ny1ZGnQUkhiilPUeaCAnR6QCgqkUQ
AEvz7fgzCLaDIcX9fsTUNww=
=q0Br
-----END PGP SIGNATURE-----

--2)/.U16_4e'yM=.9--

Hi there,
>> > Lets assume all classes try to send at their maximum speed trying
to
>> saturate the link. According to what you have written class D will
>> get 64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The
>> sum is 224kbit/s if I am correct. Am I right?
>> Yes.  So the rate of the parent B must also be at least 224kbit/s.
>> And not 8kbis/s like you wrote before.
>>
>> > I dont want it to happen since customers have paid for 128kbit/s
>> with guaranteed rato of 8kbit/s. Is there a way to acomplish my
>> task???. Can it be done using HTB only?
>> Yes, make the sum of D,E and F = 128kbit/s.
>
> 	This is the same kind of setup I've been shooting for. When you say the
> sum of D, E and F, are you talking rate or ceil? I would imagine ceil.
> That makes it hard to please the customer who is paying for 128kbit, yet
> only sees it in 3 42kbit streams, or some other breakdown of the 128k.
> If I understand you right, there is no way to run three seperate
> children for prioritizing packets, and still give full bandwidth of
> parent to any of the three, without allowing the use of all three at the
> same time to exceed the parent?
Yes, I am talking about ceil. And the problem is that (and you as well) I
want 128kbit/s go to class E in case D and F are idle. There must be a way
to set this up, but for now I don't know how. If neither ceiling nor rate
of parent class is respected, what are parents in htb for. Flat structure
would be enough. I would like to receive more feedback on this.

The real reason for this is to limit p2p traffic, still allowing customer
to surf the web, send email, play counter strike at full speed.

Robert Krycza³o

On Thursday 02 January 2003 19:03, ISC Robert Krycza=B3o wrote:
> Hi there,
>
> >> > Lets assume all classes try to send at their maximum speed
trying =
to
> >>
> >> saturate the link. According to what you have written class D will
> >> get 64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The
> >> sum is 224kbit/s if I am correct. Am I right?
> >> Yes.  So the rate of the parent B must also be at least 224kbit/s.
> >> And not 8kbis/s like you wrote before.
> >>
> >> > I dont want it to happen since customers have paid for
128kbit/s
> >>
> >> with guaranteed rato of 8kbit/s. Is there a way to acomplish my
> >> task???. Can it be done using HTB only?
> >> Yes, make the sum of D,E and F =3D 128kbit/s.
> >
> > =09This is the same kind of setup I've been shooting for. When you sa=
y the
> > sum of D, E and F, are you talking rate or ceil? I would imagine ceil=
=2E
> > That makes it hard to please the customer who is paying for 128kbit, =
yet
> > only sees it in 3 42kbit streams, or some other breakdown of the 128k=
=2E
> > If I understand you right, there is no way to run three seperate
> > children for prioritizing packets, and still give full bandwidth of
> > parent to any of the three, without allowing the use of all three at =
the
> > same time to exceed the parent?
>
> Yes, I am talking about ceil.
I was talking about the rate.  The ceil must be 128kbit/s so they can bor=
row=20
up to 128kbit/s. =20
> And the problem is that (and you as well) I
> want 128kbit/s go to class E in case D and F are idle. There must be a =
way
> to set this up, but for now I don't know how. If neither ceiling nor ra=
te
> of parent class is respected, what are parents in htb for. Flat structu=
re
> would be enough. I would like to receive more feedback on this.
That's no problem IF you follow some basic rules.  You can find them on=20
www.docum.org on the faq page.  You have to see the ceil as a maximum of =
the=20
class.  And the rate as a minimum bandwidth and a "divider". =20

The other problem is the parent rate and ceil.  I did some test and I hav=
e=20
strange results.  I still have to find a way to find out how you can=20
configure the parent rate and ceil to get a good setup.

Stef

--=20

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

Hello Stef,
> The other problem is the parent rate and ceil.  I did some test and I
> have  strange results.  I still have to find a way to find out how you
> can  configure the parent rate and ceil to get a good setup.
During my search I did some quick tests based on what you and others had
written. Those tests revealed that I can setup things the way I want.

I used following commands to set everything up.

tc qdisc add dev lo root handle 1:0 htb default 4
tc class add dev lo classid 1:1 parent 1:0 htb rate 512kbit ceil 512kbit
tc class add dev lo classid 1:2 parent 1:1 htb rate 8kbit ceil 128kbit
tc class add dev lo classid 1:3 parent 1:1 htb rate 8kbit ceil 128kbit
#class used for unclassified traffic
tc class add dev lo classid 1:4 parent 1:1 htb rate 8kbit ceil 128kbit
#children of class B
#interactive traffic
tc class add dev lo classid 1:21 parent 1:2 htb rate 4kbit ceil 64kbit
#web traffic
tc class add dev lo classid 1:22 parent 1:2 htb rate 3kbit ceil 128kbit
#other traffic
tc class add dev lo classid 1:23 parent 1:2 htb rate 1kbit ceil 32kbit
#children of class C
tc class add dev lo classid 1:31 parent 1:3 htb rate 4kbit ceil 64kbit
tc class add dev lo classid 1:32 parent 1:3 htb rate 3kbit ceil 128kbit
tc class add dev lo classid 1:33 parent 1:3 htb rate 1kbit ceil 32kbit

tc filter add dev lo parent 1:0 prio 10 protocol ip u32 divisor 256
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5021 0xffff
flowid 1:21
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5022 0xffff
flowid 1:22
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5023 0xffff
flowid 1:23
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5031 0xffff
flowid 1:31
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5032 0xffff
flowid 1:32
tc filter add dev lo parent 1:0 prio 10 u32 match ip dport 5033 0xffff
flowid 1:33

MTU of loopback device is modified and set to 1500 bytes.

Measurements results are (a little modified) output from command
"showtraf -i lo -n". Traffic was generated by command "ttcp -s -r
127.0.0.1 -p portno -v -f k" and "ttcp -s -t 127.0.0.1 -p portno -v -f
k"
where portno is 5021,5022,5023,5031,5032,5033.

Here goes the results.

>From Address     To Address       Proto Bytes   CPS
======================================================================
127.0.0.1..3143  127.0.0.1..5022  tcp   9864112 6300
127.0.0.1..3142  127.0.0.1..5021  tcp   9307500 6300
127.0.0.1..3145  127.0.0.1..5031  tcp   8332612 6300
127.0.0.1..3147  127.0.0.1..5032  tcp   7641112 6000
127.0.0.1..3144  127.0.0.1..5023  tcp   6237112 3900
127.0.0.1..3148  127.0.0.1..5033  tcp   5326612 3900

It is clear that ceiling of parent class (2 and 3) is respected. The sum of
22+23+24 and 31+32+33 is around 128kbit/s. Bandwidth inside class 2 and 3 is
divided according (proportional) to rates and never exceeds leaf class ceil.

During the tests I discovered that in case of root class (1:0 in my example)
only rate matters not ceil. I accidentally changed 1:2 and 1:3 to root
classes and then 22,23,24 were limited to 8kbit/s.

Then I simulated congestion on main link by issuing
tc class change dev lo classid 1:1 parent 1:0 htb rate 192kbit ceil 192kbit

>From Address     To Address       Proto Bytes    CPS
======================================================================
127.0.0.1..3143  127.0.0.1..5022  tcp   14361112 4200
127.0.0.1..3142  127.0.0.1..5021  tcp   13909500 3900
127.0.0.1..3145  127.0.0.1..5031  tcp   12940612 4200
127.0.0.1..3147  127.0.0.1..5032  tcp   12130612 3900
127.0.0.1..3144  127.0.0.1..5023  tcp    9298612 3600
127.0.0.1..3148  127.0.0.1..5033  tcp    8386612 3600

And again, everything worked as expected. Bandwidth inside 1:1 was divided
equaly between 1:2 and 1:3.

It was time to change rate of 1:2 to 64kbit (change to 16kbit/s hadn't
introduced any significant effects).
tc class change dev lo classid 1:2 parent 1:1 htb rate 64kbit ceil 128kbit

Results: (i will refer to them further as Exapmle A results)

>From Address     To Address       Proto Bytes    CPS
======================================================================
127.0.0.1..3203  127.0.0.1..5021  tcp   6640612 5400
127.0.0.1..3204  127.0.0.1..5022  tcp   5955112 5100
127.0.0.1..3205  127.0.0.1..5023  tcp   3372612 4200
127.0.0.1..3206  127.0.0.1..5031  tcp   3309112 3000
127.0.0.1..3207  127.0.0.1..5032  tcp   2487112 3000
127.0.0.1..3209  127.0.0.1..5033  tcp   1716112 2700

Hm, strange - it seems bandwidth is divided about 2:1. tc class show dev lo
gives us:

class htb 1:3 parent 1:1 prio 0 rate 8Kbit ceil 128Kbit burst 1609b/8 mpu 0b
cburst 1762b/8 mpu 0b quantum 102 level 2
 Sent 61860798 bytes 40875 pkts (dropped 0, overlimits 0)
 rate 8786bps 5pps
 lended: 849 borrowed: 37041 giants: 0 injects: 0
 tokens: -4918741 ctokens: -25831

class htb 1:2 parent 1:1 prio 0 rate 64Kbit ceil 128Kbit burst 1680b/8 mpu
0b cburst 1762b/8 mpu 0b quantum 819 level 2
 Sent 74831096 bytes 49440 pkts (dropped 0, overlimits 0)
 rate 14859bps 9pps
 lended: 11565 borrowed: 34638 giants: 0 injects: 0
 tokens: -23468 ctokens: -15768

Only things that differ are quantum and rate, but they differ in propotion
1:8. After changing class 1:0 rate and bandwidth to 256kbits the difference
is smaller.

I doubled rate of 1:2 and do the same to 1:21,22,23.

Results: (i will refer to them further as Exapmle B results)

>From Address     To Address       Proto Bytes    CPS
======================================================================
127.0.0.1..3216  127.0.0.1..5021  tcp            4782112 4500
127.0.0.1..3217  127.0.0.1..5022  tcp            4504612 3900
127.0.0.1..3219  127.0.0.1..5031  tcp            4143112 3900
127.0.0.1..3220  127.0.0.1..5032  tcp            4078612 3600
127.0.0.1..3218  127.0.0.1..5023  tcp            3412612 3600
127.0.0.1..3221  127.0.0.1..5033  tcp            3304612 3900

Well..... I hoped 1:2 will be given more bandwidth:) ....

In my opinion when prios are equal every leaf at the same level is given
bandwith equal to rate, then bandwidth equal to the rate of parent class is
divided between all children equally. If some children request less (due to
ceil limitations or real needs), available bandwidth is distributed equally
between other children. And then available bandwidth of grandparent is
divided between children equally (ceil of parent is respected).

Let's analyze Example A to make things clear:
192kbit/s = 24000 CPS .... showtraf always show sum of 23400 (it is possible
that 192kbit/s is layer 2 value, showtraf takes its measurements at level
4).

Step 1.
1:21 is given 500 CPS
1:22 is given 400 CPS
1:23 is given 100 CPS
1:31 is given 500 CPS
1:32 is given 400 CPS
1:33 is given 100 CPS

Step 1 SUM=2000CPS

Step2.
1:2 rate of 8000 CPS is divided between 21,22,23 equally.
1:21 is given ca 2650 CPS
1:22 is given ca 2650 CPS
1:23 is given ca 2650 CPS

1:3 rate of 1000 CPS is divided between 31,32,33 equally.
1:31 is given ca 350 CPS
1:32 is given ca 350 CPS
1:33 is given ca 350 CPS

Step 2 SUM==9000 CPS

None of leaf classes is currently limited by ceil or parent ceil.

Step 3.
There is 13000 CPS to share between six classes

Each class gets 13000/6=2150, but then class 1:23 would be transmitting over
ceil so class 1:23 gets ca 1450 and is sending packets at speed 4200 CPS
(this value oscilates around 4000 CPS).
13000-1450=11550
11850 is divided into five parts (five classes), each class gets 2300

Final results are:
1:21 should be transmitting at rate 500+2650+2300=5450 (real time
measurement oscilates around this value, Example A shows value of 5400)
1:22 should be transmitting at rate 400+2650+2300=5350 (show traf shows
5100)
1:23 sould be transmitting at 4000 (limited by ceil, it is sending packets
at speed of 4200)

1:31 500+350+2300=3150 (showtraf shows 3000)
1:32 400+250+2300=3050 (showtraf shows 3000)
1:33 100+350+2300=2750 (showtraf shows 2700)

As for Example B - calculations allocate bandwidth in following fashion:
1:21 - 4650 CPS
1:22 - 4450 CPS
1:23 - 3850 CPS
1:31 - 3850 CPS
1:32 - 3750 CPS
1:33 - 3450 CPS
which is close to reality (values from show traf oscilate around values
about 3-5% lower).

I think if I use prios child classes with lower prio value will be offered
bandwidth first. I will do some future research in this area. I am thinking
about analyzing a little bigger, real life example with 4 levels of classes,
different prios etc..
I hope somebody except me benefits from this post.

Robert Krycza³o

On Saturday 04 January 2003 21:21, ISC Robert Krycza=B3o wrote:
> Hello Stef,
Hi,

I have 1 big remark.  You results are incorrect and not usable.  The quan=
tum=20
of your class is too low.  You have a rate of 1kbit, so quantum is 1kbit =
/ 10=20
=3D 100 bit.  In your case, quantum is important.  Each class may send qu=
antum=20
bytes to use all remaining bandwidth from the parent.  Let's say class 21=
 and=20
22 are fighting for bandwidth.  The qauntums are 400 and 300.  So class 2=
1=20
may send 400bit and after that class 22 may send 300bit.  But most packet=
s=20
are 1500byte !!!   So 1500 byte is sended.  So the internal calulations o=
f=20
htb are totally fucked up (sorry for the language :).

If you want to get some good results, use at minimum a rate of 15kbit or=20
change the r2q parameter.  If you are curious about the qauntum parameter=
, I=20
have some more info on www.docum.org on the faq page.

I redid some of your test with higher rates.  I multiplied the rates with=
 16=20
(so kbit -> kpbs and x2).  And I added r2q =3D 1 so lowest quantum is=20
2000bytes.

And I toke a wrong conclusion.  The parent ceil is not respected IF the s=
um of=20
the rates of childs exceed the parent ceil.  So if you have 2 childs clas=
ses=20
with rate =3D 100 and a parent ceil of 100, the parent ceil is not respec=
ted=20
and the childs will get each 100.  But if you have 2 class with rate =3D =
2 and=20
parent ceil =3D 100, then the childs to gether will never get more then 1=
00. =20
So the parent ceil is respected.

Traffic in class 21 : 127.5 KB/s
Traffic in class 22 : 256.1 KB/s
Traffic in class 21 and 22 : 120.4 KB/s and 137.7 KB/s
=09so the parent ceil is respected.  But I still don't know how the traff=
ic is=20
divided.  The sum is 258.1 so class 21 gets 46.6%.  I think each class ge=
ts=20
the configured rate and the remaining traffic is splitted 50-50.

> During the tests I discovered that in case of root class (1:0 in my
> example) only rate matters not ceil. I accidentally changed 1:2 and 1:3=
 to
> root classes and then 22,23,24 were limited to 8kbit/s.
I did the same.  And indeed.  For a root class, ceil =3D rate even if you=
=20
specify a higher ceil.  Strange.  On the other hand, it's logic to create=
 1=20
root class that holds all traffic so it has rate =3D ceil.  It's the=20
"bottleneck" within the htb structure.

Stef

--=20

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

>Hi,
Hello,
>
> I have 1 big remark.  You results are incorrect and not usable.  The
> quantum  of your class is too low.  You have a rate of 1kbit, so quantum
> is 1kbit / 10  = 100 bit.  In your case, quantum is important.  Each
> class may send quantum  bytes to use all remaining bandwidth from the
> parent.  Let's say class 21 and  22 are fighting for bandwidth.  The
> qauntums are 400 and 300.  So class 21  may send 400bit and after that
> class 22 may send 300bit.  But most packets  are 1500byte !!!   So 1500
> byte is sended.  So the internal calulations of  htb are totally fucked
> up (sorry for the language :).
They are broken, indeed:).
>
> If you want to get some good results, use at minimum a rate of 15kbit or
>  change the r2q parameter.  If you are curious about the qauntum
> parameter, I  have some more info on www.docum.org on the faq page.
That's correct. I set up r2q to 1 and for rates of 12kbit/s quantum is
bigger than 1500 bytes. Do you have any idea what happens if I set up very
small rates like 1kbit/s or 0.01kbit/s=1bps :) and quantum=1500? Is htb
precise in such situations. Is it equal to setting rate to 12kbit and
quantum to 1500.Unfortunatelly, I am unable to set quantums in my version
of htb and do some empirical tests. (It some early version of htb3 - patch
against kernel 2.4.18 - it is extremely stable for me so I am not going to
change it.). I want to create guarantees at certain level of a htb tree,
so I am interested in rates close to zero:). Of course if I have many
customers, it is not so important... I'm just curious:)

>
> I redid some of your test with higher rates.  I multiplied the rates
> with 16  (so kbit -> kpbs and x2).  And I added r2q = 1 so lowest
> quantum is  2000bytes.
>

> And I toke a wrong conclusion.  The parent ceil is not respected IF the
> sum of  the rates of childs exceed the parent ceil.  So if you have 2
> childs classes  with rate = 100 and a parent ceil of 100, the parent
> ceil is not respected  and the childs will get each 100.  But if you
> have 2 class with rate = 2 and  parent ceil = 100, then the childs to
> gether will never get more then 100.   So the parent ceil is respected.
Your statement is absolutelly correct.
>
> Traffic in class 21 : 127.5 KB/s
> Traffic in class 22 : 256.1 KB/s
> Traffic in class 21 and 22 : 120.4 KB/s and 137.7 KB/s
> 	so the parent ceil is respected.  But I still don't know how the
> traffic is
> divided.  The sum is 258.1 so class 21 gets 46.6%.  I think each class
> gets  the configured rate and the remaining traffic is splitted 50-50.
My others tests shows that each leaf class gets the configured rate (even
if rates are overbooked - then parent ceil is not respected). Then rate of
parent is divided beetween children according to quantums - so it is not
equally distributed. And then children borrow free bandwidth from parents
who borrow from grandparents etc... borrowing is proportional to quantums.
The difference is visible only if rates difference is significant. In case
rates are the same every children gets the same share.

I found this in hbt manual: "You should see that if several classes are
competing for parent's bandwidth then they get it in proportion of their
quantums. It is important to know that for precise operation quantums need
to be as small as possible and larger than MTU. ".

Now, thanks to your advices and www.docum.org's docs I am able to predict
htb behaviour. Thanx Stef:)

>
>> During the tests I discovered that in case of root class (1:0 in my
>> example) only rate matters not ceil. I accidentally changed 1:2 and
>> 1:3 to root classes and then 22,23,24 were limited to 8kbit/s.
> I did the same.  And indeed.  For a root class, ceil = rate even if you
> specify a higher ceil.  Strange.  On the other hand, it's logic to
> create 1  root class that holds all traffic so it has rate = ceil.  It's
> the  "bottleneck" within the htb structure.
That's correct. Multiple root classes inside htb qdisc can be used to
simulate "circuits". Those circuits cannot borrow from each other. It
can
be useful sometimes.

I just have to check if root class can go over rate in case sum of its
children rates exceeds root class rate.

BTW. Those scripts from www.docum.org are very nice.

Robert Krycza³o

> > I have 1 big remark.  You results are incorrect and not usable.  The
> > quantum  of your class is too low.  You have a rate of 1kbit, so
quantum
> > is 1kbit / 10  = 100 bit.  In your case, quantum is important.  Each
> > class may send quantum  bytes to use all remaining bandwidth from the
> > parent.  Let's say class 21 and  22 are fighting for bandwidth.  The
> > qauntums are 400 and 300.  So class 21  may send 400bit and after that
> > class 22 may send 300bit.  But most packets  are 1500byte !!!   So
1500
> > byte is sended.  So the internal calulations of  htb are totally
fucked
> > up (sorry for the language :).
> They are broken, indeed:).
>
> > If you want to get some good results, use at minimum a rate of 15kbit
or
> >  change the r2q parameter.  If you are curious about the qauntum
> > parameter, I  have some more info on www.docum.org on the faq page.
>
> That's correct. I set up r2q to 1 and for rates of 12kbit/s quantum is
> bigger than 1500 bytes. Do you have any idea what happens if I set up very
> small rates like 1kbit/s or 0.01kbit/s=1bps :) and quantum=1500? Is htb
> precise in such situations. Is it equal to setting rate to 12kbit and
> quantum to 1500.Unfortunatelly, I am unable to set quantums in my version
> of htb and do some empirical tests. (It some early version of htb3 - patch
> against kernel 2.4.18 - it is extremely stable for me so I am not going to
> change it.). I want to create guarantees at certain level of a htb tree,
> so I am interested in rates close to zero:). Of course if I have many
> customers, it is not so important... I'm just curious:)
Quantum is used when 2 or more classes are fighting for remaining bandwidth
from the same parent.  Each class may send quantum bytes at a turn.  So if
you have 1 class with quantum = 100 and an other with quantum = 900, you
should expect that the first class gets 10% and the second 90%.  But if the
packet size is 1000 byte, each class can send the same amount of traffic.
Basically, nothing bad will hapenn, but it will less obviously to find out
what's going on.
And I think it's save to upgrade to the latest htb.

> My others tests shows that each leaf class gets the configured rate (even
> if rates are overbooked - then parent ceil is not respected). Then rate of
> parent is divided beetween children according to quantums - so it is not
> equally distributed. And then children borrow free bandwidth from parents
> who borrow from grandparents etc... borrowing is proportional to quantums.
> The difference is visible only if rates difference is significant. In case
> rates are the same every children gets the same share.
>
> I found this in hbt manual: "You should see that if several classes
are
> competing for parent's bandwidth then they get it in proportion of their
> quantums. It is important to know that for precise operation quantums need
> to be as small as possible and larger than MTU. ".
But is this also for classes with childs?  I'm not sure.  I mailed Devik about
it, and I will forward you his answer.

> Now, thanks to your advices and www.docum.org's docs I am able to predict
> htb behaviour. Thanx Stef:)
No problem.

> >> During the tests I discovered that in case of root class (1:0 in
my
> >> example) only rate matters not ceil. I accidentally changed 1:2
and
> >> 1:3 to root classes and then 22,23,24 were limited to 8kbit/s.
> >
> > I did the same.  And indeed.  For a root class, ceil = rate even if
you
> > specify a higher ceil.  Strange.  On the other hand, it's logic to
> > create 1  root class that holds all traffic so it has rate = ceil. 
It's
> > the  "bottleneck" within the htb structure.
>
> That's correct. Multiple root classes inside htb qdisc can be used to
> simulate "circuits". Those circuits cannot borrow from each
other. It can
> be useful sometimes.
>
> I just have to check if root class can go over rate in case sum of its
> children rates exceeds root class rate.
I tested it, they can.

> BTW. Those scripts from www.docum.org are very nice.
Thx.

Stef

--

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

Hello,
>>   A
>>  / \
>> B   C
>>
>>  Class A rate and ceiling is set to something like 490kbit/sek (or
>> lower)
>> to move queues to my linux traffic shaper. I set equal prios, rate for
>> B and C equals 8 kbit/s, ceiling for B and C equals 128 kbit/s. This
>> setup is supposed to fulfill my needs. It should work as expected,
>> right?
> No.  Why don't you take rate = ceil = 128kbit/s ??  If A uses all it
> can, 128,  there is still 512-128 left for the other as his mimum
> bandwidth.  Or do you  want so say 8kilobyte ?
No, 8 kilobits or 1kilobyte per second. The reason for this is that I
want to have around 40-50 customer classes inside A. With guaranteed rate
of 8kbit/s and ceiling of 128kbit/sek. The statistics data I have gathered
leads me to a conclusion that one of every three users utilizes 33% of his
bandwidth (33% of 128kbit/s), two others are idle. In pratice we
can have 80-100 customers per 1Mbit and it works well if no p2p software
(like Kazaa is involved). This is why I want to switch to htb and its
ceiling feature.

Customers pay relatively low rates and thanks to current cbq  setup
receive relatively good service. Anyway, we have customers using
128/128(rate/ceiling) which pay 8 times more.

>>      A
>>    /   \
>>   B     C
>>  /|\   /|\
>> D E F G H I
>>
>> D,G rate=4/ceil=64
>> E,H rate=3/ceil=128
>> F,I rate=1/ceil=32
>>
>> Priorities for all classes are the same.
>>
>> Lets assume all classes try to send at their maximum speed trying to
>> saturate the link. According to what you have written class D will get
>> 64kbit/s, class E 128kbit/s and class F will get 32kbit/s. The sum is
>> 224kbit/s if I am correct. Am I right?
> Yes.  So the rate of the parent B must also be at least 224kbit/s.  And
> not  8kbis/s like you wrote before.
I understand but... I still want to limit customer B to 128 kbit/s. And
guarantee them 8kbit/s.

>
>> I dont want it to happen since customers have paid for 128kbit/s with
>> guaranteed rato of 8kbit/s. Is there a way to acomplish my task???.
>> Can it be done using HTB only?
> Yes, make the sum of D,E and F = 128kbit/s.
Let's say I give D bandwidth of 32 kbit/s, E bandwidth od 64 kbit/s and F
bandwidth 32 kbit/s. When every classes are idle ond only E attempte to
transmit at maximum speed, it will be given 64kbit/sek not 128kbit/s:(.
Customers will not be happy, because in current setup they can reach
128kbit/s (and thanks to cbq inaccuracies even more - around 155kbit/s).

>
>> >> >Remaining bandwidth inside class B is distributed first to
class
>> D,
>> >>
>> >> then E and then F and is limited by ceiling parameter .
Right???
>> >>       yes, what you have said is right.
>> >
>> > Confirmed.  Lowest prio classes are allowed to send first.
>>
>> It is intuitive. Thanks for confirmation. Anyway, classes D,E,F are
>> limited only by their ceil, not by ceil of class B. Hm?. Performance
>> reasons, right?
> I have some rules on the faq page on www.docum.org regarding the rate
> and ceil  of parent and child classes.  If you follow these rules, it
> will be easier to  understand how each class will behave.
I have read it. In simple cases I totaly agree. But there must be
something missing (or I am totally wrong:) ). Or the setup I require is
not possible to create.

>> > Nor the rate, nor the ceil are respected if the child class can
>> send. So if B  end C can send the remaining bandwidth, they will.
>> Even if the parent ceil  is not permitting it.
>>
>> Well... I hoped for opposite. If the ceiling of parent class is not
>> respected, then setting htb up the way I require is rather
>> impossible.? Right:( ?
> No.  You have to see the rate as a minimum bandwidth and also like a
> "proportion".  I mean if you have a class with child rate = 10
and an
> other  child class with rate = 30, the first class will get 25% of the
> bandwidth.   So the real value of rate doesn't mather.

I think exactly the way you described. I see rate the same way you do.
Unfotunetaly this leads me to a conclusion that htb has serious drawbacks.
If the ceiling of the parent is not respected then it is impossible to
create setup I require. In this case it is also not necesary to create
parent classes.

I hardly belive that. I must omit something important. There follows
interesting quote from
http://luxik.cdi.cz/~devik/qos/htb/old/develnotes.htm

"Lower levels are completely dequeueued before borrowing from higher
levels.". So is it possible that prio and rate and ceiling of parent class
matters?

And teator of htb qdisc often uses term of borrowing bandwidth from
parent......

>> >> >What if C and B have different rates?
>> >> >Is prio parameter taken into account when htb tries to
meet
>> >> guaranteed rate rules?
>> >>
>> >>    I think the "prio" parameter is only used after
all classes have
>> >> reached their guaranteed minimum rate, to allow the user to
create
>> classes that will borrow bandwidth over other classes.
>> >
>> > Yep.
>>
>> OK. You have cleared thing up:)
> I think there is a page on the htb homepage that state that only prio of
> leaf  classes are used.  In fact if you have a htb setup, and you asks
> the setup,  the prio of non-leaf classes isn't even shown.  On the other
> hand, I did some  small tests and the prio parameter seems to be
> important.  I really have to  do some tests .....

Yes, I will try to do some research on my own, too. Anyway, I would like
to read your opinions Stef about my doubts. I think this thread can be
useful to many people.

Robert Krycza³o