LARTC - Jun 2005 - htb child classes don''t borrow form "parent''s parent class" :(

Dear list,

I have the problem that child classes don''t seem to issue the parent
class to borrow from their parent.

This is a simplified version of my script:
tc qdisc del dev $DEV root
tc qdisc add dev $DEV root handle 1: htb default 10
tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 1kbit ceil
512kbit


#groups

tc class add dev $DEV parent 1: classid 1:f1 htb rate 2048kbit ceil
10mbit burst 50k
tc class add dev $DEV parent 1:f1 classid 1:a1 htb rate 2048kbit ceil
10mbit prio 2 burst 50k
tc class add dev $DEV parent 1:f1 classid 1:b1 htb rate 2048kbit ceil
6mbit prio 3 burst 50k
tc class add dev $DEV parent 1:f1 classid 1:c1 htb rate 2048kbit ceil
5mbit prio 4 burst 50k

tc class add dev $DEV parent 1: classid 1:f2 htb rate 1024kbit ceil
10mbit burst 50k
tc class add dev $DEV parent 1:f2 classid 1:a2 htb rate 1024kbit ceil
10mbit prio 2 burst 50k
tc class add dev $DEV parent 1:f2 classid 1:b2 htb rate 1024kbit ceil
6mbit prio 3 burst 50k
tc class add dev $DEV parent 1:f2 classid 1:c2 htb rate 1024kbit ceil
5mbit prio 4 burst 50k

#users

tc class add dev $DEV parent 1:a1 classid 1:3001 htb rate 64kbit ceil
512kbit burst 50k
tc qdisc add dev $DEV parent 1:3001 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst
172.30.10.1 flowid 1:3001

tc class add dev $DEV parent 1:b1 classid 1:3002 htb rate 64kbit ceil
1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3002 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst
172.30.10.2 flowid 1:3002

tc class add dev $DEV parent 1:c1 classid 1:3003 htb rate 64kbit ceil
1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3003 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst
172.30.10.3 flowid 1:3003


tc class add dev $DEV parent 1:c2 classid 1:3004 htb rate 64kbit ceil
1024kbit burst 50k
tc qdisc add dev $DEV parent 1:3004 sfq perturb 10
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst
172.30.10.4 flowid 1:3004


.
.
.

The script is dynamically generated, periodically called, and should put
the users (distinguished through their IP address) according to their
traffic usages into lower priorities (axx,bxx,cxx) respecting different
bandwidth needs but also respecting minimum rates for different groups
(fxx).

Now the borrowing among user classes (1:3xxx) is working; but for some
reason and I just can''t figure out why, the group classes (fxx)
don''t
borrow from each other when their is unused bandwidth available. The
"ceil 10mbit" just seems to be ignored :(

e.g.:

class htb 1:3003 parent 1:c1 leaf 92f4: prio 0 quantum 1000 rate
64000bit ceil 1024Kbit burst 50Kb/8 mpu 0b overhead 0b cburst 1727b/8
mpu 0b overhead 0b level 0
 Sent 22500226 bytes 28784 pkts (dropped 0, overlimits 0)
 rate 245080bit 39pps
 lended: 15306 borrowed: 13478 giants: 0  <------------ user borrowed
from others
 tokens: -429574 ctokens: 13314

but:

class htb 1:f1 root rate 2048Kbit ceil 10000Kbit burst 50Kb/8 mpu 0b
overhead 0b cburst 2848b/8 mpu 0b overhead 0b level 7
 Sent 221521122 bytes 257990 pkts (dropped 0, overlimits 0)
 rate 1818Kbit 260pps
 lended: 0 borrowed: 0 giants: 0 <------------ nothing going on here
 tokens: -1118817 ctokens: -274

class htb 1:c1 parent 1:f1 rate 2048Kbit ceil 6000Kbit burst 50Kb/8 mpu
0b overhead 0b cburst 2349b/8 mpu 0b overhead 0b level 6
 Sent 22500226 bytes 28784 pkts (dropped 0, overlimits 0)
 rate 230776bit 37pps
 lended: 13478 borrowed: 0 giants: 0 <------------ or here
 tokens: 204546 ctokens: 3121


Any help is highly appreciated!

Thanx,
 Bernd Froemel

On Saturday 18 June 2005 10:16, Bernd Froemel wrote:
> tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit
> tc class add dev $DEV parent 1: classid 1:f1 htb rate 2048kbit ceil
> 10mbit burst 50k
> tc class add dev $DEV parent 1: classid 1:f2 htb rate 1024kbit ceil
> 10mbit burst 50k
That''s most likely because you''re using different rate/ceil
for root
classes, which does not make sense at all. A root class does not have 
parents or siblings to borrow from. It can always use it''s full rate.
> The "ceil 10mbit" just seems to be ignored :(
It might very well be the case that root classes just ignore the ceil 
parameter since they can''t borrow anyway.

You should rebuild your tree so that you only have one root class.

HTH
Andreas

Thanks for your input.
> 
> > The "ceil 10mbit" just seems to be ignored :(
> 
> It might very well be the case that root classes just ignore the ceil 
> parameter since they can''t borrow anyway.
> 
> You should rebuild your tree so that you only have one root class.
Could you give me a short example what you mean?

The only other way I can think of would be a tree (respecting the described
group and user requirements) with qdiscs on every node down:

#just to be sure that I have the beginning right:
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate 10mbit #total bandwidth of
link

#the two main nodes - approved and non approved traffic:
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 8mbit ceil 10mbit prio 1
tc class add dev $DEV parent 1:1 classid 1:40 htb rate 2mbit ceil 2mbit prio 2

#attach qdiscs:
tc qdisc add dev $DEV parent 1:10 handle 10: htb
tc qdisc add dev $DEV parent 1:20 sfq perturb 10

#groups

tc class add dev $DEV parent 10: classid 10:f1 htb rate 2048kbit ceil 10mbit
tc qdisc add dev $DEV parent 10:f1 handle f1: htb
#bandwidth priorities of group f1:
tc class add dev $DEV parent f1: classid f1:a1 htb rate 2048kbit ceil 10mbit
prio 1
tc qdisc add dev $DEV parent f1:a1 handle a1: htb
tc class add dev $DEV parent f1: classid f1:b1 htb rate 2048kbit ceil 10mbit
prio 2
tc qdisc add dev $DEV parent f1:b1 handle b1: htb
tc class add dev $DEV parent f1: classid f1:c1 htb rate 2048kbit ceil 10mbit
prio 3
tc qdisc add dev $DEV parent f1:c1 handle c1: htb
#further groups omitted

#users
tc class add dev $DEV parent a1: classid a1:1 htb rate 64kbit ceil 1024kbit
tc qdisc add dev $DEV parent a1:1 sfq perturb 10
tc filter add dev $DEV parent a1:0 protocol ip prio 16 u32 match ip dst
172.30.10.1 flowid a1:1
tc filter add dev $DEV parent f1:0 protocol ip prio 16 u32 match ip dst
172.30.10.1 flowid f1:a1
tc filter add dev $DEV parent 10:0 protocol ip prio 16 u32 match ip dst
172.30.10.1 flowid 10:f1
tc filter add dev $DEV parent 1:0 protocol ip prio 16 u32 match ip dst
172.30.10.1 flowid 1:10


but this hadn''t brought the desired effect either.

I need to schedule the traffic output 4 times: 
-for other purposes & unknown traffic [10:,20:]
-for groups (that they have a minimum rate at all times) [fxx:]
-for priority (fair use, traffic usage based)[axx:,bxx:,cxx:]
-for users (that they don''t exceed a given maximum and are fairly
treated within their priority) [axx:yyyy],[bxx:yyyy],[cxx:yyyy]

Users should be allowed to borrow from each other, insufficient bandwidth
"requests" should be forwarded to the parent class/qdisc which should
try
to fulfill it by borrowing from other groups.

Am I missing an important point? Unfortunately I couldn''t find any 
examples showing a more complex tree.

Thank you,
 Bernd

On Saturday 18 June 2005 15:09, Bernd Froemel wrote:
> Am I missing an important point? Unfortunately I couldn''t find any
> examples showing a more complex tree.
Yes, unless I''m completely mistaken, you misunderstood something about
the
way QDiscs and especially HTB works.

Your old setup looked something like this:

1: HTB Qdisc
 | | |
 | | \--- 1:1 HTB Root Class (10mbit)
 | |      |
 | |      \--- 1:10 HTB Child Class (1kbit/512kbit)
 | |
 | \--- 1:f1 HTB Root Class (2048kbit)
 |      |
 |      \--- 1:a1 HTB Child Class (2048kbit)
 |      \--- 1:b1 HTB Child Class (2048kbit)
 |      \--- 1:c1 HTB Child Class (2048kbit)
 |
 \--- 1:f2 HTB Root Class (1024kbit)
      |
      \--- 1:a2 HTB Child Class (1024kbit)
      \--- 1:b2 HTB Child Class (1024kbit)
      \--- 1:c2 HTB Child Class (1024kbit)

Root classes are independent from each other. A HTB qdisc by itself has 
unlimited bandwidth. Limiting is done by the classes, the root class 
imposing a global limit. You have three root classes, and a total 
bandwidth capacity of 10mbit+2048kbit+1024kbit on your link. This is 
assuming that the ceil parameter does not have any effect for root 
classes, otherwise you''d have 30mbit altogether.

Now, the 1:f1 class has a guaranteed rate of 2048kbit. It''s children,
1:a1
1:b1 and 1:c1 also have 2048kbit guaranteed each. This is impossible, 
since a parent with only 2048kbit guaranteed bandwidth cannot guarantee 
three times as much bandwidth to it''s children. The 1:f2 class has the 
same problem.

Your new setup isn''t any different; instead of using more than one root
class, you now use nested HTB qdiscs, which is extremely nasty:

1: HTB QDisc
 |
 \--- 1:1 HTB Root Class (10MBit)
      |
      \--- 1:10 HTB Child Class (8MBit/10MBit)
      |    *
      |    \--- 10: HTB QDisc
      |         |
      |         \--- 10:f1 HTB Root Class (2MBit)
      |              *
      |              \--- f1: HTB QDisc
      |                   [...]
      |
      \--- 1:40 HTB Child Class (2MBit/2MBit)

The tree is broken at *. Although it looks like the HTB QDiscs depend on 
each other, they actually don''t really know about each other. There is
no
borrowing/lending possible across different QDiscs / root classes.

You should build a tree with only one HTB QDisc and only one root class.
Something similar to this:

1: HTB QDisc
 |
 \--- 1:1 HTB Root Class (10MBit) [The total link capacity]
      |
      \--- 1:10 HTB Child Class (7MBit/10MBit) [High priority stuff]
      |    |
      |    \--- 1:a1 HTB Child Class (3MBit/10MBit)
      |    \--- 1:b1 HTB Child Class (2MBit/10MBit)
      |    \--- 1:c1 HTB Child Class (2MBit/10MBit)
      |
      \--- 1:40 HTB Child Class (3MBit/10MBit) [Low priority stuff]
           |
           \--- 1:a2 HTB Child Class (1MBit/10MBit)
           \--- 1:b2 HTB Child Class (1MBit/10MBit)
           \--- 1:c2 HTB Child Class (1MBit/10MBit)

Make sure your child class rates add up to the parent class rate (7MBit = 
3+2+2MBit, 3MBit = 1+1+1MBit). Use smaller units than MBit if you want to 
divide more accurately.

That''s about it. Use only one HTB Qdisc, and only one HTB Root Class,
and
divide rates properly, then things should start working the way you expect 
them to.

HTH
Andreas

> You should build a tree with only one HTB QDisc and only one root class.
> Something similar to this:
> 
> 1: HTB QDisc
>  |
>  \--- 1:1 HTB Root Class (10MBit) [The total link capacity]
>       |
>       \--- 1:10 HTB Child Class (7MBit/10MBit) [High priority stuff]
>       |    |
>       |    \--- 1:a1 HTB Child Class (3MBit/10MBit)
>       |    \--- 1:b1 HTB Child Class (2MBit/10MBit)
>       |    \--- 1:c1 HTB Child Class (2MBit/10MBit)
>       |
>       \--- 1:40 HTB Child Class (3MBit/10MBit) [Low priority stuff]
>            |
>            \--- 1:a2 HTB Child Class (1MBit/10MBit)
>            \--- 1:b2 HTB Child Class (1MBit/10MBit)
>            \--- 1:c2 HTB Child Class (1MBit/10MBit)
> 
> Make sure your child class rates add up to the parent class rate (7MBit = 
> 3+2+2MBit, 3MBit = 1+1+1MBit). Use smaller units than MBit if you want to 
> divide more accurately.
> 
> That''s about it. Use only one HTB Qdisc, and only one HTB Root
Class, and
> divide rates properly, then things should start working the way you expect 
> them to.
First, thank you very much for your effort!
Now everything becomes a lot  more clear (especially about of the qdisc roles,
and the complete nonsense I tried)!

Just that I don''t make any more mistakes, would this tree work
as I expect it (with the groups and users and priority stuff)?

1: HTB QDisc
 |
 \--- 1:1 HTB Root Class (10MBit) [The total link capacity]
      |
      \--- 1:f1 HTB Child Class (3MBit/10MBit) [group one]
      |    |
      |    \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1
      |    |    |
      |    |    \---1:e1 HTB child Class (120kbit/1Mbit) [user 1]
      |    |    \---1:e2 HTB child Class (120kbit/2Mbit) [user 2]
      |    |    \---1:e3 HTB child Class (120kbit/512kbit) [user 3]
      |    |    
      |    \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2
      |    |    |
      |    |    \---1:e4 HTB child Class (120kbit/1Mbit) [user 1]
      |    |    \---1:e5 HTB child Class (120kbit/2Mbit) [user 2]
      |    |    \---1:e6 HTB child Class (120kbit/512kbit) [user 3]
      |    | 
      |    \--- 1:c1 HTB Child Class (1MBit/10MBit) prio 3
      |
      \--- 1:f2 HTB Child Class (3MBit/10MBit) [group two]
      |    |
      |    \--- 1:a2 HTB Child Class (1MBit/10MBit) prio 1
      |    \--- 1:b2 HTB Child Class (1MBit/10MBit) prio 2
      |    \--- 1:c2 HTB Child Class (1MBit/10MBit) prio 3
      |
      \--- 1:f3 HTB Child Class (3Mbit/10Mbit) [group three]
	.
	.
	.

Regards,
 Bernd

On Saturday 18 June 2005 16:53, Bernd Froemel wrote:
> Just that I don''t make any more mistakes, would this tree work
> as I expect it (with the groups and users and priority stuff)?
Sorry for the late reply, I wasn''t at home over the weekend.
The tree looks better now, except the class rates still don''t add up
>  \--- 1:1 HTB Root Class (10MBit) [The total link capacity]
Children of this class have 3MBit+3MBit+3MBit == 9MBit, should be 10.
>       |    \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1
Children of this class have 120kbit+120kbit+120kbit == 360kbit, should be 
1MBit.
>       |    \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2
Same here.

For my own setup, I''m using a script that converts all rates into bps
(or
whatever is the smallest unit available) and then does the math for the 
child classes automatically. I recommend you do something similar. Your 
classes will have a lot more borrowing to do if you don''t maximize
their
guaranteed rates.

About the priority stuff, I''m not sure if it will work the way you
expect
it, because I don''t know exactly what you expect. Classes with low 
priority won''t be allowed to borrow, if there is another class with
higher
priority that wants to borrow.

As a result, on full links (and links that require shaping are usually 
full), low priority classes usually don''t get to borrow anything at
all.

I suggest not using the prio parameter at all in the beginning; logical 
errors in the shaping setup are much more obvious if you don''t have to 
consider priority also. Later, if you find that you require additional 
prio for HTB classes, you can start experimenting with it and see if the 
results fit your requirements.

HTH
Andreas

No problem! You''ve been a great help, now I  have a rather good working
shaping system. I expect the last "problems" (shaping adaptation is a
bit slow, it takes up to 5-10 minutes until the full link is used after
script reinit) will be solved when I make the script implementation
"proper". But to sum it up: one programmer (usually, I don''t
do system
administrating stuff) and around 300 users are much happier now :)
> Same here.
I see - thanks for pointing it out.
> For my own setup, I''m using a script that converts all rates into
bps (or
> whatever is the smallest unit available) and then does the math for the 
> child classes automatically. I recommend you do something similar. Your 
> classes will have a lot more borrowing to do if you don''t maximize
their
> guaranteed rates.
I didn''t know it is that strict (right now I guarantee around
90kbit/sec
to everyone but allow an up-to bandwidth of the selected subscription 
(512kbit-2Mbit) through the ceil parameter. I will fix this and will
calculate a fixed rate for every user class which sums up to the full
bandwidth of the parent class.


Again, thanks for all your help Andreas!

Byebye,
Bernd



On Sat, 2005-06-18 at 16:53 +0200, Bernd Froemel wrote:
> 
> 1: HTB QDisc
>  |
>  \--- 1:1 HTB Root Class (10MBit) [The total link capacity]
>       |
>       \--- 1:f1 HTB Child Class (3MBit/10MBit) [group one]
>       |    |
>       |    \--- 1:a1 HTB Child Class (1MBit/10MBit) prio 1
>       |    |    |
>       |    |    \---1:e1 HTB child Class (120kbit/1Mbit) [user 1]
>       |    |    \---1:e2 HTB child Class (120kbit/2Mbit) [user 2]
>       |    |    \---1:e3 HTB child Class (120kbit/512kbit) [user 3]
>       |    |    
>       |    \--- 1:b1 HTB Child Class (1MBit/10MBit) prio 2
>       |    |    |
>       |    |    \---1:e4 HTB child Class (120kbit/1Mbit) [user 1]
>       |    |    \---1:e5 HTB child Class (120kbit/2Mbit) [user 2]
>       |    |    \---1:e6 HTB child Class (120kbit/512kbit) [user 3]
>       |    | 
>       |    \--- 1:c1 HTB Child Class (1MBit/10MBit) prio 3
>       |
>       \--- 1:f2 HTB Child Class (3MBit/10MBit) [group two]
>       |    |
>       |    \--- 1:a2 HTB Child Class (1MBit/10MBit) prio 1
>       |    \--- 1:b2 HTB Child Class (1MBit/10MBit) prio 2
>       |    \--- 1:c2 HTB Child Class (1MBit/10MBit) prio 3
>       |
>       \--- 1:f3 HTB Child Class (3Mbit/10Mbit) [group three]
> 	.
> 	.
> 	.
> 
> Regards,
>  Bernd
>