Hello,
after spending several hours reading archives, I decided to write new
post.
I successfully set up packet classification, made some basic HTB
setup, made some simple graphical representation from HTB statistics
data...
BUT, I cannot figure out how to refine HTB to get this behaviour:
I need that class "p2p" should be the last one to get some link
capacity.
If I set both RATE and CEIL for "p2p" to some constant value,
everything is OK. When I set CEIL for "p2p" e.g. to 2048 kbps (total
link capacity), "p2p" starts to get most of the link capacity and does
not want to give capacity to other classes with higher priorities!
I made a screen to help explain my problem. Please, see this picture:
http://elusion.sk/visual_inet_hory.png
Point A - "p2p" class (red line) has both RATE and CEIL set to fixed
value. I started download from web to fill "www" class. You can see,
"www" got remaining link capacity, "p2p" without a change -
all as
expected.
Point B - I reset CEIL of "p2p" class to 2048 kbit (to use rest of the
unused capacity by other classes with higher priorites, up to link
capacity)
Point C - I started the same download from web as in Point A and this
is the PROBLEM. You can see, that "p2p" lowered a bit, and
"www" class
got some capacity. What I expect is, that "www" class get most of the
link capacity and "p2p" class will fall to it''s RATE, because
"http"
class has higher priority than "p2p"
Point D - CEIL of "p2p" reset to fixed value
Point E - I tried the same download from web just to be sure it works.
This is my HTB config (using latest htb-init script):
root:
RATE=100Mbit
( local traffic:
RATE=50Mbit
CEIL=100Mbit
PRIO=0 )
"ctrl" class:
RATE=64kbit
CEIL=256kbit
PRIO=1
"other" class:
RATE=128kbit
CEIL=768kbit
PRIO=2
"www" class:
RATE=512kbit
CEIL=2048kbit
PRIO=3
"p2p" class:
RATE=386kbit
CEIL=386kbit (or 2048, for Points B and C in figure above)
PRIO=4
All want to set up is: If class "www" with prio 3 gets fully utilized
(I start 10 http downloads at the same time), I expect "p2p" class
with prio 4 to fall down to it''s RATE 386 kbit - to free capacity for
"www" class
In another words: I expect "www" to overrule "p2p"
("p2p" still
getting it''s RATE)
Can someone help to explain this? Maybe I did not get purpose of PRIO
setting, not sure. Help, please
Best Regards,
B. Gereg
On Mon, Feb 20, 2006 at 10:59:33PM +0100, Boris Gereg wrote:> I made a screen to help explain my problem. Please, see this picture: > http://elusion.sk/visual_inet_hory.pngNice graph. I assume this is on downstream, and you rely on HTB to drop packets for you. You may have read this in the archives already - it''s much harder to shape downstream than upstream, because you can''t really influence what the other side is sending you. So no matter what you do it''s probably hard to get near-optimal results.> This is my HTB config (using latest htb-init script):I must admit I''m not familiar with htb-init. What are the parent-child relationships here? I''m missing the "internet" parent class that groups all the other traffics (except local) together. Does htb-init generate that on it''s own somehow? If not, chances are your HTB tree is just exceeding your line capacity in general, as all classes are allowed to borrow without limit, rendering the prio setting uneffective, leading to random results. Could you post the output of ''tc -d qdisc/class show dev $DEVICE''? Regards, Andreas Klauer
On Tue, Feb 21, 2006 at 12:49:59AM +0100, Boris Gereg wrote:> (first of all, please, how to reply to some article in LARTC via mail > to post it into right thread?)Using ''reply all'', or ''reply list'' if your mail software offers it. If all else fails, just hit ''reply'' and add the mailing list to CC.> So, I am definitely shaping outgoing traffic (upstream)Yes, outgoing traffic from router to your network, which actually contains the downstream traffic from the internet. Right?> tc -d class show dev eth0 > > class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu > 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7 > > class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate > 50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst > 51587b/8 mpu 0b overhead 0b level 0 > > class htb 1:3010 parent 1:2 leaf 3010: prio 1 quantum 1000 rate > 64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst > 1727b/8 mpu 0b overhead 0b level 0 > > class htb 1:3020 parent 1:2 leaf 3020: prio 2 quantum 1600 rate > 128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst > 1983b/8 mpu 0b overhead 0b level 0 > > class htb 1:3030 parent 1:2 leaf 3030: prio 3 quantum 6400 rate > 512000bit ceil 2048Kbit burst 1855b/8 mpu 0b overhead 0b cburst > 2623b/8 mpu 0b overhead 0b level 0 > > class htb 1:5040 parent 1:2 leaf 5040: prio 4 quantum 4825 rate > 386000bit ceil 386000bit burst 1792b/8 mpu 0b overhead 0b cburst > 1792b/8 mpu 0b overhead 0b level 0It''s as I suspected, your current HTB tree looks like this: 1: HTB Qdisc | \--- 1:2 HTB root class (100000Kbit:100000kbit) | \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit) \--- 1:3010 HTB leaf class (64000bit:256000bit) \--- 1:3020 HTB leaf class (128000bit:768000bit) \--- 1:3030 HTB leaf class (512000bit:2048Kbit) \--- 1:5040 HTB leaf class (386000bit:386000bit) HTB classes borrow from their parent; in this setup, the parent class offers a whopping 100000Kbit for that purpose. Unless the 1:2000 class has got a very high priority and is maxing out the line all the time, there is no limit to borrowing at all, because the other classes will never reach the 100000Kbit from their parent. So the classes above are actually not limited by their rate, but by their ceil; the only class that will respect its rate in this setup is 1:5040, because it''s got the same rate and ceil. Assuming that 1:5040 was your P2P class, if you set the ceil of this class to 2048Kbit, it will (try to) use 2048Kbit at all times, because the parent (thinks it) is able to offer it. You need a class that knows of your total internet bandwidth somewhere. Assuming that it is 2048Kbit, your tree should maybe look more like this: 1: HTB Qdisc | \--- 1:2 HTB root class (100000Kbit:100000kbit) | \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit) | \--- 1:3000 HTB parent class (2048Kbit:2048Kbit) | \--- 1:3010 HTB leaf class (64000bit:256000bit) \--- 1:3020 HTB leaf class (128000bit:768000bit) \--- 1:3030 HTB leaf class (512000bit:2048Kbit) \--- 1:5040 HTB leaf class (386000bit:386000bit) In this setup, the 2048Kbit class is the limiting factor for the leaf classes, except for the 1:2000 class, which should be used for local LAN traffic only. HTH Andreas Klauer
Hello, thanks Andreas, I reconfigured HTB to get your suggested hierarhy: AK> 1: HTB Qdisc AK> | AK> \--- 1:2 HTB root class (100000Kbit:100000kbit) AK> | AK> \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit) #local AK> | AK> \--- 1:3000 HTB parent class (2048Kbit:2048Kbit) AK> | AK> \--- 1:3010 HTB leaf class (64000bit:256000bit) #ctrl AK> \--- 1:3020 HTB leaf class (128000bit:768000bit) #other AK> \--- 1:3030 HTB leaf class (512000bit:2048Kbit) #www AK> \--- 1:5040 HTB leaf class (386000bit:386000bit) #p2p tc -d class show dev eth0 class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7 # local: class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate 50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 0 # root for internet trafic class htb 1:3000 parent 1:2 rate 2048Kbit ceil 2048Kbit burst 2623b/8 mpu 0b overhead 0b cburst 2623b/8 mpu 0b overhead 0b level 6 # "ctrl" class class htb 1:3010 parent 1:3000 leaf 3010: prio 1 quantum 1000 rate 64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst 1727b/8 mpu 0b overhead 0b level 0 # "other" class class htb 1:3020 parent 1:3000 leaf 3020: prio 2 quantum 1600 rate 128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst 1983b/8 mpu 0b overhead 0b level 0 # "www" class class htb 1:3030 parent 1:3000 leaf 3030: prio 3 quantum 12800 rate 1024Kbit ceil 2048Kbit burst 2111b/8 mpu 0b overhead 0b cburst 2623b/8 mpu 0b overhead 0b level 0 # "p2p" class class htb 1:5040 parent 1:3000 leaf 5040: prio 4 quantum 1600 rate 128000bit ceil 256000bit burst 1663b/8 mpu 0b overhead 0b cburst 1727b/8 mpu 0b overhead 0b level 0 Sorry to say, results are not as expected. I made new measurements, please see following picture (based on HTB config above): http://elusion.sk/visual_inet_6.png Point A - until now, "p2" clas had CEIL 386 kbit. Now, I reset "p2p" class CEIL to 2048 kbit. As you can see, "p2p" rised as expected Point B - "www" class RATE-CEIL is 512-2048 kbit, "p2p" class RATE-CEIL is 128-2048 kbit. PROBLEM: why did not "www" get more capacity? Point C - "www" class RATE-CEIL is 1536-2048 kbit, "p2p" class RATE-CEIL is 128-2048 kbit. PROBLEM: RATE of "www" is 1536, but class did not get this capacity; "p2p" should fall to it''s RATE 256 kbit Point D - "www" class RATE-CEIL is 1024-2048 kbit, "p2p" class RATE-CEIL is 128-2048 kbit - no change Point E - "www" class RATE-CEIL is 1024-2048 kbit, "p2p" class RATE-CEIL is 128-256 kbit - as expected: "p2p" is bounded to 256 kbit limit, so the rest of the capacity was used by "www" class I have expected that in points B, C and D, "p2p" class falls down to it''s specified rate and "www" gets most of the capacity. But this is not happening - "p2p" will not fall down Any other things to test, please? Best regards, B. Gereg mailto:qwerty@elusion.sk -----Originalna sprava----- Od: Andreas Klauer [mailto:Andreas.Klauer@metamorpher.de] Poslane: Tuesday, February 21, 2006, 8:52:49 AM Komu: qwerty@elusion.sk Predmet: [LARTC] HTB, strange capacity distribution AK> On Tue, Feb 21, 2006 at 12:49:59AM +0100, Boris Gereg wrote:>> (first of all, please, how to reply to some article in LARTC via mail >> to post it into right thread?)AK> Using ''reply all'', or ''reply list'' if your mail software offers it. AK> If all else fails, just hit ''reply'' and add the mailing list to CC.>> So, I am definitely shaping outgoing traffic (upstream)AK> Yes, outgoing traffic from router to your network, which actually AK> contains the downstream traffic from the internet. Right?>> tc -d class show dev eth0 >> >> class htb 1:2 root rate 100000Kbit ceil 100000Kbit burst 51587b/8 mpu >> 0b overhead 0b cburst 51587b/8 mpu 0b overhead 0b level 7 >> >> class htb 1:2000 parent 1:2 leaf 2000: prio 0 quantum 200000 rate >> 50000Kbit ceil 100000Kbit burst 26593b/8 mpu 0b overhead 0b cburst >> 51587b/8 mpu 0b overhead 0b level 0 >> >> class htb 1:3010 parent 1:2 leaf 3010: prio 1 quantum 1000 rate >> 64000bit ceil 256000bit burst 1631b/8 mpu 0b overhead 0b cburst >> 1727b/8 mpu 0b overhead 0b level 0 >> >> class htb 1:3020 parent 1:2 leaf 3020: prio 2 quantum 1600 rate >> 128000bit ceil 768000bit burst 1663b/8 mpu 0b overhead 0b cburst >> 1983b/8 mpu 0b overhead 0b level 0 >> >> class htb 1:3030 parent 1:2 leaf 3030: prio 3 quantum 6400 rate >> 512000bit ceil 2048Kbit burst 1855b/8 mpu 0b overhead 0b cburst >> 2623b/8 mpu 0b overhead 0b level 0 >> >> class htb 1:5040 parent 1:2 leaf 5040: prio 4 quantum 4825 rate >> 386000bit ceil 386000bit burst 1792b/8 mpu 0b overhead 0b cburst >> 1792b/8 mpu 0b overhead 0b level 0AK> It''s as I suspected, your current HTB tree looks like this: AK> 1: HTB Qdisc AK> | AK> \--- 1:2 HTB root class (100000Kbit:100000kbit) AK> | AK> \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit) AK> \--- 1:3010 HTB leaf class (64000bit:256000bit) AK> \--- 1:3020 HTB leaf class (128000bit:768000bit) AK> \--- 1:3030 HTB leaf class (512000bit:2048Kbit) AK> \--- 1:5040 HTB leaf class (386000bit:386000bit) AK> HTB classes borrow from their parent; in this setup, the parent class AK> offers a whopping 100000Kbit for that purpose. Unless the 1:2000 class AK> has got a very high priority and is maxing out the line all the time, AK> there is no limit to borrowing at all, because the other classes will AK> never reach the 100000Kbit from their parent. AK> So the classes above are actually not limited by their rate, but by AK> their ceil; the only class that will respect its rate in this setup AK> is 1:5040, because it''s got the same rate and ceil. AK> Assuming that 1:5040 was your P2P class, if you set the ceil of this AK> class to 2048Kbit, it will (try to) use 2048Kbit at all times, because AK> the parent (thinks it) is able to offer it. AK> You need a class that knows of your total internet bandwidth somewhere. AK> Assuming that it is 2048Kbit, your tree should maybe look more like this: AK> 1: HTB Qdisc AK> | AK> \--- 1:2 HTB root class (100000Kbit:100000kbit) AK> | AK> \--- 1:2000 HTB leaf class (50000Kbit:100000Kbit) AK> | AK> \--- 1:3000 HTB parent class (2048Kbit:2048Kbit) AK> | AK> \--- 1:3010 HTB leaf class (64000bit:256000bit) AK> \--- 1:3020 HTB leaf class (128000bit:768000bit) AK> \--- 1:3030 HTB leaf class (512000bit:2048Kbit) AK> \--- 1:5040 HTB leaf class (386000bit:386000bit) AK> In this setup, the 2048Kbit class is the limiting factor for the leaf AK> classes, except for the 1:2000 class, which should be used for local AK> LAN traffic only. AK> HTH AK> Andreas Klauer
On Tue, Feb 21, 2006 at 02:21:36PM +0100, Boris Gereg wrote:> thanks Andreas, I reconfigured HTB to get your suggested hierarhy:One thing I forgot in my graph: Make sure that the rates always add up, i.e. the sum of the child class rates should equal the parent class rate. It''s unlikely to be the cause of your problem, but it''s important to get this right nevertheless.> Any other things to test, please?Just to see wether we are going into the right direction at all, could you run the following experiment: - Lower the rate and ceil of class 1:2 to 8096kbit. - Lower the rate and ceil of class 1:2000 to 7072kbit. - Lower the rate and ceil of class 1:3000 to 1024kbit. - For class 1:3010, set rate to 64kbit, ceil to 256kbit. - For class 1:3020, set rate to 128kbit, ceil to 768kbit. - For class 1:3040, set rate to 704kbit, ceil to 1024kbit. - For class 1:5040, set rate to 128kbit, ceil to 1024kbit. (You can adjust the rates for these classes as you like, just make sure that the sum is 1024kbit) If in this setup the shaping works as expected - WWW should get 704kbit at all times, P2P only slightly more than 128kbit while WWW downloads are active - then the limiting and distribution of HTB most likely works, and it''s just too high rates or r2q/quantum that make it go bad. In this case, you''d have to measure realistic throughput rates of your network (even a 100mbit LAN may not be able to guarantee 100000kbit at all times) and of your internet connection (may not be able to serve 2048kbit at all times). For downstream shaping to work, you have to be the bottleneck. If you get the same problem in this setup (P2P taking all the bandwidth away from WWW), then the problem is most likely something different, and we have to look at it from a different angle. Regards Andreas Klauer
Hello Andreas, AK> Just to see wether we are going into the right direction at all, could AK> you run the following experiment: AK> - Lower the rate and ceil of class 1:2 to 8096kbit. AK> - Lower the rate and ceil of class 1:2000 to 7072kbit. AK> - Lower the rate and ceil of class 1:3000 to 1024kbit. AK> - For class 1:3010, set rate to 64kbit, ceil to 256kbit. AK> - For class 1:3020, set rate to 128kbit, ceil to 768kbit. AK> - For class 1:3040, set rate to 704kbit, ceil to 1024kbit. AK> - For class 1:5040, set rate to 128kbit, ceil to 1024kbit. I did what you suggested and the results are as expected! You can see this picture to verify: http://elusion.sk/visual_inet_7.png At 4:25 I started HTTP download. P2P class immediately droped down to it''s RATE, WWW class got it''s RATE. At 4:33 I stopped HTTP download, P2P class got rest of capacity. AK> ... it''s just too high rates or r2q/quantum that make it go bad. In this AK> case, you''d have to measure realistic throughput rates of your network AK> (even a 100mbit LAN may not be able to guarantee 100000kbit at all times) AK> and of your internet connection (may not be able to serve 2048kbit at AK> all times). For downstream shaping to work, you have to be the bottleneck. There are messages in syslog like this: kernel: HTB: quantum of class 10002 is big. Consider r2q change. kernel: HTB: quantum of class 12000 is big. Consider r2q change. kernel: HTB: quantum of class 13010 is small. Consider r2q change. Please, are there some hints for setting r2q or quantum parameters? http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm does not give extensive information on this. Or, is it just the matter of testing and searching for optimal parameters? Anyway, thanks a lot Andreas. This was the most important break for me. Rest is just tuning. Best Regards B. Gereg
On Thu, Feb 23, 2006 at 05:00:12AM +0100, Boris Gereg wrote:> I did what you suggested and the results are as expected! > You can see this picture to verify: http://elusion.sk/visual_inet_7.png > > At 4:25 I started HTTP download. P2P class immediately droped down to > it''s RATE, WWW class got it''s RATE. At 4:33 I stopped HTTP download, > P2P class got rest of capacity.Alright. I suggest you do some measuring, to find out your real rates, and set HTB rates to be slightly lower so that you are the bottleneck. Most likely you''ll have to experiment a little until you find the best setting for your setup.> Please, are there some hints for setting r2q or quantum parameters?Actually, I specify the quantum directly, with ''quantum $MTU'' for every class. I don''t know wether that''s a good thing or a bad thing, but it worked very well for me, and seems to work well for others... at least nobody reported a problem to me so far that could be traced to be caused by this quantum setting. It should not be smaller than your MTU, and not too big. With a huge difference in rates (100Mbit vs 64kbit) there is no r2q that will fit all classes. So there is no other way as to set quantum directly at least for some classes (and I set it for all...). Regards Andreas Klauer