Eliot, Wireless and Server Administrator, Great Lakes Internet
2006-May-11 17:02 UTC
HFSC and prioritization
I''m using HFSC to limit bandwidth for our wireless customers. However, I''d also like the bandwidth prioritized based on packet type. This is what I''m trying right now, and I''d just like some input from anyone out there knowledgeable in this on whether it does what I want it to do: Eth1 -> HFSC ........|-> HFSC (User1) (Min 512 Kb, Max 1024 Kb, Burst 1536 Kb@2s) ........|...|-> Prio_qdisc ........|.......|-> 1 ........|.......|...|->HFSC VoIP/Interactive(max 30ms, Real 128Kb) ........|.......| ........|.......|-> 2 ........|.......|...|->HFSC Web,FTP(Min 512Kb,Max 1024Kb,Burst 1536Kb@2s) ........|.......| ........|.......|-> 3 ........|...........|->HFSC P2P(Min 0 Kb, Max 1024 Kb, all shared) ........| ........|-> HFSC (User2) (etc) ..etc... What I''m aiming for is: No matter what type of traffic is being transferred, the user is guaranteed 512Kbps and can max out at 1024Kbps, but can also burst up to 1536Kbps for 2 seconds. VoIP / SSH / Telnet / non-data ACK packets get priority over everything else. It would be guaranteed 128Kbps of bandwidth if it were needed. Ideally, it would not reserve that bandwidth unless it was actually needed. It should also get more bandwidth than 128Kbps if it is needed, but anything after 128Kbps it has to fight for with everything else (except it has a higher priority so it should win out in 99.99999% of cases?) Web, FTP, and other unclassifiable traffic would get second priority and would be guaranteed 512Kbps of bandwidth with a maximum of 1024Kbps and can burst up to 1536Kbps for 2 seconds. Finally, P2P traffic is not guaranteed anything, but could use all available bandwidth if the user is not doing anything else. No matter how much bandwidth P2P wants, if something else needs bandwidth, the other traffic should win out and receive the bandwidth. These are my actual rules: # Base user class tc class add dev wivl4 parent 5:0 classid 5:130 hfsc ls m1 1536.0Kbit d 2000ms m2 512.00Kbit ul m2 1024Kbit # Priority queue tc qdisc add dev wivl4 parent 5:130 handle 134: prio bands 3 tc qdisc add dev wivl4 parent 134:1 handle 135: hfsc default 1 tc qdisc add dev wivl4 parent 134:2 handle 136: hfsc default 1 tc qdisc add dev wivl4 parent 134:3 handle 137: hfsc default 1 # VoIP / Interactive tc class add dev wivl4 parent 135: classid 135:1 hfsc sc umax 1500b dmax 30ms rate 128Kbit # Web tc class add dev wivl4 parent 136: classid 136:1 hfsc rt m2 512.00Kbit ls m2 512.0Kbit ul m2 1024Kbit # P2P tc class add dev wivl4 parent 137: classid 137:1 hfsc ls m2 1024.0Kbit ul m2 1024.0Kbit Does this look appropriate? Eliot Gable Certified Wireless Network Administrator (CWNA) Certified Wireless Security Professional (CWSP) Cisco Certified Network Associate (CCNA) CompTIA Security+ Certified CompTIA Network+ Certified Network and Systems Administrator Great Lakes Internet, Inc. 112 North Howard Croswell, MI 48422 (810) 679-3395 (877) 558-8324 Now offering Broadband Wireless Internet access in Croswell, Lexington, Brown City, Yale, and Sandusky. Call for details.
> # Base user class > tc class add dev wivl4 parent 5:0 classid 5:130 hfsc ls m1 1536.0Kbit d > 2000ms m2 512.00Kbit ul m2 1024Kbit > > # Priority queue > tc qdisc add dev wivl4 parent 5:130 handle 134: prio bands 3 > tc qdisc add dev wivl4 parent 134:1 handle 135: hfsc default 1 > tc qdisc add dev wivl4 parent 134:2 handle 136: hfsc default 1 > tc qdisc add dev wivl4 parent 134:3 handle 137: hfsc default 1 > > # VoIP / Interactive > tc class add dev wivl4 parent 135: classid 135:1 hfsc sc umax 1500b dmax > 30ms rate 128Kbit > > # Web > tc class add dev wivl4 parent 136: classid 136:1 hfsc rt m2 512.00Kbit > ls m2 512.0Kbit ul m2 1024Kbit > > # P2P > tc class add dev wivl4 parent 137: classid 137:1 hfsc ls m2 1024.0Kbit > ul m2 1024.0Kbit > > > Does this look appropriate? > >My understanding of HFSC is limited, but i''m fairly sure its similar to all other qdiscs in one respect that would make the config you have shown, not actually work as you''ve described. Each of those HFSC qdiscs is a seperate entity, no sharing will occur between those HFSC classes because each one belongs to a different qdisc. If you implemented this, the priority portion would likely work, but if all 3 classes were sending they would be trying for their individual max, resulting in a combined total send of 3072kbps. If you want what you''ve described then there would need to be some sort of way to define that same priority within one HSFC between the classes. I know HTB has the prio parameter but I''ve never found good documentation on HSFC and don''t know if it has an equivalent. Of course all of what I said would be wrong if HSFC has inter-qdisc communication but I highly doubt that as it seems to go against the design of most qdiscs. What is there for good HSFC documentation out there right now anyways? Thanks, Jody
Jody Shumaker wrote:> My understanding of HFSC is limited, but i''m fairly sure its similar > to all other qdiscs in one respect that would make the config you have > shown, not actually work as you''ve described. Each of those HFSC > qdiscs is a seperate entity, no sharing will occur between those HFSC > classes because each one belongs to a different qdisc. If you > implemented this, the priority portion would likely work, but if all > 3 classes were sending they would be trying for their individual max, > resulting in a combined total send of 3072kbps.Correct.> If you want what > you''ve described then there would need to be some sort of way to > define that same priority within one HSFC between the classes. I know > HTB has the prio parameter but I''ve never found good documentation on > HSFC and don''t know if it has an equivalent.HFSC doesn''t support strict priorities (and neither does HTB, the priorities just affect unused bandwidth and is still limited by the ceiling). At least in the case of HFSC this is intentional, strict priority is not very friendly because it allows traffic to be entirely excluded, HFSC''s goals are to enable flexible sharing by allowing to seperately specify bandwidth and delay requirements. If you really want strict priority you can use the prio qdisc as _child_ of HFSC.> Of course all of what I said would be wrong if HSFC has inter-qdisc > communication but I highly doubt that as it seems to go against the > design of most qdiscs. > > What is there for good HSFC documentation out there right now anyways?There is the original papers by Hui Zhang et al., which is mostly about the theory and not very suitable for users - but still worth reading if you''re not scared by use of some math. There used to be some documentation called "HFSC for Router Plugins", which is partially applicable for Linux .. and some ALTQ and *BSD documentation which is partially applicable as well. Besides that there seem to be a few german student research projects about this subject, but all I know of are in german. Last thing I know of is an article written by a friend of mine for the german Linux Magazin, unfortunately also only in german, but reviewed by myself and mostly correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are welcome :)
Patrick McHardy wrote:> Jody Shumaker wrote: > >>What is there for good HSFC documentation out there right now anyways? > > > There is the original papers by Hui Zhang et al., which is mostly > about the theory and not very suitable for users - but still worth > reading if you''re not scared by use of some math. > There used to be some documentation called "HFSC for Router Plugins", > which is partially applicable for Linux .. and some ALTQ and *BSD > documentation which is partially applicable as well. Besides that > there seem to be a few german student research projects about this > subject, but all I know of are in german. Last thing I know of is > an article written by a friend of mine for the german Linux Magazin, > unfortunately also only in german, but reviewed by myself and mostly > correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are > welcome :)I forgot to mention, there is actually a quite easy way to get started with HFSC if you already have a working HTB setup. Just delete the "prio" statements, replace "htb" by "hfsc", "rate" by "sc rate", "ceil" by "ul rate" and delete all the remaining htb specific parameters and you have something working.
Alexandru Dragoi wrote:> I think i''d like more docs in english about hfsc.Me too. I don''t have time to write one myself (and I''m not good at this), but I can assist if anyone wants to do it.> I would like to know > also some tips about scalability at large amount of traffic, like more > than 100mbit and more than 20kpps. I had a setup one that share 200mbit > on 2 imq devices(both with a parent class of 200mbit), each with about > 1000 hfsc classes. About 2 classes with only rt courves, another 2 with > rt and ls and ul courves, and the rest (end users) with ls and ul > courves. All only with m2 parametter. And at high traffic packet loss > appeared. After switching to htb, no more packet loss. > > Thanks in advance.Mhh .. I know of setups where HFSC is running with 10k classes and high bandwidth (>= 100mbit, don''t know the exact amount). When switchting to rbtrees I made some benchmarks and it performed almost identical to HTB, so my guess is that its related to IMQ, which still seems to be pretty broken. It could of course also be a different configuration mistake, hard to tell without seeing the actual configuration.
Patrick McHardy wrote:> Patrick McHardy wrote: > >>Jody Shumaker wrote: >> >> >>>What is there for good HSFC documentation out there right now anyways? >> >> >>There is the original papers by Hui Zhang et al., which is mostly >>about the theory and not very suitable for users - but still worth >>reading if you''re not scared by use of some math. >>There used to be some documentation called "HFSC for Router Plugins", >>which is partially applicable for Linux .. and some ALTQ and *BSD >>documentation which is partially applicable as well. Besides that >>there seem to be a few german student research projects about this >>subject, but all I know of are in german. Last thing I know of is >>an article written by a friend of mine for the german Linux Magazin, >>unfortunately also only in german, but reviewed by myself and mostly >>correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are >>welcome :) > > > I forgot to mention, there is actually a quite easy way to get > started with HFSC if you already have a working HTB setup. Just > delete the "prio" statements, replace "htb" by "hfsc", > "rate" by "sc rate", "ceil" by "ul rate" and delete all the > remaining htb specific parameters and you have something working.One thing to note is that HFSC will drop, rather than pass unshaped, traffic that is unclassified. So if you don''t use a default class and don''t filter arp to a class then HFSC will appear broken whereas HTB will work. Andy.
> HFSC doesn''t support strict priorities (and neither does HTB, the > priorities just affect unused bandwidth and is still limited by the > ceiling). At least in the case of HFSC this is intentional, strict > priority is not very friendly because it allows traffic to be > entirely excluded, HFSC''s goals are to enable flexible sharing by > allowing to seperately specify bandwidth and delay requirements. > If you really want strict priority you can use the prio qdisc as > _child_ of HFSC. >I always forget this about the prio and HTB. With HSFC does use of the the max latency settings possibly get the desired goal from using prio? I think this is what always appealed to me about HSFC from the little I could understand. That if I had an interactive class, it''d always favor getting those packets through sooner than others, trying to honor a latency, if I set it up correctly.> > What is there for good HSFC documentation out there right now anyways? > > There is the original papers by Hui Zhang et al., which is mostly > about the theory and not very suitable for users - but still worth > reading if you''re not scared by use of some math. > There used to be some documentation called "HFSC for Router Plugins", > which is partially applicable for Linux .. and some ALTQ and *BSD > documentation which is partially applicable as well. Besides that > there seem to be a few german student research projects about this > subject, but all I know of are in german. Last thing I know of is > an article written by a friend of mine for the german Linux Magazin, > unfortunately also only in german, but reviewed by myself and mostly > correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are > welcome :) >Unfortunately the original papers were a bit too confusing for me when I last read them. I could understand parts of it, but translating that into hsfc options I would actually want to use, I couldn''t do. This german article looks like it''d have a lot of the clerification I''d need, but a babelfish translation was still too difficult to understand. If someone''s up for writing a good translation I''d greatly appreciate it. - Jody
Andy Furniss wrote:> One thing to note is that HFSC will drop, rather than pass unshaped, > traffic that is unclassified. > > So if you don''t use a default class and don''t filter arp to a class then > HFSC will appear broken whereas HTB will work.Good point, that is a trap that might easily make it appear as if HFSC is stalling under some conditions.
Jody Shumaker wrote:>> HFSC doesn''t support strict priorities (and neither does HTB, the >> priorities just affect unused bandwidth and is still limited by the >> ceiling). At least in the case of HFSC this is intentional, strict >> priority is not very friendly because it allows traffic to be >> entirely excluded, HFSC''s goals are to enable flexible sharing by >> allowing to seperately specify bandwidth and delay requirements. >> If you really want strict priority you can use the prio qdisc as >> _child_ of HFSC. >> > > I always forget this about the prio and HTB. With HSFC does use of > the the max latency settings possibly get the desired goal from using > prio? I think this is what always appealed to me about HSFC from the > little I could understand. That if I had an interactive class, it''d > always favor getting those packets through sooner than others, trying > to honor a latency, if I set it up correctly.Exactly. I think strict priority is mostly used because of laziness, unknown conditions or unflexible algorithms. You almost never want one kind of traffic to be able to stall everything else (which BTW raises some doubts about Linux''s default choice of a 3band prio qdisc). HFSC solves one and a half of these problems: without seperated bandwidth/delay specifications you are unable to express that some traffic should get good delay but doesn''t need much guaranteed bandwidth, so you have to use priorities. The half solved problem is unknown conditions, it can also work in work-conserving mode, which means that it will work fine on wireless or similar networks or without any maximum bandwidth specification, but in that case it can only provide fair sharing, no absolute guarantees. Only half-solved because the unknown condition could also be the amount of bandwidth or the delay required, in which case strict priority might be the only viable option.
Patrick McHardy wrote:>Alexandru Dragoi wrote: > > > >>I think i''d like more docs in english about hfsc. >> >> > >Me too. I don''t have time to write one myself (and I''m not good at >this), but I can assist if anyone wants to do it. > > > > >>I would like to know >>also some tips about scalability at large amount of traffic, like more >>than 100mbit and more than 20kpps. I had a setup one that share 200mbit >>on 2 imq devices(both with a parent class of 200mbit), each with about >>1000 hfsc classes. About 2 classes with only rt courves, another 2 with >>rt and ls and ul courves, and the rest (end users) with ls and ul >>courves. All only with m2 parametter. And at high traffic packet loss >>appeared. After switching to htb, no more packet loss. >> >>Thanks in advance. >> >> > > >Mhh .. I know of setups where HFSC is running with 10k classes and high >bandwidth (>= 100mbit, don''t know the exact amount). When switchting to >rbtrees I made some benchmarks and it performed almost identical to HTB, >so my guess is that its related to IMQ, which still seems to be pretty >broken. It could of course also be a different configuration mistake, >hard to tell without seeing the actual configuration. >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > >Hello, here are some lines of the hfsc script i used. #!/bin/bash tc=/sbin/tc $tc qdisc add dev imq0 root handle 1: hfsc default 3 $tc class add dev imq0 parent 1: classid 1:3 hfsc ls m2 200mbit ul m2 200mbit $tc class add dev imq0 parent 1: classid 1:2 hfsc ls m2 200mbit ul m2 200mbit $tc qdisc add dev imq1 root handle 1: hfsc default 3 $tc class add dev imq1 parent 1: classid 1:3 hfsc ls m2 20mbit ul m2 20mbit $tc class add dev imq1 parent 1: classid 1:2 hfsc ls m2 20mbit ul m2 20mbit $tc qdisc add dev imq2 root handle 1: hfsc default 3 $tc class add dev imq2 parent 1: classid 1:3 hfsc ls m2 200mbit ul m2 200mbit $tc class add dev imq2 parent 1: classid 1:2 hfsc ls m2 200mbit ul m2 200mbit $tc qdisc add dev imq3 root handle 1: hfsc default 3 $tc class add dev imq3 parent 1: classid 1:3 hfsc ls m2 20mbit ul m2 20mbit $tc class add dev imq3 parent 1: classid 1:2 hfsc ls m2 20mbit ul m2 20mbit ## An important client $tc class add dev imq0 parent 1:2 classid 1:0x6031 hfsc rt m2 100mbit $tc qdisc add dev imq0 parent 1:0x6031 sfq $tc filter add dev imq0 parent 1: protocol ip prio 10 u32 match ip dst x.y.z.0/22 flowid 1:0x6031 $tc class add dev imq1 parent 1:2 classid 1:0x6031 hfsc rt m2 9mbit $tc qdisc add dev imq1 parent 1:0x6031 sfq $tc filter add dev imq1 parent 1: protocol ip prio 10 u32 match ip dst x.y.z.0/22 flowid 1:0x6031 $tc class add dev imq2 parent 1:2 classid 1:0x6031 hfsc rt m2 100mbit $tc qdisc add dev imq2 parent 1:0x6031 sfq $tc filter add dev imq2 parent 1: protocol ip prio 10 u32 match ip src x.y.z.0/22 flowid 1:0x6031 $tc class add dev imq3 parent 1:2 classid 1:0x6031 hfsc rt m2 9mbit $tc qdisc add dev imq3 parent 1:0x6031 sfq $tc filter add dev imq3 parent 1: protocol ip prio 10 u32 match ip src x.y.z.0/22 flowid 1:0x6031 There were also a client with rt m2 20mbit ls m2 20mbit ul m2 100mbit on imq0 and im2, then rt m2 5mbit ls m2 5mbit ul m2 10mbit on other 2 imqs. The rest of clients , arround 1000, has each something like: $tc class add dev imq0 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul m2 20480Kbit $tc qdisc add dev imq0 parent 1:0x116a sfq $tc filter add dev imq0 parent 1: protocol ip prio 10 u32 match ip dst a.b.c.d/32 flowid 1:0x116a $tc class add dev imq1 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul m2 256Kbit $tc qdisc add dev imq1 parent 1:0x116a sfq $tc filter add dev imq1 parent 1: protocol ip prio 10 u32 match ip dst a.b.c.d/32 flowid 1:0x116a $tc class add dev imq2 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul m2 20480Kbit $tc qdisc add dev imq2 parent 1:0x116a sfq $tc filter add dev imq2 parent 1: protocol ip prio 10 u32 match ip src a.b.c.d/32 flowid 1:0x116a $tc class add dev imq3 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul m2 256Kbit $tc qdisc add dev imq3 parent 1:0x116a sfq $tc filter add dev imq3 parent 1: protocol ip prio 10 u32 match ip src a.b.c.d/32 flowid 1:0x116a Some of them has rates half or double than the numbers above, depending how much they pay. Last lines of script are: $tc class change dev imq0 parent 1: classid 1:3 hfsc ls m2 16Kbit ul m2 256Kbit $tc class change dev imq1 parent 1: classid 1:3 hfsc ls m2 8Kbit ul m2 128Kbit $tc class change dev imq2 parent 1: classid 1:3 hfsc ls m2 16Kbit ul m2 256Kbit $tc class change dev imq3 parent 1: classid 1:3 hfsc ls m2 8Kbit ul m2 128Kbit Now, the machine has multiple interfaces, i think 2 gigabit cards with about 5-6 vlans. The ideea of imqs was to shape only traffic that comes or goes from or to some vlans, traffic between clients being unshaped. The machine also did bgp with 1400 prefixes learned and default route, everything running on an 2.6.11 kernel, i think. I''m sure there was a need for lots of tuning, like u32 hash filters .. and many others. With this setup on high traffic things went crazy, like packet loss, real time classes also didn;t get theyr traffic and so on. But ONLY changing from hfsc to a htb, things worked much better, and the important clients got theyr guatanteed bandwidth. Since then .. things changed a lot, u32 hash filters are used, with htb, i get a job somewhere else and so on. :) Any ideas about what i did there are really welcome. Thank You.