Hi,
One of my customer needs per flow rate control, so I write one.
The code I post here is not finished, but it seems to work as expected.
The kernel patch is agains kernel 2.6.11, the iproute2 patch is against
iproute2-2.6.11-050314.
I write the code in a hurry to meet deadline. There are many other things
to do ahead for me. The code is written in 2 days (including read other
queue''s code) and tested for a while to find obvious mistake.
Don''t be
suprised when you find many many bugs.
The test scenario is like this
www server <- [ eth0 eth1 ] -> www clients
The attached t.sh is used to generate test rules. Clients download a
big ISO file from www server, so flows'' rate can be estimated by view
progress. However I use wget to test the speed, so the speed is
accumulated, not current.
The problems I know:
1. The rtnetlink related code is quick hack. I am not familiar with
rtnetlink, so I look at other queue''s code and use the simplest one.
2. perflow queue has no stats code. It will be added later.
3. I don''t know what is the dump() method ''s purpose, so I
didn''t write
dump() method. I will add it later when I know what it is for and how to
write rtnetlink code.
Any feedback is welcome. And test it if you can :)
PS: the code is licensed under GPL. If it is acceptable by upstream, it
will be submitted.
--
lark
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Wang Jian wrote:> Hi, > > One of my customer needs per flow rate control, so I write one. > > The code I post here is not finished, but it seems to work as expected. > > The kernel patch is agains kernel 2.6.11, the iproute2 patch is against > iproute2-2.6.11-050314. > > I write the code in a hurry to meet deadline. There are many other things > to do ahead for me. The code is written in 2 days (including read other > queue''s code) and tested for a while to find obvious mistake. Don''t be > suprised when you find many many bugs.It looks quite clean, but couldn''t the same be achieved with just providing per-flow fairness and leaving the rate-limiting to an upper qdisc like HTB or HFSC? Regards Patrick
Hi Patrick McHardy, HTB + SQF can only achieve part of funcionality. Per flow rate control means per flow bandwidth assurance + bandwidth constraint. When we use HTB + SQF, 1. We can''t achieve bandwidth assurance when flow count is higher than expected; this often means we fail to meet the quality requirement. 2. We can''t enforce bandwidth constraint when flow count is very low; this often means waste of bandwidth. On Mon, 04 Apr 2005 10:51:15 +0200, Patrick McHardy <kaber@trash.net> wrote:> Wang Jian wrote: > > Hi, > > > > One of my customer needs per flow rate control, so I write one. > > > > The code I post here is not finished, but it seems to work as expected. > > > > The kernel patch is agains kernel 2.6.11, the iproute2 patch is against > > iproute2-2.6.11-050314. > > > > I write the code in a hurry to meet deadline. There are many other things > > to do ahead for me. The code is written in 2 days (including read other > > queue''s code) and tested for a while to find obvious mistake. Don''t be > > suprised when you find many many bugs. > > It looks quite clean, but couldn''t the same be achieved with just > providing per-flow fairness and leaving the rate-limiting to an > upper qdisc like HTB or HFSC? > > Regards > Patrick-- lark
Wang Jian wrote:> Hi, > > One of my customer needs per flow rate control, so I write one. > > The code I post here is not finished, but it seems to work as expected. > > The kernel patch is agains kernel 2.6.11, the iproute2 patch is against > iproute2-2.6.11-050314. > > I write the code in a hurry to meet deadline. There are many other things > to do ahead for me. The code is written in 2 days (including read other > queue''s code) and tested for a while to find obvious mistake. Don''t be > suprised when you find many many bugs.Wow - I wish I could write that in 2 days :-)> > The test scenario is like this > > www server <- [ eth0 eth1 ] -> www clients > > The attached t.sh is used to generate test rules. Clients download a > big ISO file from www server, so flows'' rate can be estimated by view > progress. However I use wget to test the speed, so the speed is > accumulated, not current.What if the client uses a download accelerator and has 12 connections (I suppose server could limit this - but if client is behind nat you may hurt others - which is what sfq does now AIUI, because it doesn''t hash on dst port.)> > The problems I know: > > 1. The rtnetlink related code is quick hack. I am not familiar with > rtnetlink, so I look at other queue''s code and use the simplest one. > > 2. perflow queue has no stats code. It will be added later. > > 3. I don''t know what is the dump() method ''s purpose, so I didn''t write > dump() method. I will add it later when I know what it is for and how to > write rtnetlink code. > > Any feedback is welcome. And test it if you can :) > > PS: the code is licensed under GPL. If it is acceptable by upstream, it > will be submitted.Having per flow without the drawbacks of sfq is really cool, but I agree with Patrick about letting htb/hfsc limit. You say in the code - "You should use HTB or other classful qdisc to enclose this qdisc" So if you do that (unless you meant should not) then you can''t guarentee per flow rate anyway without knowing the number of flows, unless you can set rate so high that max flows x flow rate < htb rate. I think you can still limit per flow ceil if you use htb/hfsc to ratelimit. I suppose you are solving a different problem with this than I normally shape for ie. you have loads of bandwidth and I have hardly any. It still could be something really usefull for me though, as I suspect it wouldn''t be too hard to add lots of features/switches which (e)sfq doesn''t have like - Per flow queue length limit - and more choice than just tail drop (I am thinking of me shaping from wrong and of link here - server with BIC tcp is horrible with tail drop - others are not as bad). For people who use esfq for hundreds of users, you could still do fairness of tcp flows within fairness per user address. Requeue properly which (e)sfq doesn''t. Andy.
Hi Andy Furniss, On Mon, 04 Apr 2005 12:42:21 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> > Wow - I wish I could write that in 2 days :-) >I think if you have a deadline, then you can do that :-)> > > > The test scenario is like this > > > > www server <- [ eth0 eth1 ] -> www clients > > > > The attached t.sh is used to generate test rules. Clients download a > > big ISO file from www server, so flows'' rate can be estimated by view > > progress. However I use wget to test the speed, so the speed is > > accumulated, not current. > > What if the client uses a download accelerator and has 12 connections (I > suppose server could limit this - but if client is behind nat you may > hurt others - which is what sfq does now AIUI, because it doesn''t hash > on dst port.) >The test scenario is not the real scenario under which it will be used. I just use this for test, because it is simple. This per flow control is good when used for VoIP (Voice and Video).> > > > The problems I know: > > > > 1. The rtnetlink related code is quick hack. I am not familiar with > > rtnetlink, so I look at other queue''s code and use the simplest one. > > > > 2. perflow queue has no stats code. It will be added later. > > > > 3. I don''t know what is the dump() method ''s purpose, so I didn''t write > > dump() method. I will add it later when I know what it is for and how to > > write rtnetlink code. > > > > Any feedback is welcome. And test it if you can :) > > > > PS: the code is licensed under GPL. If it is acceptable by upstream, it > > will be submitted. > > Having per flow without the drawbacks of sfq is really cool, but I agree > with Patrick about letting htb/hfsc limit. You say in the code - > > "You should use HTB or other classful qdisc to enclose this qdisc" > > So if you do that (unless you meant should not) then you can''t guarentee > per flow rate anyway without knowing the number of flows, unless you can > set rate so high that max flows x flow rate < htb rate.HTB is providing guaranteed bandwidth. per flow control has its own bandwidth limit ( rate * 1.05 * limit ).> > I think you can still limit per flow ceil if you use htb/hfsc to ratelimit. > > I suppose you are solving a different problem with this than I normally > shape for ie. you have loads of bandwidth and I have hardly any. >Let me explain the idea more clear. For example, you may have 50 streams. These stream can work perfectly at 10kbps - 15kbps. With HTB + SFQ, you should give 50*15 guaranteed. but then, if only one stream is using this, it can use up to 50*15 guaranteed. You have risk of waste 49*15 on it. In another hand, if your have more than 50 streams, say, 80 streams. With perfect fairness, every stream can get less than 10kbps. The quality is not met however, no one is satisfied with fairness. So, you have risk of waste and still you don''t have guarantee. With per flow rate control, you can give a guaranteed 12*65, and set per flow control to rate=12,ceil=15,limit=60. When you have only a few streams, you don''t worry that you waste bandwidth. If more than 60 streams occurs, the first 60 streams still works fine. Fairness is good, but sometimes, fairness means everyone hurts. If you have more than enough bandwidth, you can use fairness to get good QoS. But it is not the case when bandwidth is not so enough. BTW: Are there any good document for HFSC? I don''t even know how it works :( Maybe it''s can be used to achieve per flow control.> It still could be something really usefull for me though, as I suspect > it wouldn''t be too hard to add lots of features/switches which (e)sfq > doesn''t have like - > > Per flow queue length limit - and more choice than just tail drop (I am > thinking of me shaping from wrong and of link here - server with BIC tcp > is horrible with tail drop - others are not as bad). > > For people who use esfq for hundreds of users, you could still do > fairness of tcp flows within fairness per user address. > > Requeue properly which (e)sfq doesn''t.Because this per-flow queue is new, you can add things useful to it.> > Andy.-- lark
Hi Andy Furniss, I just tried HTB+SFQ. I replace ''perflow ...'' in t.sh with ''sfq''. The test result is very bad. The speed is not stable, and speed variation is too large when considering fairness. The HTB is rate=80kbps,ceil=80kbps. I use 7 streams to test. Streams''s speed vary from 3.4kbps to 28.7kbps. The test last about 10 minutes, and the speeds don''t like to converge. Maybe the fairness is achived in long run, but it hurts applications that need bandwidth guarantee. On Mon, 04 Apr 2005 12:42:21 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> Wang Jian wrote: > > Hi, > > > > One of my customer needs per flow rate control, so I write one. > > > > The code I post here is not finished, but it seems to work as expected. > > > > The kernel patch is agains kernel 2.6.11, the iproute2 patch is against > > iproute2-2.6.11-050314. > > > > I write the code in a hurry to meet deadline. There are many other things > > to do ahead for me. The code is written in 2 days (including read other > > queue''s code) and tested for a while to find obvious mistake. Don''t be > > suprised when you find many many bugs. > > Wow - I wish I could write that in 2 days :-) > > > > > The test scenario is like this > > > > www server <- [ eth0 eth1 ] -> www clients > > > > The attached t.sh is used to generate test rules. Clients download a > > big ISO file from www server, so flows'' rate can be estimated by view > > progress. However I use wget to test the speed, so the speed is > > accumulated, not current. > > What if the client uses a download accelerator and has 12 connections (I > suppose server could limit this - but if client is behind nat you may > hurt others - which is what sfq does now AIUI, because it doesn''t hash > on dst port.) > > > > > > The problems I know: > > > > 1. The rtnetlink related code is quick hack. I am not familiar with > > rtnetlink, so I look at other queue''s code and use the simplest one. > > > > 2. perflow queue has no stats code. It will be added later. > > > > 3. I don''t know what is the dump() method ''s purpose, so I didn''t write > > dump() method. I will add it later when I know what it is for and how to > > write rtnetlink code. > > > > Any feedback is welcome. And test it if you can :) > > > > PS: the code is licensed under GPL. If it is acceptable by upstream, it > > will be submitted. > > Having per flow without the drawbacks of sfq is really cool, but I agree > with Patrick about letting htb/hfsc limit. You say in the code - > > "You should use HTB or other classful qdisc to enclose this qdisc" > > So if you do that (unless you meant should not) then you can''t guarentee > per flow rate anyway without knowing the number of flows, unless you can > set rate so high that max flows x flow rate < htb rate. > > I think you can still limit per flow ceil if you use htb/hfsc to ratelimit. > > I suppose you are solving a different problem with this than I normally > shape for ie. you have loads of bandwidth and I have hardly any. > > It still could be something really usefull for me though, as I suspect > it wouldn''t be too hard to add lots of features/switches which (e)sfq > doesn''t have like - > > Per flow queue length limit - and more choice than just tail drop (I am > thinking of me shaping from wrong and of link here - server with BIC tcp > is horrible with tail drop - others are not as bad). > > For people who use esfq for hundreds of users, you could still do > fairness of tcp flows within fairness per user address. > > Requeue properly which (e)sfq doesn''t. > > > Andy.-- lark
Wang Jian wrote:> Hi Andy Furniss, > > I just tried HTB+SFQ. I replace ''perflow ...'' in t.sh with ''sfq''. > > The test result is very bad. The speed is not stable, and speed > variation is too large when considering fairness. > > The HTB is rate=80kbps,ceil=80kbps. I use 7 streams to test. Streams''s > speed vary from 3.4kbps to 28.7kbps. The test last about 10 minutes, and > the speeds don''t like to converge. > > Maybe the fairness is achived in long run, but it hurts applications > that need bandwidth guarantee.Yes - I can make sfq look bad in tests, if the only difference is dst port then it just doesn''t work and if the ip addresses are sequential it''s not too good. In practice I use esfq as you can use more hash buckets - but perturb is horrable for the packet reordering. I think perflow is going to be far better for me - just that having low bandwidth means I would never send interactive to sfq anyway and only use it for bulk whose rate is controlled by htb per user and is quite variable - so for me just letting htb do rate would be fine. Andy.
Wang Jian wrote:> This per flow control is good when used for VoIP (Voice and Video).Ahh yes - your needs are totally different to mine - with low bandwidth I just have to seperate interactive from bulk and use sfq for bulk only as if queuing interactive would mean I have run out of bandwidth anyway.> > Let me explain the idea more clear. > > For example, you may have 50 streams. These stream can work perfectly at > 10kbps - 15kbps. > > With HTB + SFQ, you should give 50*15 guaranteed. but then, if only one > stream is using this, it can use up to 50*15 guaranteed. You have risk > of waste 49*15 on it. > > In another hand, if your have more than 50 streams, say, 80 streams. > With perfect fairness, every stream can get less than 10kbps. The > quality is not met however, no one is satisfied with fairness. > > So, you have risk of waste and still you don''t have guarantee. > > With per flow rate control, you can give a guaranteed 12*65, and set per > flow control to rate=12,ceil=15,limit=60. When you have only a few > streams, you don''t worry that you waste bandwidth. If more than 60 > streams occurs, the first 60 streams still works fine. > > Fairness is good, but sometimes, fairness means everyone hurts. If you > have more than enough bandwidth, you can use fairness to get good QoS. > But it is not the case when bandwidth is not so enough.I can see now why you do it this way.> > BTW: Are there any good document for HFSC? I don''t even know how it > works :( Maybe it''s can be used to achieve per flow control.No not really many docs and you can''t really do per flow as such - more per user/class. I haven''t played with it enough yet, but the strength is being able to seperate interactive from bulk and still limit per user/class , without making each users interactive wait for other users bulk - at slow rates the bitrate latency of single packets can add up enough to messup interactive.> > Because this per-flow queue is new, you can add things useful to it.It does look good :-) I''ll test when I get time. Andy.
Hi Andy Furniss, On Mon, 04 Apr 2005 16:23:30 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> > > > > Because this per-flow queue is new, you can add things useful to it. > > It does look good :-) I''ll test when I get time. >The attached is the latest. The last one doesn''t sync time: queue has a variable time slot length; every flow has it own ticks. This new patch against 2.6.11 sync queue and flows'' time. Every new flow has it jiffies set to q->jiffies and use that as start. As q->jiffies and flow->jiffies increament in HZ step, time is synced. This will improved accuracy. But HZ is too long for token calculation. Sometimes, one of flow borrows too much and get no enough penalty, so another flow hurts. But anyway, per flow queue provides better fairness in my test, either in short time period or long time period. Looking forward to your feedback :) -- lark _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Wang Jian wrote:> Hi Andy Furniss, > > > On Mon, 04 Apr 2005 16:23:30 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote: > > >>>Because this per-flow queue is new, you can add things useful to it. >> >>It does look good :-) I''ll test when I get time. >> > > > The attached is the latest. The last one doesn''t sync time: queue has a > variable time slot length; every flow has it own ticks. > > This new patch against 2.6.11 sync queue and flows'' time. Every new flow > has it jiffies set to q->jiffies and use that as start. As q->jiffies > and flow->jiffies increament in HZ step, time is synced. This will > improved accuracy. > > But HZ is too long for token calculation. Sometimes, one of flow borrows > too much and get no enough penalty, so another flow hurts. But anyway, > per flow queue provides better fairness in my test, either in > short time period or long time period. > > Looking forward to your feedback :)It works OK for me - though I would really need it to be variable rate to use really - but as you say it''s designed for your needs. I noticed that it drops icmp so you need to be careful about what you send to it. If you limit connections and use them all up then alive but not always active connections will get locked out - there is a netfilter connection limit already. As you say above it''s not always fair - I didn''t test that much it seemed OK apart from if htb limited it ie. htb rate higher than sum of rates but less than sum of ceils made it unfair to a flow with smaller packet size. Andy.
Hi Andy Furniss, On Tue, 05 Apr 2005 23:40:54 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> > > > Looking forward to your feedback :) > > It works OK for me - though I would really need it to be variable rate > to use really - but as you say it''s designed for your needs. > > I noticed that it drops icmp so you need to be careful about what you > send to it.I plan to optionally reclassify unhandled traffic to another class if specified. So a default class may handle it.> > If you limit connections and use them all up then alive but not always > active connections will get locked out - there is a netfilter connection > limit already. > > As you say above it''s not always fair - I didn''t test that much it > seemed OK apart from if htb limited it ie. > > htb rate higher than sum of rates but less than sum of ceils made it > unfair to a flow with smaller packet size.Yes. I also think that low rate or small packet size stream will have problem. I didn''t test that case yet. I read back your post and I think the best solution for you is use HTB + PRIO. Let interactive but low rate traffic have highest priority, and let bulk transfer have lowest priority and constrain them using HTB. TCP itself has some fairness: slower stream get faster, and faster stream get slower. The sliding window is for this. -- lark
Wang Jian wrote:> > I read back your post and I think the best solution for you is use HTB + > PRIO.I sort of have htb setup like prio but it''s more flexable.> > Let interactive but low rate traffic have highest priority, and let bulk > transfer have lowest priority and constrain them using HTB. > > TCP itself has some fairness: slower stream get faster, and faster > stream get slower. The sliding window is for this.TCP can be very unfair in some cases - different window sizes/scale on off and 56k vs broadband peer. I am rebuilding stuff on my gateway at the moment and noticed the iproute patch doesn''t compile with gcc 2.95.3 it''s fine with 3.3. q_perflow.c: In function `perflow_print_opt'': q_perflow.c:141: parse error before `char'' q_perflow.c:142: `b1'' undeclared (first use in this function) q_perflow.c:142: (Each undeclared identifier is reported only once q_perflow.c:142: for each function it appears in.) make[1]: *** [q_perflow.o] Error 1 Andy.
Hi Andy Furniss, On Wed, 06 Apr 2005 13:29:56 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> > > > I read back your post and I think the best solution for you is use HTB + > > PRIO. > > I sort of have htb setup like prio but it''s more flexable.I am glad to hear that :)> > > > Let interactive but low rate traffic have highest priority, and let bulk > > transfer have lowest priority and constrain them using HTB. > > > > TCP itself has some fairness: slower stream get faster, and faster > > stream get slower. The sliding window is for this. > > TCP can be very unfair in some cases - different window sizes/scale on > off and 56k vs broadband peer. >Yes. This unfairness is generally a good thing (but not always). It is in favour of tcp connection in the fast/wide path, so bandwidth can be used "efficiently" :)> I am rebuilding stuff on my gateway at the moment and noticed the > iproute patch doesn''t compile with gcc 2.95.3 it''s fine with 3.3. > > q_perflow.c: In function `perflow_print_opt'': > q_perflow.c:141: parse error before `char'' > q_perflow.c:142: `b1'' undeclared (first use in this function) > q_perflow.c:142: (Each undeclared identifier is reported only once > q_perflow.c:142: for each function it appears in.) > make[1]: *** [q_perflow.o] Error 1 >This is due to the included <linux/jhash.h>. Regards -- lark