OK, here it is. Near perfect bandwidth calculation for ADSL users.
Patch iproute2 with the HTB stuff and then this:
It''s still a hack (as far as I can tell) because we are patching the
rates tables, and hence I think it is only loosly coupled with the
actual calculation of bytes in each bucket.
However, it works very nicely for me! I have only been lightly testing
with downloading stuff (hence packet dropping to slow rates), and I can
set the max download rate to within a few kbyte/sec of the maximum and
still keep near min ping times. I assume that the remaining sliver of
bandwidth is taken up passing packets which arrive in a slight cluster,
and for packets which I later need to drop (since I''m testing on an
incoming interface and dropped packets don''t count for bandwidth used
calcs). However, I seem to be able to get *extremely* close to the max
with this patch
Obviously all the numbers are hard coded, but they should be suitable
for all ATM users. PPoE users will need to do something different (if
someone can supply the details then I will see what we can do to make a
more generic patch and use module params.
Note: That this code will probably affect the policer and CBQ modules in
the same way as HTB, however, I don''t have such a setup, so I
can''t test
effectiveness (or detriment...). Feedback appreciated
Note also that rates in your scripts will now be expressed in terms of
the ATM bandwidth, ie you put in something like the bandwidth you paid
for, but (of course) you get roughly (bw * 48/53) passing through (this
is normal, it''s the overhead of running ATM).
--- iproute2-2.4.7.20020116/tc/tc_core.c 2000-04-16
18:42:55.000000000 +0100
+++ iproute2/tc/tc_core.c 2004-06-18 12:20:39.912974518 +0100
@@ -59,10 +59,19 @@
while ((mtu>>cell_log) > 255)
cell_log++;
}
+
+ // HACK - UK ATM Params
+ int encaps_cell_sz = 53;
+ int encaps_cell_overhead = 5;
+ int encaps_data_sz = encaps_cell_sz - encaps_cell_overhead;
+ int proto_overhead = 10; // PPP Overhead
+
for (i=0; i<256; i++) {
- unsigned sz = (i<<cell_log);
- if (sz < mpu)
- sz = mpu;
+ unsigned sz = ((i+1)<<cell_log)-1;
+ sz = sz + proto_overhead;
+ sz = ( (int)((sz-1)/encaps_data_sz) + 1) * encaps_cell_sz;
+// if (sz < mpu)
+// sz = mpu;
rtab[i] = tc_core_usec2tick(1000000*((double)sz/bps));
}
return cell_log;
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Ed Wildgoose wrote:> OK, here it is. Near perfect bandwidth calculation for ADSL users. > Patch iproute2 with the HTB stuff and then this: > > It''s still a hack (as far as I can tell) because we are patching the > rates tables, and hence I think it is only loosly coupled with the > actual calculation of bytes in each bucket. > However, it works very nicely for me! I have only been lightly testing > with downloading stuff (hence packet dropping to slow rates), and I can > set the max download rate to within a few kbyte/sec of the maximum and > still keep near min ping times. I assume that the remaining sliver of > bandwidth is taken up passing packets which arrive in a slight cluster, > and for packets which I later need to drop (since I''m testing on an > incoming interface and dropped packets don''t count for bandwidth used > calcs). However, I seem to be able to get *extremely* close to the max > with this patch > > Obviously all the numbers are hard coded, but they should be suitable > for all ATM users. PPoE users will need to do something different (if > someone can supply the details then I will see what we can do to make a > more generic patch and use module params. > > Note: That this code will probably affect the policer and CBQ modules in > the same way as HTB, however, I don''t have such a setup, so I can''t test > effectiveness (or detriment...). Feedback appreciated > > Note also that rates in your scripts will now be expressed in terms of > the ATM bandwidth, ie you put in something like the bandwidth you paid > for, but (of course) you get roughly (bw * 48/53) passing through (this > is normal, it''s the overhead of running ATM). > > > --- iproute2-2.4.7.20020116/tc/tc_core.c 2000-04-16 > 18:42:55.000000000 +0100 > +++ iproute2/tc/tc_core.c 2004-06-18 12:20:39.912974518 +0100 > @@ -59,10 +59,19 @@ > while ((mtu>>cell_log) > 255) > cell_log++; > } > + > + // HACK - UK ATM Params > + int encaps_cell_sz = 53; > + int encaps_cell_overhead = 5; > + int encaps_data_sz = encaps_cell_sz - encaps_cell_overhead; > + int proto_overhead = 10; // PPP Overhead > + > for (i=0; i<256; i++) { > - unsigned sz = (i<<cell_log); > - if (sz < mpu) > - sz = mpu; > + unsigned sz = ((i+1)<<cell_log)-1; > + sz = sz + proto_overhead; > + sz = ( (int)((sz-1)/encaps_data_sz) + 1) * encaps_cell_sz; > +// if (sz < mpu) > +// sz = mpu; > rtab[i] = tc_core_usec2tick(1000000*((double)sz/bps)); > } > return cell_log; > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >Nice one Ed :-) After a bit of messing about - Patch would''t apply and I couldn''t see why. Then did it by hand and had to move vars to top of function to get it to compile. I set my uprate to 280kbit in TC = 286720 bit/s I am synced at 288000 - as you probably are, in UK, on what BT call 250/500 and isps call 256/512. I left a bit of slack just to let buffer empty if the odd packet extra slips through. FWIW maxing downlink (576000 for me) will probably mess up - you need to be slower or you don''t get to build up queues and will often be using your isp''s buffer. I''ve been maxing uplink with bt for the last couple of hours and it''s working fine - 100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 15.586/44.518/67.984/13.367 ms It''s just as it should be for my MTU. When I get some time later I''ll start hitting it with lots of small packets aswell. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> After a bit of messing about - > Patch would''t apply and I couldn''t see why. Then did it by hand and > had to move vars to top of function to get it to compile.Hmm, perhaps it got corrupted because of the change in line endings when I pasted it in on a windows machine? Piece of cake to apply manually. If I can get some PPoE settings then I will make a more generic patch and stick it on a website. Can you paste the compile errors and tell me versions of gcc please? I can''t see any probs with that code though? (They will be params passed in later anyway)> I set my uprate to 280kbit in TC = 286720 bit/s I am synced at 288000 > - as you probably are, in UK, on what BT call 250/500 and isps call > 256/512. I left a bit of slack just to let buffer empty if the odd > packet extra slips through. FWIW maxing downlink (576000 for me) will > probably mess up - you need to be slower or you don''t get to build up > queues and will often be using your isp''s buffer. > > I''ve been maxing uplink with bt for the last couple of hours and it''s > working fine -Yes, excellent isn''t it! I tested download rates (bearing in mind the difficulty of controlling those, and could get within a sliver of full bandwidth before the rate rises! I see a two stage rise in ping times. First it stays on 30ms, then it rises to 60ms-90ms, then it queues like crazy. Interesting the kind of three step ramp up. I have a hunch that packets don''t arrive smoothly and queuing occurs at the ISP end (once we get near the limit) even though the average rate is below the max rate...? (ie from time to time you start to see two packets ahead of you instead of just one)> 100 packets transmitted, 100 packets received, 0% packet loss > round-trip min/avg/max/stddev = 15.586/44.518/67.984/13.367 ms > > It''s just as it should be for my MTU.Hmm, what''s your MTU? Those numbers look extremely low for 1500 byte packets (at least if you have a little downlink congestion as well?)> When I get some time later I''ll start hitting it with lots of small > packets aswell.I have a 1meg downstream with 256 upstream, and I turned on bittorrent to try and flog the connection a bit. Upstream was maxed out, but downstream was only half full. However, ping times are 20-110. I think they ought to be only 20-80 ish, and I''m trying to work out why there is some excess queuing (1500 ish mtu). My QOS is based on the the (excellent) script from: http://digriz.org.uk/jdg-qos-script/ Basically, HTB in both directions. RED on the incoming stream (works nicely). Outgoing classifies into 10 buckets, and ACK + ping are definitely going out ok in the top prio bucket, and the rest is going out in the prio 10 bucket.... But still these high pings... Hmm I would be interested to hear if anyone has a CBQ based setup and can tell me if that patch works for them? Or even whether it works on the incoming policer properly? It looks as though this is an adequate way to tackle the problem. The alternative would be to hook into the enqueue side of the qdisk, calculate a new size value then and fix the code to refer to this value from then on. It would be quite invasive though because it modifies kernel headers. I would need someone who understands the scheduler in more detail to guide me as to whether it was neccessary Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Ed Wildgoose wrote:> >> After a bit of messing about - >> Patch would''t apply and I couldn''t see why. Then did it by hand and >> had to move vars to top of function to get it to compile. > > > > Hmm, perhaps it got corrupted because of the change in line endings when > I pasted it in on a windows machine? Piece of cake to apply manually. > If I can get some PPoE settings then I will make a more generic patch > and stick it on a website. > > Can you paste the compile errors and tell me versions of gcc please? I > can''t see any probs with that code though? (They will be params passed > in later anyway)GCC 2.95.3 - Moved the var declarations back after the while ((mtu>>cell_log) > 255) cell_log++; } and again get - tc_core.c: In function `tc_calc_rtable'': tc_core.c:65: parse error before `int'' tc_core.c:77: `proto_overhead'' undeclared (first use in this function) tc_core.c:77: (Each undeclared identifier is reported only once tc_core.c:77: for each function it appears in.) tc_core.c:78: `encaps_data_sz'' undeclared (first use in this function) tc_core.c:78: `encaps_cell_sz'' undeclared (first use in this function) make[1]: *** [tc_core.o] Error 1 Works if I move them to the top of the function - maybe the formatting is messed up (I copy n pasted).> >> I set my uprate to 280kbit in TC = 286720 bit/s I am synced at 288000 >> - as you probably are, in UK, on what BT call 250/500 and isps call >> 256/512. I left a bit of slack just to let buffer empty if the odd >> packet extra slips through. FWIW maxing downlink (576000 for me) will >> probably mess up - you need to be slower or you don''t get to build up >> queues and will often be using your isp''s buffer. >> >> I''ve been maxing uplink with bt for the last couple of hours and it''s >> working fine - > > > > Yes, excellent isn''t it! I tested download rates (bearing in mind the > difficulty of controlling those, and could get within a sliver of full > bandwidth before the rate rises!I still think that you need to throttle back on down rates - when really full you may find that new/bursty connections mean that you loose control of the queue. Of course having twice as much bandwidth always helps.> > I see a two stage rise in ping times. First it stays on 30ms, then it > rises to 60ms-90ms, then it queues like crazy. Interesting the kind of > three step ramp up. I have a hunch that packets don''t arrive smoothly > and queuing occurs at the ISP end (once we get near the limit) even > though the average rate is below the max rate...? (ie from time to time > you start to see two packets ahead of you instead of just one)I am only testing uploads - looking at some more pings, it does appear that they are not quite as random as they were - but apart from the odd double dequeue (in a way I think you can expect this with HTB using quanta/DRR rather than per byte), the max is right. I suspect this is nothing to do with ISP/Teleco end. I could actually do slightly better on my latency, but I am running at 100Hz - and I can tell with pings - they slowly rise then snap down by 10ms. This is nothing to do with TC, I normally run 500 but forgot to set it for this kernel.> >> 100 packets transmitted, 100 packets received, 0% packet loss >> round-trip min/avg/max/stddev = 15.586/44.518/67.984/13.367 ms >> >> It''s just as it should be for my MTU. > > > > Hmm, what''s your MTU? Those numbers look extremely low for 1500 byte > packets (at least if you have a little downlink congestion as well?) >My downlink is clear, I am using 1478 MTU (so I don''t waste half a cell per packet). Just did another hundred to my first hop - 100 packets transmitted, 100 packets received, 0% packet loss round-trip min/avg/max/stddev = 16.661/43.673/70.841/14.634 ms>>When I get some time later I''ll start hitting it with lots of small >> packets aswell.Been going for 7 hours now - I started a couple of games and 60 small pps didn''t cause any problems.> > > > I have a 1meg downstream with 256 upstream, and I turned on bittorrent > to try and flog the connection a bit. Upstream was maxed out, but > downstream was only half full. However, ping times are 20-110. I think > they ought to be only 20-80 ish, and I''m trying to work out why there is > some excess queuing (1500 ish mtu).I worked out once bitrate latency is about 44ms for 1478. Your figures look like you may be sending packets in pairs - or have you already done the tweak which fixed this for me. set HTB_HYSTERESIS 0 in net/sched/sch_htb.c My QOS is based on the the> (excellent) script from: > http://digriz.org.uk/jdg-qos-script/Yea it''s nice - mine is a modified version.> > Basically, HTB in both directions. RED on the incoming stream (works > nicely). Outgoing classifies into 10 buckets, and ACK + ping are > definitely going out ok in the top prio bucket, and the rest is going > out in the prio 10 bucket.... But still these high pings... Hmm >I did find in the worst case I could do better than RED (not much though) and now I do per IP for bulk so Its harder to get the right settings with more than one RED, that may have different bandwidths at any given time. I also reduced the number of classes so each could have a higher rate. Andy.> > I would be interested to hear if anyone has a CBQ based setup and can > tell me if that patch works for them? Or even whether it works on the > incoming policer properly? > > It looks as though this is an adequate way to tackle the problem. The > alternative would be to hook into the enqueue side of the qdisk, > calculate a new size value then and fix the code to refer to this value > from then on. It would be quite invasive though because it modifies > kernel headers. I would need someone who understands the scheduler in > more detail to guide me as to whether it was neccessary > > Ed W > >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> I still think that you need to throttle back on down rates - when > really full you may find that new/bursty connections mean that you > loose control of the queue. Of course having twice as much bandwidth > always helps.Agree that it might be best in practice, but baring the initial surge you can now really crank everything up to the limit. Actually, just starting one or two connections (wget), I didn''t notice more than a slight blip on ping speeds on startup - for single users you could probably leave it quite close to the max (as long as you can tolerate very brief increases in latency)> I am only testing uploads - looking at some more pings, it does appear > that they are not quite as random as they were - but apart from the > odd double dequeue (in a way I think you can expect this with HTB > using quanta/DRR rather than per byte), the max is right. I suspect > this is nothing to do with ISP/Teleco end. I could actually do > slightly better on my latency, but I am running at 100Hz - and I can > tell with pings - they slowly rise then snap down by 10ms. This is > nothing to do with TC, I normally run 500 but forgot to set it for > this kernel.Aha, I switched off the HTB_HYSTERIS option and now my pings are nailed to the floor. I''m also on a 2.6 kernel which I *think* means 1000Hz scheduling by default?> My downlink is clear, I am using 1478 MTU (so I don''t waste half a > cell per packet). Just did another hundred to my first hop -Hmm, quick question. I''m using Nildram as ISP, and I can''t ping at more than 1430 packet sizes and non-fragment bit set. However, according to iptraf on the bridge (behind adsl router), I am receiving 1492 byte packets quite regularly... Since the ADSL router is doing NAT, does this likely imply that it is doing packet reassembly before the bridge sees each packet? I just changed the MTU on the router, but I don''t see any difference in the size of packets. Shouldn''t the router MTU be enough to trigger the initial connection to drop to a smaller MTU? Do I need to apply MTU changes to all host machines here..? (My understanding is not)> Been going for 7 hours now - I started a couple of games and 60 small > pps didn''t cause any problems.Perhaps a small perl script to do something like contact your local SMTP server, setup the connection to start the body of the message, then feed the message in 1 byte pieces (perhaps a random message with crlf every 100 chars or so). This should give lots of mostly empty packets. Perhaps this can be done with the ping flood option as well...?>> Basically, HTB in both directions. RED on the incoming stream (works >> nicely). Outgoing classifies into 10 buckets, and ACK + ping are >> definitely going out ok in the top prio bucket, and the rest is going >> out in the prio 10 bucket.... But still these high pings... Hmm >> > > I did find in the worst case I could do better than RED (not much > though) and now I do per IP for bulk so Its harder to get the right > settings with more than one RED, that may have different bandwidths at > any given time. I also reduced the number of classes so each could > have a higher rate.Can you describe a little more about how you are improving on RED? I would quite like to tweak the incoming to have 3 buckets. One for ACK''s etc, the second for normal incoming and the last for bulk incoming. The idea would be to drop the bulk in preference to the normal incoming. I appreciate this won''t work all that well, but it would be nice to say that my web browsing takes slight preference over a bulk ftp download. Or perhaps that incoming connections to a server machine take big preference over downloads to a local client machine (think office environment). I would be interested on your ideas to achieve that. Thanks Andy. Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Ed Wildgoose <lists@wildgooses.com> writes:> I would be interested to hear if anyone has a CBQ based setup and can > tell me if that patch works for them? Or even whether it works on the > incoming policer properly?Would that work with HFSC ? what about PPPoE ? i didn''t really understand what I have to change if I''m using PPPoE and MTU=1492. TIA -- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
syrius.ml@no-log.org wrote:>Ed Wildgoose <lists@wildgooses.com> writes: > > > >>I would be interested to hear if anyone has a CBQ based setup and can >>tell me if that patch works for them? Or even whether it works on the >>incoming policer properly? >> >> > >Would that work with HFSC ? > >To be honest, I can''t see these bits of code being used in HFSC, so I doubt it. However, if performance isn''t an issue, it would be fairly easy to patch the enqueue function in any of these schedulers to just lie about the size of the packet.>what about PPPoE ? i didn''t really understand what I have to change if >I''m using PPPoE and MTU=1492. > >I''m not really sure how PPPoE is working? Is it as simple as wrapping IP in IP? The patch is trivial though, the first interesting line is this one: + sz = sz + proto_overhead; Here we just lie about the size of the packet and add the overhead PPP causes (this is per original packet) The next interesting line is this: + sz = ( (int)((sz-1)/encaps_data_sz) + 1) * encaps_cell_sz; All we are doing is dividing by the size of the packet that our data is encapsulated in, (perhaps in your case, encaps_data_sz should be 1492 ?). Then we multiply up by the actual size of the packet when it crosses the link "encaps_cell_size". Perhaps in your case this should be 1492 + 40 = 1532 ? The other lines were just trying to make it look a little neater. I will look up how PPPoE works and suggest some new values Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Friday 18 June 2004 07:45, Ed Wildgoose wrote:> OK, here it is. Near perfect bandwidth calculation for ADSL users. > Patch iproute2 with the HTB stuff and then this:Did I miss this the first time around? There''s a patch to sch_htb.c as well, or not? If there is, I haven''t applied it which would explain why the original tc patch did not have any effect for me. I''m eager to try this out as soon as I have some time. Thanks for the patch! <snip> -- Jason Boxman Perl Programmer / *NIX Systems Administrator Shimberg Center for Affordable Housing | University of Florida http://edseek.com/ - Linux and FOSS stuff _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Ed Wildgoose wrote:> >> I still think that you need to throttle back on down rates - when >> really full you may find that new/bursty connections mean that you >> loose control of the queue. Of course having twice as much bandwidth >> always helps. > > > > Agree that it might be best in practice, but baring the initial surge > you can now really crank everything up to the limit. Actually, just > starting one or two connections (wget), I didn''t notice more than a > slight blip on ping speeds on startup - for single users you could > probably leave it quite close to the max (as long as you can tolerate > very brief increases in latency)Latency is what I care about most - 300ms blip is really annoying for me, I suppose what I would see as 300 you would see as 150 which I could live with. If I ran close on ingress it would hurt too much - but two long connections would be OK. It''s new connections and many bursty connections that are the problem. If you let things get really bad and you start having ISP drop packets then you will loose fairness/control aswell. Can you test how big your ISP/BT buffer is - Mine is 600ms I wonder if yours is half or the same.> > >> I am only testing uploads - looking at some more pings, it does appear >> that they are not quite as random as they were - but apart from the >> odd double dequeue (in a way I think you can expect this with HTB >> using quanta/DRR rather than per byte), the max is right. I suspect >> this is nothing to do with ISP/Teleco end. I could actually do >> slightly better on my latency, but I am running at 100Hz - and I can >> tell with pings - they slowly rise then snap down by 10ms. This is >> nothing to do with TC, I normally run 500 but forgot to set it for >> this kernel. > > > > Aha, I switched off the HTB_HYSTERIS option and now my pings are nailed > to the floor. I''m also on a 2.6 kernel which I *think* means 1000Hz > scheduling by default? > >> My downlink is clear, I am using 1478 MTU (so I don''t waste half a >> cell per packet). Just did another hundred to my first hop - > > > > Hmm, quick question. I''m using Nildram as ISP, and I can''t ping at more > than 1430 packet sizes and non-fragment bit set.Hmm ISTR reading of an ISP, maybe nildram, that enforced tweaks - thought they chose 1478 but alot of people like 1430. However, according to> iptraf on the bridge (behind adsl router), I am receiving 1492 byte > packets quite regularly... Since the ADSL router is doing NAT, does > this likely imply that it is doing packet reassembly before the bridge > sees each packet?I know that for me, if I set ppp0 to say 576 and eth0 and LAN machines to 1500, then linux will frag and reconstruct ICMP - but let larger packets through.> > I just changed the MTU on the router, but I don''t see any difference in > the size of packets. Shouldn''t the router MTU be enough to trigger the > initial connection to drop to a smaller MTU?No - well it won''t for me. You can use mss clamping to make tcp comply, but I change LAN machines.>> I did find in the worst case I could do better than RED (not much >> though) and now I do per IP for bulk so Its harder to get the right >> settings with more than one RED, that may have different bandwidths at >> any given time. I also reduced the number of classes so each could >> have a higher rate. > > > > Can you describe a little more about how you are improving on RED? > I would quite like to tweak the incoming to have 3 buckets. One for > ACK''s etc, the second for normal incoming and the last for bulk > incoming. The idea would be to drop the bulk in preference to the > normal incoming. I appreciate this won''t work all that well, but it > would be nice to say that my web browsing takes slight preference over a > bulk ftp download. Or perhaps that incoming connections to a server > machine take big preference over downloads to a local client machine > (think office environment). > > I would be interested on your ideas to achieve that.I''ll answer this later - it''s hard to get on a PC in my house on a Sunday - "Not my turn" apparently :-) Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Jason Boxman wrote:>On Friday 18 June 2004 07:45, Ed Wildgoose wrote: > > >>OK, here it is. Near perfect bandwidth calculation for ADSL users. >>Patch iproute2 with the HTB stuff and then this: >> >> > >Did I miss this the first time around? There''s a patch to sch_htb.c as well, >or not? If there is, I haven''t applied it which would explain why the >original tc patch did not have any effect for me. > >Nope, it''s just a case of patching the TC prog. Make sure it is all up to date with the latest HTB patches first though (not the HTB 2 stuff for example). It works by changing the "rate" calculations which are then later used in the HTB, policer and CBQ modules. I haven''t tested the effect in anything other than HTB though, and it seems to be pretty reliable. I *think* it might be a hack because now "rates" are measured correctly, but bytes per bucket are measured in the original units. However, it seems to be very accurate, so I think it is "good enough" Basically, just lie slightly about the rates that the device can achieve for a given packet size... It''s actually just a one line patch done in a slightly verbose way (you might need to patch manually if line endings got mangled in my email - easy though) Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
syrius.ml@no-log.org wrote:>Ed Wildgoose <lists@wildgooses.com> writes: > > > >>I would be interested to hear if anyone has a CBQ based setup and can >>tell me if that patch works for them? Or even whether it works on the >>incoming policer properly? >> >> > >Would that work with HFSC ? >what about PPPoE ? i didn''t really understand what I have to change if >I''m using PPPoE and MTU=1492. > >After further investigation I think you need the patch a little more like this for PPPoE and the params you gave (Andy do you think this looks OK?) Please let me know if this works for you. (As near as I can tell, there is no PPP overhead(?), and you then basically encapsulate your packets in IP packets of max size 1492, with an 8 byte overhead - Perhaps Andy can correct me as to whether there is still a PPP overhead?) --- iproute2-2.4.7.20020116/tc/tc_core.c 2000-04-16 18:42:55.000000000 +0100 +++ iproute2/tc/tc_core.c 2004-06-18 12:20:39.912974518 +0100 @@ -59,10 +59,19 @@ while ((mtu>>cell_log) > 255) cell_log++; } + + // HACK - UK ATM Params + int encaps_cell_sz = 1492; + int encaps_cell_overhead = 8; + int encaps_data_sz = encaps_cell_sz - encaps_cell_overhead; + int proto_overhead = 0; // PPP Overhead + for (i=0; i<256; i++) { - unsigned sz = (i<<cell_log); - if (sz < mpu) - sz = mpu; + unsigned sz = ((i+1)<<cell_log)-1; + sz = sz + proto_overhead; + sz = ( (int)((sz-1)/encaps_data_sz) + 1) * encaps_cell_sz; +// if (sz < mpu) +// sz = mpu; rtab[i] = tc_core_usec2tick(1000000*((double)sz/bps)); } return cell_log; _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> After further investigation I think you need the patch a little more > like this for PPPoE and the params you gave (Andy do you think this > looks OK?) Please let me know if this works for you.Duh, scratch that. I think the original patch will work for PPPoE (give or take). Can someone help work out how the heck you calculate the real usage of a packet over PPPoE? Reading a bit further I get the impression that you have an 8 byte overhead for PPPoE, then it still travels down an ATM circuit (just like PPPoA)? Can anyone confirm or deny? I have seen some reports that you actually have two ethernet frames wrapped inside each other (which is very inefficient), but I can''t see which is right? If it''s the former, then I think the original patch will work well enough for PPPoE people, it''s 2 bytes out, but that shouldn''t really show up. Can someone please try and report back Thanks Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Sunday 20 June 2004 09:16, Ed Wildgoose wrote:> Jason Boxman wrote: > >On Friday 18 June 2004 07:45, Ed Wildgoose wrote: > >>OK, here it is. Near perfect bandwidth calculation for ADSL users. > >>Patch iproute2 with the HTB stuff and then this: > > > >Did I miss this the first time around? There''s a patch to sch_htb.c as > > well, or not? If there is, I haven''t applied it which would explain why > > the original tc patch did not have any effect for me. > > Nope, it''s just a case of patching the TC prog. Make sure it is all up > to date with the latest HTB patches first though (not the HTB 2 stuff > for example).Interesting.> It works by changing the "rate" calculations which are then later used > in the HTB, policer and CBQ modules. I haven''t tested the effect in > anything other than HTB though, and it seems to be pretty reliable. I > *think* it might be a hack because now "rates" are measured correctly, > but bytes per bucket are measured in the original units. However, it > seems to be very accurate, so I think it is "good enough"So does it just let you be honest about your rated speed, or is there an added performance bonus from being able to specify your true ATM rate? <snip> -- Jason Boxman Perl Programmer / *NIX Systems Administrator Shimberg Center for Affordable Housing | University of Florida http://edseek.com/ - Linux and FOSS stuff _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>So does it just let you be honest about your rated speed, or is there an added >performance bonus from being able to specify your true ATM rate? > >The idea is that your ethernet packets get broken up and encapsulated in 53 byte ATM packets. Each ATM packet is fixed size and has a 5 byte header = 48 bytes of data. Straight off that means that the 512Kbit bandwidth you bought is only good for 512/53 * 48 = 463 Kbit or 58 Kbytes/s. However, unless your packets are exactly a multiple of 48 then you will be wasting bandwidth big time (think about your MTU). Consider small ack packets which will take up two cells, but the second cell is basically empty (bandwidth slashed in half). Unless all your packets are the same size, this is why the current mutliply by some fixed constant approach is not working perfectly with ADSL. So the patch simply works out how many cells you will need and multiplies up by the real size of those cells to work out bw used. It''s also more conservative than before. Previously it was working out an effective rate for 8 byte packets, and using that rate for 8-15 byte packets. Now it works out the rate for 15 byte packets and uses *that* for 8-15 byte packets. It''s probably not perfect, but it''s extremely accurate for me. I can bump up the bandwidth on my interfaces to basically 99% and for stable streams there is no queuing. P2P still seems to cause so many new incoming connections that you need to throttle down the incoming to leave some space for an unexpected incoming rush - but for most users I think it will work nicely Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> After further investigation I think you need the patch a little more > like this for PPPoE and the params you gave (Andy do you think this > looks OK?) Please let me know if this works for you. > > (As near as I can tell, there is no PPP overhead(?), and you then > basically encapsulate your packets in IP packets of max size 1492, with > an 8 byte overhead - Perhaps Andy can correct me as to whether there is > still a PPP overhead?)I wonder if there''s any easy way we can get the ATM/PPPoE support enabled at runtime. I guess a tc option would be needed to enable ADSL (atm) numbers, but the HTB qdisc should be able to detect PPP packets for PPPoE, right ? at least egress on any pppX interface. Are you using this patch just for inbound shaping on pppX, or does it still make sense to use it on an IMQ device, or on a LAN-side ethX device to do the equivalent of ingress shaping ? regards -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Damion de Soto - Software Engineer email: damion@snapgear.com SnapGear - A CyberGuard Company --- ph: +61 7 3435 2809 | Custom Embedded Solutions fax: +61 7 3891 3630 | and Security Appliances web: http://www.snapgear.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ --- Free Embedded Linux Distro at http://www.snapgear.org --- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Monday 21 June 2004 06:04, Ed Wildgoose wrote:> >So does it just let you be honest about your rated speed, or is there an > > added performance bonus from being able to specify your true ATM rate? ><snip>> > So the patch simply works out how many cells you will need and > multiplies up by the real size of those cells to work out bw used.Interesting.> It''s also more conservative than before. Previously it was working out > an effective rate for 8 byte packets, and using that rate for 8-15 byte > packets. Now it works out the rate for 15 byte packets and uses *that* > for 8-15 byte packets.10 byte PPP overhead + 5 byte ATM overhead?> It''s probably not perfect, but it''s extremely accurate for me. I can > bump up the bandwidth on my interfaces to basically 99% and for stable > streams there is no queuing. P2P still seems to cause so many new > incoming connections that you need to throttle down the incoming to > leave some space for an unexpected incoming rush - but for most users I > think it will work nicelyI think we have a winner. I had to futz with the patch some, as Andy did. Once I got it to compile, I kicked my rate up too 256. That didn''t work, but 224 seems stable. I started to `scp` a copy of linux-2.6.3.tar.bz2 since it''s all I could find on short notice to a remote host. Both machines have host CPUs fast enough for 3DES encryption not be the limiting factor. I ping out around 15ms - 70ms to my gateway. That''s about what I was seeing at 160kbit before. I''m running PPPoEoATM here. I don''t know what my actual PPPoE overhead is, but I guess 10 bytes is reasonably close enough. PPPoE is handled by my Westell Wirespeed, which doesn''t provide any useful cell information. At the moment I cannot easily obtain a cell count to determine my actual PPP overhead. Perhaps I need to idle everything and do one of those ''speed tests'' to see what my actual upstream is. Could be it''s really around 224, since I''m not guaranteed 256 by my ISP anyway. 218 packets transmitted, 218 packets received, 0% packet loss round-trip min/avg/max = 11.5/41.3/70.6 ms I have attached a unified `diff` with both the HTB `tc` patch and Ed Wildgoose''s new PPPoA overhead patch as a unified patch. -- Jason Boxman Perl Programmer / *NIX Systems Administrator Shimberg Center for Affordable Housing | University of Florida http://edseek.com/ - Linux and FOSS stuff
Damion de Soto wrote:>> After further investigation I think you need the patch a little more >> like this for PPPoE and the params you gave (Andy do you think this >> looks OK?) Please let me know if this works for you. >> >> (As near as I can tell, there is no PPP overhead(?), and you then >> basically encapsulate your packets in IP packets of max size 1492, >> with an 8 byte overhead - Perhaps Andy can correct me as to whether >> there is still a PPP overhead?) > > I wonder if there''s any easy way we can get the ATM/PPPoE support enabled > at runtime.Yeah, I was thinking of adding a module param in the final version, something like "encaps PPPoA" or similar> I guess a tc option would be needed to enable ADSL (atm) numbers, > but the HTB qdisc should be able to detect PPP packets for PPPoE, right ? > at least egress on any pppX interface.Now that''s a very interesting question. The size is passed in as the size of the packet that the kernel sees (including headers). We mangle that to get the size sent over the ATM link. However, I have ethernet on both sides in my setup. I wonder what the kernel sees if one interface is PPP? Is it already encapsulated in PPP before QOS gets it, or not?> Are you using this patch just for inbound shaping on pppX, or does it > still make sense to use it on an IMQ device, or on a LAN-side ethX > device to do the equivalent of ingress shaping ?Use it for both. Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>I''m running PPPoEoATM here. I don''t know what my actual PPPoE overhead is, >but I guess 10 bytes is reasonably close enough. PPPoE is handled by my >Westell Wirespeed, which doesn''t provide any useful cell information. At the >moment I cannot easily obtain a cell count to determine my actual PPP >overhead. > >Try bumping the protocol_overhead up to 16 for PPPoE (from 10). You should also make sure that your MTU is lower than that required by your PPPoE provider or else you will get ethernet packet fragmentation and I doubt we want to extend the patch to cover those situations anyway. The patch should actually have most benefit when you are doing transfers with smaller packets. I think with larger constant streams like the one you tested, there will be little difference between bumping up the interface speed with the patch or leaving it all as it was (at the end of the day we are mostly just shifting the calculation of interface speed somewhere else).>Perhaps I need to idle everything and do one of those ''speed tests'' to see >what my actual upstream is. Could be it''s really around 224, since I''m not >guaranteed 256 by my ISP anyway. > >Your upstream will be 256Kbits of ATM bandwidth. This consists of 53 byte packets with 48 bytes of data. So you already only have 256 * 48/53 of real bandwidth. We then have to take off PPP headers and PPPoE headers. We are obviously still a few bytes out with this patch or else you should be able to crank up the speed to 250 ish and still see your ping speeds stay low. I will investigate further Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Tuesday 22 June 2004 06:00, Ed Wildgoose wrote:> >I''m running PPPoEoATM here. I don''t know what my actual PPPoE overhead > > is, but I guess 10 bytes is reasonably close enough. PPPoE is handled by > > my Westell Wirespeed, which doesn''t provide any useful cell information. > > At the moment I cannot easily obtain a cell count to determine my actual > > PPP overhead. > > Try bumping the protocol_overhead up to 16 for PPPoE (from 10). You > should also make sure that your MTU is lower than that required by your > PPPoE provider or else you will get ethernet packet fragmentation and I > doubt we want to extend the patch to cover those situations anyway.The largest MTU I could use is 1492. I have the Westell Wirespeed handling PPPoE, so I speak through eth0 locally. Until recently that was fine, but now I need to use 1492 instead of 1500 on eth0 due to strange SSH hangs that haven''t happened in a year at 1500 with the same configuration. It''s odd. All other machines are using 1500 without incident.> The patch should actually have most benefit when you are doing transfers > with smaller packets. I think with larger constant streams like the one > you tested, there will be little difference between bumping up the > interface speed with the patch or leaving it all as it was (at the end > of the day we are mostly just shifting the calculation of interface > speed somewhere else).Without the patch, if I set my rate to 256 * 0.8, I die. The connection is not completely unusable, but gaming is extremely laggy and Web traffic is noticeably laggy, although pages still load with about 2s (versus a few hundred ms without the patch at 160kbit). With the patch I can set it to 224, so there''s obviously a large improvement even with mostly large TCP packets going out doing a bulk `scp` copy.> >Perhaps I need to idle everything and do one of those ''speed tests'' to see > >what my actual upstream is. Could be it''s really around 224, since I''m > > not guaranteed 256 by my ISP anyway. > > Your upstream will be 256Kbits of ATM bandwidth. This consists of 53 > byte packets with 48 bytes of data. So you already only have 256 * > 48/53 of real bandwidth. We then have to take off PPP headers and PPPoE > headers.That''s the maximum promised speed from my ISP. In reality, of course, line conditions might result in my true speed not being that high, before accounting for overhead and things.> We are obviously still a few bytes out with this patch or else you > should be able to crank up the speed to 250 ish and still see your ping > speeds stay low. I will investigate furtherOkay, if you''re sure it''s not just my line having a true upstream less than the consumer rated speed it was assigned. ;)> Ed W >-- Jason Boxman Perl Programmer / *NIX Systems Administrator Shimberg Center for Affordable Housing | University of Florida http://edseek.com/ - Linux and FOSS stuff _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Tuesday 22 June 2004 06:00, Ed Wildgoose wrote: <snip>> Your upstream will be 256Kbits of ATM bandwidth. This consists of 53 > byte packets with 48 bytes of data. So you already only have 256 * > 48/53 of real bandwidth. We then have to take off PPP headers and PPPoE > headers.<snip>> We are obviously still a few bytes out with this patch or else you > should be able to crank up the speed to 250 ish and still see your ping > speeds stay low. I will investigate furtherI guess I''m still missing the point. Isn''t my _effective_ bandwidth still entirely dependent on how big my PPP encapsulated packets are just prior to being broken into streams of 53 byte ATM cells? With the patch does HTB behave appropriately if it gets hit with mostly ~ 100 byte packets where you''d encounter around 30% of your ATM rate lost due to the overhead of null byte padding?> Ed W_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
hi ! this patch can work good with last tc release ? which performance haves with rp-pppoe ? bests andres. -> -----Mensaje original----- -> De: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]En -> nombre de Jason Boxman -> Enviado el: Martes, 22 de Junio de 2004 02:17 p.m. -> Para: lartc@mailman.ds9a.nl -> Asunto: Re: [LARTC] patch: HTB update for ADSL users -> -> -> On Tuesday 22 June 2004 06:00, Ed Wildgoose wrote: -> > >I''m running PPPoEoATM here. I don''t know what my actual -> PPPoE overhead -> > > is, but I guess 10 bytes is reasonably close enough. PPPoE -> is handled by -> > > my Westell Wirespeed, which doesn''t provide any useful cell -> information. -> > > At the moment I cannot easily obtain a cell count to -> determine my actual -> > > PPP overhead. -> > -> > Try bumping the protocol_overhead up to 16 for PPPoE (from 10). You -> > should also make sure that your MTU is lower than that required by your -> > PPPoE provider or else you will get ethernet packet fragmentation and I -> > doubt we want to extend the patch to cover those situations anyway. -> -> The largest MTU I could use is 1492. I have the Westell -> Wirespeed handling -> PPPoE, so I speak through eth0 locally. Until recently that was -> fine, but -> now I need to use 1492 instead of 1500 on eth0 due to strange -> SSH hangs that -> haven''t happened in a year at 1500 with the same configuration. -> It''s odd. -> All other machines are using 1500 without incident. -> -> > The patch should actually have most benefit when you are doing -> transfers -> > with smaller packets. I think with larger constant streams -> like the one -> > you tested, there will be little difference between bumping up the -> > interface speed with the patch or leaving it all as it was (at the end -> > of the day we are mostly just shifting the calculation of interface -> > speed somewhere else). -> -> Without the patch, if I set my rate to 256 * 0.8, I die. The -> connection is -> not completely unusable, but gaming is extremely laggy and Web -> traffic is -> noticeably laggy, although pages still load with about 2s (versus a few -> hundred ms without the patch at 160kbit). With the patch I can -> set it to -> 224, so there''s obviously a large improvement even with mostly large TCP -> packets going out doing a bulk `scp` copy. -> -> > >Perhaps I need to idle everything and do one of those ''speed -> tests'' to see -> > >what my actual upstream is. Could be it''s really around 224, -> since I''m -> > > not guaranteed 256 by my ISP anyway. -> > -> > Your upstream will be 256Kbits of ATM bandwidth. This consists of 53 -> > byte packets with 48 bytes of data. So you already only have 256 * -> > 48/53 of real bandwidth. We then have to take off PPP headers -> and PPPoE -> > headers. -> -> That''s the maximum promised speed from my ISP. In reality, of -> course, line -> conditions might result in my true speed not being that high, before -> accounting for overhead and things. -> -> > We are obviously still a few bytes out with this patch or else you -> > should be able to crank up the speed to 250 ish and still see your ping -> > speeds stay low. I will investigate further -> -> Okay, if you''re sure it''s not just my line having a true -> upstream less than -> the consumer rated speed it was assigned. ;) -> -> > Ed W -> > -> -> -- -> -> Jason Boxman -> Perl Programmer / *NIX Systems Administrator -> Shimberg Center for Affordable Housing | University of Florida -> http://edseek.com/ - Linux and FOSS stuff -> -> _______________________________________________ -> LARTC mailing list / LARTC@mailman.ds9a.nl -> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
ThE LinuX_KiD wrote:>hi ! > >this patch can work good with last tc release ? >which performance haves with rp-pppoe ? > >I guess so. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/