Hey list I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a moment. At least this is my best guess at the problem :) There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most. Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed? My efforts so far can be seen here: http://tc.schmakk.dk/betashaper -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
It is possible to slow down tcp start and it realy helped for me to get better pings, but expense is too high, so I think it do not work as it should, if you start 5 coonections at once you receive 5 1.5kbyte packets, what fills your queue at isp side the more connections you start at once the worse delay will be. this can be partialy fixed if you give some reserve using about 80-90% of link like everybody usualy do now I am working on new driver which could help to solve this, the only way probably is predict new coonecions and reduce speed of exsisting ones before new ones start. ----- Original Message ----- From: "Patrick Petersen" <lartc@schmakk.dk> To: <lartc@mailman.ds9a.nl> Sent: Thursday, April 15, 2004 1:26 PM Subject: [LARTC] Making tcp start transfers slow> Hey list > I have almost gotten my shaping setup up and running as planned. The > last barrier seems to be tcp overshooting availible bandwidth when its > starting a transfer, and thereby bursting the line, so ping rises for a > moment. At least this is my best guess at the problem :) > There is a possibility that its just plain old traffic being bursty for > some reason.. I am using bittorrent to test this, as it seems to be what > stresses the line the most. > Would it be possible to lower the default window size, and thereby > making tcp start up slower, or would this just lower the overall speed? > My efforts so far can be seen here: > http://tc.schmakk.dk/betashaper > -- > Patrick Petersen <lartc@schmakk.dk> > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Patrick Petersen wrote:> Hey list > I have almost gotten my shaping setup up and running as planned. The > last barrier seems to be tcp overshooting availible bandwidth when its > starting a transfer, and thereby bursting the line, so ping rises for a > moment. At least this is my best guess at the problem :)I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth. It works well in the case where the link is empty apart from a gamestream and someone is browsing "heavy .jpg" type web paged. It also helps a bit if there is other traffic - but if there are enough tcp connections on the go there will be higher latency bursts caused by new connections as HTB can''t throttle until it''s a bit too late.> There is a possibility that its just plain old traffic being bursty for > some reason.. I am using bittorrent to test this, as it seems to be what > stresses the line the most.Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it''s own algorithm on allready open tcps - so existing connections may go back into slowstart.> Would it be possible to lower the default window size, and thereby > making tcp start up slower, or would this just lower the overall speed?It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).> My efforts so far can be seen here: > http://tc.schmakk.dk/betashaperHad a quick look - Some thoughts: I don''t think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic? Be carefull about priorotising acks - don''t all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-) For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
I just had one idea, about this. what if make iptables module which will make something like enlarged copy of syn packet and send it back to the sender? (another option would be to kill 1 or 2 ack packets for one syn packet this whould force server to reduce speed) that way htb could count upcoming packet and prepare by reducing other connetions speed? of course that synthetic packet will have higest possible priority since it supposed to be appear in future so cant be shaped anyway I will try to add this functionality to my imq module next week probably. ------------------------ connbytes solution is not good for this, it slows down small picture loading in web pages very much, and big downloads get even more "unused" bandwitch. so effect is not good. expecialy that looks bad on network, when pages become incredibly slow, but big downloads fast. ----- Original Message ----- From: "Andy Furniss" <andy.furniss@dsl.pipex.com> To: "Patrick Petersen" <lartc@schmakk.dk> Cc: <lartc@mailman.ds9a.nl> Sent: Sunday, April 18, 2004 2:13 AM Subject: Re: [LARTC] Making tcp start transfers slow> Patrick Petersen wrote: > > Hey list > > I have almost gotten my shaping setup up and running as > planned. The > > last barrier seems to be tcp overshooting availible bandwidth > when its > > starting a transfer, and thereby bursting the line, so ping > rises for a > > moment. At least this is my best guess at the problem > :) > > I sortof workaround this using the connbytes netfilter patch to make the > first 80k of new connections go into a short queue limited to 1/3 - 1/2 > of my downstream bandwidth. It works well in the case where the link is > empty apart from a gamestream and someone is browsing ''"''heavy .jpg''"''type> web paged. It also helps a bit if there is other traffic - but if there > are enough tcp connections on the go there will be higher latency bursts > caused by new connections as HTB can''t throttle until it''s a bit too late. > > > > There is a possibility that its just plain old traffic being > bursty for > > some reason.. I am using bittorrent to test this, as it seems > to be what > > stresses the line the most. > > Ahh bittorrent - this is a special case. It uses full duplex tcp - so > may break some upstream shapers, you can assume that a fair number of > your peers have flooded modem buffers - so there could be quite a few > ''"''unstoppable''"'' packets headed your way. The worse thing about it,though> is that it runs it''s own algorithm on allready open tcps - so existing > connections may go back into slowstart. > > > Would it be possible to lower the default window size, and > thereby > > making tcp start up slower, or would this just lower the > overall speed? > > It could help, but may also give you less of a share of a given > uploaders bandwidth. Reducing MTU may also help (if you run BT on linux > it may reduce rwin for you aswell). > > > My efforts so far can be seen here: > > http://tc.schmakk.dk/betashaper > > Had a quick look - Some thoughts: > > I don''t think you can catch all BT traffic by marking the BT ports, I > see ipp2p - can you do it with this or maybe do per IP fairness for bulk > traffic? > > Be carefull about priorotising acks - don''t all TCP packets after syn > have ack set. Being lazy on a home setup I get away with giving small > packets priority - saves alot of marking :-) > > For ingress shaping - I find it nicer to shape per IP with htb and use > esfq classic to get per tcp fairness rather than esfq on dst which is > going to effectively make many bittorrent connections go into a FIFO, > which could make for more burstiness. > > Andy. > > > > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Roy wrote:> I just had one idea, about this. > > what if make iptables module which will make something like enlarged copy > of syn packet and send it back to the sender? > (another option would be to kill 1 or 2 ack packets for one syn packet this > whould force server to reduce speed) > > > that way htb could count upcoming packet and prepare by reducing other > connetions speed? > of course that synthetic packet will have higest possible priority since it > supposed to be appear in future so cant be shaped anyway >I don''t really get this :-)> > I will try to add this functionality to my imq module next week probably. > ------------------------ > connbytes solution is not good for this, it slows down small picture loading > in web pages very much, and big downloads get even more "unused" bandwitch. > so effect is not good. expecialy that looks bad on network, when pages > become incredibly slow, but big downloads fast.Depends on lots of things I suppose - the way I have it set new connections get 256kbit - not that bad for browsing. ISTR seeing one of your scripts that did similar, IIRC using sfq with low rates. I don''t quite do it like that - for a start sfq 128 queue length is too much and if you use it on ingress sfq will hash the ~4 simoultaneous connections your browser makes into one slot. I guess yours simulated a drop with the reordering when they swapped queues rather than really dropping with a short queue to get out of slowstart. SFQ causes instability every time it rehashes on ingress because of this - there is a todo in the code somewhere. I like to set perturb high. This ingress shaping with stuff made for egress is a bit tricky - but it can be tweaked a bit. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> Roy wrote: > > I just had one idea, about this. > > > > what if make iptables module which will make something like > enlarged copy > > of syn packet and send it back to the sender? > > (another option would be to kill 1 or 2 ack packets for one syn > packet this > > whould force server to reduce speed) > > > > > > that way htb could count upcoming packet and prepare by > reducing other > > connetions speed? > > of course that synthetic packet will have higest possible > priority since it > > supposed to be appear in future so cant be shaped > anyway > > > > I don''t really get this :-) >----------------------------------------------- Ok, probably I was not able to explain it quite well, but basicaly it is how to predict incomming connections and decrease speed of exsisting one before new connections will start this prediction method should be simple and easy to implement (all concept of slow start is wrong, we exactly need fast start and slow download later)> > > > I will try to add this functionality to my imq module next week > probably. > > ------------------------ > > connbytes solution is not good for this, it slows down small > picture loading > > in web pages very much, and big downloads get even more > ''"''unused''"'' bandwitch. > > so effect is not good. expecialy that looks bad on network, > when pages > > become incredibly slow, but big downloads fast. > > Depends on lots of things I suppose - the way I have it set new > connections get 256kbit - not that bad for browsing. ISTR seeing one of > your scripts that did similar, IIRC using sfq with low rates. I don''t > quite do it like that - for a start sfq 128 queue length is too much and > if you use it on ingress sfq will hash the ~4 simoultaneous connections > your browser makes into one slot. I guess yours simulated a drop with > the reordering when they swapped queues rather than really dropping with > a short queue to get out of slowstart. > > SFQ causes instability every time it rehashes on ingress because of this > - there is a todo in the code somewhere. I like to set perturb high. > > This ingress shaping with stuff made for egress is a bit tricky - but it > can be tweaked a bit. > > Andy. >all this work well, on small number of new coonections at once , but try 20 or more at once and you will see that it is not good at all I am also using this way now, and that why I say it is not good. I dont think sfq may create any prolems, because it is basicaly same as few random fifo''s> _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Sorry for the long response time. On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> I sortof workaround this using the connbytes netfilter patch to make the > first 80k of new connections go into a short queue limited to 1/3 - 1/2 > of my downstream bandwidth.I will try this soon.. I recently screwed my kernel over bad with patch-o-matic, so its fixing time at the moment :)> Ahh bittorrent - this is a special case. It uses full duplex tcp - so > may break some upstream shapers, you can assume that a fair number of > your peers have flooded modem buffers - so there could be quite a few > "unstoppable" packets headed your way. The worse thing about it, though > is that it runs it''s own algorithm on allready open tcps - so existing > connections may go back into slowstart.Thats pretty bad.. But if i get this working, even with bittorrent, i guess that means it will probably work with anything! :)> > Would it be possible to lower the default window size, and thereby > > making tcp start up slower, or would this just lower the overall speed? > > It could help, but may also give you less of a share of a given > uploaders bandwidth. Reducing MTU may also help (if you run BT on linux > it may reduce rwin for you aswell). > > I don''t think you can catch all BT traffic by marking the BT ports, I > see ipp2p - can you do it with this or maybe do per IP fairness for bulk > traffic?Im fortunate enough to be able to run a small network with only a few people, so theres no real bad things with people doing whatever they can to cheat. Also, ipp2p should catch torrent connections, and seems to be doing so.> Be carefull about priorotising acks - don''t all TCP packets after syn > have ack set. Being lazy on a home setup I get away with giving small > packets priority - saves alot of marking :-) > > For ingress shaping - I find it nicer to shape per IP with htb and use > esfq classic to get per tcp fairness rather than esfq on dst which is > going to effectively make many bittorrent connections go into a FIFO, > which could make for more burstiness.I''ll make a note, but shouldnt esfq make things more or less fair no matter how much traffic one client has over another? -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Patrick Petersen wrote:> Sorry for the long response time. > > On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote: > > >>I sortof workaround this using the connbytes netfilter patch to make the >>first 80k of new connections go into a short queue limited to 1/3 - 1/2 >>of my downstream bandwidth. > > > I will try this soon.. I recently screwed my kernel over bad with > patch-o-matic, so its fixing time at the moment :)One possible problem - I don''t think connbytes will patch with connmark (which IIRC ipp2p uses). I have been told that it is possible to get them to work, but patch will fail as they both change the same struct, so it''s a do it by hand job...> > >>Ahh bittorrent - this is a special case. It uses full duplex tcp - so >>may break some upstream shapers, you can assume that a fair number of >>your peers have flooded modem buffers - so there could be quite a few >>"unstoppable" packets headed your way. The worse thing about it, though >>is that it runs it''s own algorithm on allready open tcps - so existing >>connections may go back into slowstart. > > > Thats pretty bad.. But if i get this working, even with bittorrent, i > guess that means it will probably work with anything! :)LOL - I think your idea about reducing rwin may help - easy on a small setup where you have control.> > >>>Would it be possible to lower the default window size, and thereby >>>making tcp start up slower, or would this just lower the overall speed? >> >>It could help, but may also give you less of a share of a given >>uploaders bandwidth. Reducing MTU may also help (if you run BT on linux >>it may reduce rwin for you aswell). >> >>I don''t think you can catch all BT traffic by marking the BT ports, I >>see ipp2p - can you do it with this or maybe do per IP fairness for bulk >>traffic? > > > Im fortunate enough to be able to run a small network with only a few > people, so theres no real bad things with people doing whatever they can > to cheat. Also, ipp2p should catch torrent connections, and seems to be > doing so.Yes - so you don''t need to mark all those ports then.> > >>Be carefull about priorotising acks - don''t all TCP packets after syn >>have ack set. Being lazy on a home setup I get away with giving small >>packets priority - saves alot of marking :-) >> >>For ingress shaping - I find it nicer to shape per IP with htb and use >>esfq classic to get per tcp fairness rather than esfq on dst which is >>going to effectively make many bittorrent connections go into a FIFO, >>which could make for more burstiness. > > > I''ll make a note, but shouldnt esfq make things more or less fair no > matter how much traffic one client has over another?Yes esfq on dst will make things fair per IP address - but in the case of ingress with the many tcps that BT uses dropping is important to get the sender to back off. Esfq on dst is not really a fairness queue any more WRT individual tcp connections, it may not make much difference, but if you use htb for per IP fairness and esfq classic, then the drops are a bit more likely to get at the most active connections. I modified esfq to drop from the head of a slot rather than the end - which in theory should make things slightly nicer and changed hash to use dst port. I really ought to make them into a switches (when I work out how) I also intend to patch with the change to sfq that was recently talked about on here. If it looks OK I''ll put it up on my webspace sometime - I don''t really know what I am doing though :-) For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK but it often hashes into the same slot and perturb causes packet reordering which hurts more when used for ingress. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Sat, 24 Apr 2004 00:11:32 +0100, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> One possible problem - I don''t think connbytes will patch with connmark > (which IIRC ipp2p uses). I have been told that it is possible to get > them to work, but patch will fail as they both change the same struct, > so it''s a do it by hand job...Im usually all for the "do it by hand" thing, but im not really up for screwing with this yet i think :) Its correct that ipp2p uses connmark... That way it is only necessary to tag a connection once, then leave it. Great for those huge setups, where the ipp2p usage should be kept at a minimum.. Afaik its a bit heavy on ressources. [SNIP - torrent talk]> LOL - I think your idea about reducing rwin may help - easy on a small > setup where you have control.That was my idea, but so far ive only found out how to alter the window size on a per route basis. If it could just be extended a little by making new connections have a small size, then putting them back to normal after a while. But if i went all the way out there, i might just as well start a complete dynamic window chaning thing. [SNIP - more torrent and ipp2p]> Yes - so you don''t need to mark all those ports then.Probably not, but i dont think the extra overhead will do any harm as long as its a puny adsl im dealing with :) I guess im not trusting ipp2p enough.> Yes esfq on dst will make things fair per IP address - but in the case > of ingress with the many tcps that BT uses dropping is important to get > the sender to back off. Esfq on dst is not really a fairness queue any > more WRT individual tcp connections, it may not make much difference, > but if you use htb for per IP fairness and esfq classic, then the drops > are a bit more likely to get at the most active connections. > > I modified esfq to drop from the head of a slot rather than the end - > which in theory should make things slightly nicer and changed hash to > use dst port. I really ought to make them into a switches (when I work > out how) I also intend to patch with the change to sfq that was recently > talked about on here. If it looks OK I''ll put it up on my webspace > sometime - I don''t really know what I am doing though :-)Sounds nice, share when you think its stable. Im all for betatesting too if you need any.> For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK > but it often hashes into the same slot and perturb causes packet > reordering which hurts more when used for ingress.I will try fiddling without esfq with fingers crossed - thanks. -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/