Welcome to me, and my very first lartc post. As with most first timers, i made a mistake. Admin, disregard the earlier message, as i was still waiting for the subscription confirmation. Should it get through still, i apoligize. For the last few weeks i have been trying to make it so our 2048/512 adsl line can be used for gaming and for leeching at the same time. The current result is what can be found at http://www.schmakk.dk/~schmakk which is what is running at the NAT gateway. This has done a lot for the latency, but still there is huge problems with eg. massive http downloads (5+ threads makes ping go up to at least 200). I have learned a lot from the lartc list archive, but this specifik leaves me with no clue. I have been able to get real close to normal latency by capping incoming traffic at around 1200kbits, but its no fun throwing away almost half your bandwidth. Can i get any recommendations? Also, if you have the time, a look through my script is much appreciated. (Im concerned about the calculations for dividing the bandwidth, the general setup of everything and the ipp2p+connmark tagging.) -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Patrick Petersen wrote:> I have learned a lot from the lartc list archive, but this specifik > leaves me with no clue. I have been able to get real close to normal > latency by capping incoming traffic at around 1200kbits, but its no > fun throwing away almost half your bandwidth. > > Can i get any recommendations?Let''s get the problem statement clear first. First of all, it is obvious that the high latency is a result of queueing at the ISP, before the packets are send over the slow link to your router. ISPs have very long queues normally. Secondly, one needs to understand that there isn''t really a damn thing you can do about it. If someone ping-floods you, it will saturate your downlink and latency will go through the roof. This cannot be prevented except by having access to your ISP. And thirdly, the only thing you can do, is to either discard or delay perfectly good packets which have already travelled over your slow link and spent bandwidth on it. If you drop a packet, it will most likely have to be resent and again use up the same bandwidth. And the only good this does is to try and make the connections throttle themselves when they notice that packets aren''t getting through. TCP does this, and a few application level UDP protocols do this, but not much else. So, to your *goal* in a single sentence: Force TCP to send packets slower than your downlink speed. If you can manage this, then no packets are queued at your ISP and you can prioritise traffic perfectly on your router. So, how does TCP work, then? On a connection, TCP has a window size in both directions, which is the amount of new packets that can be transferred without getting an acknowledgement for the packets already sent. Every packet sent is put on a re-send queue, and removed from there when an acknowledgement is received for that packet. If an acknowledgement doesn''t arrive for a while, the packet is re-sent. So what happens when a packet is dropped, is that the connection stalls for a moment, because a packet is unacknowledged and send window limits the amount of packets that can be in transit. TCP stacks also throttle themselves when they notice that packets are being dropped. Traditionally, the maximum window size was 64kb - that is, a maximum of 64kbs of data can be unacknowledged on the link. Then the internet became full of links which have a large bandwidth, but also lots of latency. TCP window scaling was invented, and now window sizes can be much larger than that. Also, traditionally TCP only acknowledged up to the last continguous packet - that is, it wouldn''t send acknowledgements for the packets that arrived after the missing packet. A loss of a single packet usually caused a short stall in the connection. This was augmented by cool retransmission logic, which allowed TCP to recover from the dropping of a single packet without a stall. And yet later selective acknowledgements were invented, which allows TCP to tell the other end exactly which packets it is missing, and now TCP survives quite high packet loss reasonably well. So, what''s the solution? How to make TCP throttle properly? The *real* solution would be to implement a packet mangler which would mutilate outgoing TCP ACK packets such that it would only give out transmission windows with the speed the link is configured to. However, to my knowledge, no free software implements this. I might work up a patch later, if I can come up with a good design. But, short of implementing the *real* solution, there are several things you can do to improve the situation. But first, let''s see what is happening now. Right now, your scripts shove all incoming traffic to a HTB, inside which the selection of packets happens through ESFQ. The HTB has to be limited to a rate *smaller* than the actual downlink for it to have any effect what so ever. And even so, what you do is that you queue (eg. delay) packets (maximum of 128 packets as per ESFQ), and then drop fairly traffic that comes faster. So what does TCP do about it? Latency is higher because of queueing at your router, or queuing at the ISP, so the large window sizes allow for a lot of packets to be in transit, waiting to be transferred. A bunch of packets are dropped, so those are retransmitted as soon as possible (at the arrival of the next selective acknowledgement), again filling up the queue. TCP will always try to transfer a bit faster than the speed it can get packets through to take immediate use of improved situations. With a single TCP stream, the queue size at your router or ISP is neglible, so it doesn''t hurt latency much. But when there are a large amount of connections transferring as fast as they can, there''s a lot of overshooting and what you described happens - the only way to prevent queuing at ISP is to limit the bandwidth to half of the actual link speed. What can be done to improve the situation then? First of all, don''t delay traffic. Either accept it or drop it, don''t queue it. This results in dropping more packets in total if the transmissions are bursty, but only packet drops will tell TCP to transmit slower. Make a hard limit for the speed of transmission below the maximum transmission speed, over which you drop all packets, no questions asked. Taking 10% or 20% off of the theoretical maximum is probably good enough. Implement some probability drop to tell TCP early that it is approaching the maximum bandwidth and to keep it from trying to go over the allowed bandwidth. Some Random Early Drop (RED) mechanism could work well, a Time Sliding Window Three Colour Marker (TSWTCM, RFC2859) would work even better, but again - nobody has implemented that to my knowledge. I haven''t perused your ingress marking rules at all - in general, there''s no reason to delay the processing of a packet locally. And most often dropping anything but TCP data packets is not going to do anyone much good. Note that I haven''t actually given any actual instructions on how to do this, because I have yet to experiment with all the available strategies myself. I just wished to clarify why things happen as you observe them happen and help you perhaps to find your own solutions. Long rant, phew, -- Naked _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Mon, 16 Feb 2004 05:34:20 +0200, Nuutti Kotivuori <naked@iki.fi> wrote:> Let''s get the problem statement clear first.[snip - lots of tcp]> What can be done to improve the situation then?[SNIP - more info and advice] I think this is where it breaks for me. The solution looks to be just to replace esfq with eg the fifo qdisc? Or perhaps something that will do a little fair sharing without queuing, if that exists. [SNIP - use RED] I am working on making RED do my magic, but until now im a little unsure of the result. Ive started from scratch, and it seems that the RED setup im trying from Jim diGriz is working a little. At least a few packets are dropped when the link is under use, and transferrate seems to remain at configured level. A guided tour of RED is much appreciated, as i find the sparse info online to be difficult. [SNIP - carefull with ingress] In the new script, im trying to always let important packets go through, so only low priority packets gets dropped. Things such as ACK and the like are prioritized over other things, and should hopefulle be accepted no matter what, and if bandwidth is full, unimportant traffic is dropped.> Long rant, phew,Its fine with me. Sometimes getting things cut in cardboard helps one way or the other. Im still learning. I''ve put the beta of the new shaperscript here: http://schmakk.dk/~schmakk/ Its ugly, and might not even work at the moment. Can somebody explain to me where IMQ hoks into the packet stream and where it is put back in? Im thinking the packets should go out through imq at the iptables rule, but i think im wrong. -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Patrick Petersen wrote:> Welcome to me, and my very first lartc post. > As with most first timers, i made a mistake. Admin, disregard the > earlier message, as i was still waiting for the subscription > confirmation. Should it get through still, i apoligize. > > For the last few weeks i have been trying to make it so our 2048/512 > adsl line can be used for gaming and for leeching at the same time. The > current result is what can be found at http://www.schmakk.dk/~schmakk > which is what is running at the NAT gateway. This has done a lot for the > latency, but still there is huge problems with eg. massive http > downloads (5+ threads makes ping go up to at least 200). > > I have learned a lot from the lartc list archive, but this specifik > leaves me with no clue. I have been able to get real close to normal > latency by capping incoming traffic at around 1200kbits, but its no fun > throwing away almost half your bandwidth. > > Can i get any recommendations? > > Also, if you have the time, a look through my script is much appreciated. > (Im concerned about the calculations for dividing the bandwidth, the > general setup of everything and the ipp2p+connmark tagging.)I see you have a newer version now anyway, but I tried you script last night (not connmark/ipp2p as it clashed with connbytes). I have 256/512 so things in theory should be nicer for you. I am still testing myself, so can''t post a solution but can make some observations - The delete rule for iptabled needs -D not -A. esfq won''t work hashing src on egress if you are doing NAT see the KPTD on www.docum.org - egress qos comes after NAT. Ingress with dst should be ok if you are NATing as long as you used the IMQ NAT patch. The trouble with esfq hashing per user (unless you use limit) is that each user gets a 128 packet queue, which if they have many connections gets full and drops take a long time to be noticed. I have a modified esfq which overcomes this somewhat, but I also use classic hash and restrict the size of the queue. I can see why commercial bandwidth controllers use advertised window manipulation - often dropping is needed to get the sender to back off a bit and set its'' congestion window, but if you queue this may result in a resync burst later. Being able to reduce adv window on dups/sacks and increase slowly/randomly would be handy. One thing that helps me is to give new bulk connections their own class with a short queue for the first 80000 bytes using connbytes (netfilter extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp slowstart from overshooting. I have also tried connbytes just to drop early packets, but with browsers making many simultaneous connections, the resyncs cause a latency burst. I see you are trying RED - in theory this should be nice, but remember the settings/docs you read about don''t take into account that you are trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% faster than what you are aiming for. I am not convinced that just dropping ingress is best - a short queue yes, then at least you don''t ever drop game packets. If I had 2048 down I reckon I could keep down below 100 now - apart from the odd 1 sec blip. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi, Can i have more abt Random Early Drop (RED) mechanism. Help me with reference sites, sample codings, etc. I am in the process of shaping the bandwidth, in which i like to know how i can drop the packets when it exceeds the specified limit. Kindly help me with reference sites, sample codings, etc., Thanks & Regards, Kalaiselvan.> Patrick Petersen wrote: > > I have learned a lot from the lartc list archive, but this specifik > > leaves me with no clue. I have been able to get real close to normal > > latency by capping incoming traffic at around 1200kbits, but its no > > fun throwing away almost half your bandwidth. > > > > Can i get any recommendations? > > Let''s get the problem statement clear first. > > First of all, it is obvious that the high latency is a result of > queueing at the ISP, before the packets are send over the slow link > to your router. ISPs have very long queues normally. > > Secondly, one needs to understand that there isn''t really a damn > thing you can do about it. If someone ping-floods you, it will > saturate your downlink and latency will go through the roof. This > cannot be prevented except by having access to your ISP. > > And thirdly, the only thing you can do, is to either discard or > delay perfectly good packets which have already travelled over your > slow link and spent bandwidth on it. If you drop a packet, it will > most likely have to be resent and again use up the same > bandwidth. And the only good this does is to try and make the > connections throttle themselves when they notice that packets aren''t > getting through. TCP does this, and a few application level UDP > protocols do this, but not much else. > > So, to your *goal* in a single sentence: > > Force TCP to send packets slower than your downlink speed. > > If you can manage this, then no packets are queued at your ISP and you > can prioritise traffic perfectly on your router. > > So, how does TCP work, then? > > On a connection, TCP has a window size in both directions, which is > the amount of new packets that can be transferred without getting an > acknowledgement for the packets already sent. Every packet sent is > put on a re-send queue, and removed from there when an > acknowledgement is received for that packet. If an acknowledgement > doesn''t arrive for a while, the packet is re-sent. > > So what happens when a packet is dropped, is that the connection > stalls for a moment, because a packet is unacknowledged and send > window limits the amount of packets that can be in transit. TCP > stacks also throttle themselves when they notice that packets are > being dropped. > > Traditionally, the maximum window size was 64kb - that is, a maximum > of 64kbs of data can be unacknowledged on the link. Then the > internet became full of links which have a large bandwidth, but also > lots of latency. TCP window scaling was invented, and now window > sizes can be much larger than that. > > Also, traditionally TCP only acknowledged up to the last continguous > packet - that is, it wouldn''t send acknowledgements for the packets > that arrived after the missing packet. A loss of a single packet > usually caused a short stall in the connection. This was augmented > by cool retransmission logic, which allowed TCP to recover from the > dropping of a single packet without a stall. And yet later selective > acknowledgements were invented, which allows TCP to tell the other > end exactly which packets it is missing, and now TCP survives quite > high packet loss reasonably well. > > So, what''s the solution? How to make TCP throttle properly? > > The *real* solution would be to implement a packet mangler which > would mutilate outgoing TCP ACK packets such that it would only give > out transmission windows with the speed the link is configured > to. However, to my knowledge, no free software implements this. I > might work up a patch later, if I can come up with a good design. > > But, short of implementing the *real* solution, there are several > things you can do to improve the situation. But first, let''s see what > is happening now. > > Right now, your scripts shove all incoming traffic to a HTB, inside > which the selection of packets happens through ESFQ. The HTB has to > be limited to a rate *smaller* than the actual downlink for it to > have any effect what so ever. And even so, what you do is that you > queue (eg. delay) packets (maximum of 128 packets as per ESFQ), and > then drop fairly traffic that comes faster. > > So what does TCP do about it? Latency is higher because of queueing > at your router, or queuing at the ISP, so the large window sizes > allow for a lot of packets to be in transit, waiting to be > transferred. A bunch of packets are dropped, so those are > retransmitted as soon as possible (at the arrival of the next > selective acknowledgement), again filling up the queue. TCP will > always try to transfer a bit faster than the speed it can get > packets through to take immediate use of improved situations. > > With a single TCP stream, the queue size at your router or ISP is > neglible, so it doesn''t hurt latency much. But when there are a > large amount of connections transferring as fast as they can, > there''s a lot of overshooting and what you described happens - the > only way to prevent queuing at ISP is to limit the bandwidth to half > of the actual link speed. > > What can be done to improve the situation then? > > First of all, don''t delay traffic. Either accept it or drop it, > don''t queue it. This results in dropping more packets in total if > the transmissions are bursty, but only packet drops will tell TCP to > transmit slower. > > Make a hard limit for the speed of transmission below the maximum > transmission speed, over which you drop all packets, no questions > asked. Taking 10% or 20% off of the theoretical maximum is probably > good enough. > > Implement some probability drop to tell TCP early that it is > approaching the maximum bandwidth and to keep it from trying to go > over the allowed bandwidth. Some Random Early Drop (RED) mechanism > could work well, a Time Sliding Window Three Colour Marker (TSWTCM, > RFC2859) would work even better, but again - nobody has implemented > that to my knowledge. > > I haven''t perused your ingress marking rules at all - in general, > there''s no reason to delay the processing of a packet locally. And > most often dropping anything but TCP data packets is not going to do > anyone much good. > > Note that I haven''t actually given any actual instructions on how to > do this, because I have yet to experiment with all the available > strategies myself. I just wished to clarify why things happen as you > observe them happen and help you perhaps to find your own solutions. > > Long rant, phew, > -- Naked > > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thu, 19 Feb 2004 09:45:16 +0000, Andy Furniss <andy.furniss@dsl.pipex.com> wrote:> The delete rule for iptabled needs -D not -A.Yes, that one was bad. I noticed it when i discovered how to list rules in chains... I think all my rules was there about 10 times each, since i never removed anything :)> esfq won''t work hashing src on egress if you are doing NAT see the KPTD > on www.docum.org - egress qos comes after NAT. Ingress with dst should > be ok if you are NATing as long as you used the IMQ NAT patch.I thought that with the NAT patch, imq would see incoming packets with the real ip on the internal net?> The trouble with esfq hashing per user (unless you use limit) is that > each user gets a 128 packet queue, which if they have many connections > gets full and drops take a long time to be noticed. I have a modified > esfq which overcomes this somewhat, but I also use classic hash and > restrict the size of the queue.I didnt thing a 128 packet queue would do any real difference, but im testing with other qdiscs at the moment, since it seems that bandwidth is being divided, but there is still latency problems.> I can see why commercial bandwidth controllers use advertised window > manipulation - often dropping is needed to get the sender to back off a > bit and set its'' congestion window, but if you queue this may result in > a resync burst later. Being able to reduce adv window on dups/sacks and > increase slowly/randomly would be handy.Ah yes, the holy grail it seems. Its a mystery that noone has started an open source project for this.> One thing that helps me is to give new bulk connections their own class > with a short queue for the first 80000 bytes using connbytes (netfilter > extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp > slowstart from overshooting. I have also tried connbytes just to drop > early packets, but with browsers making many simultaneous connections, > the resyncs cause a latency burst.If im getting this right, you are using iptables to manage bandwidth directly? Im real bad with iptables still, i dont think ive gotten to know half of it yet.> I see you are trying RED - in theory this should be nice, but remember > the settings/docs you read about don''t take into account that you are > trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% > faster than what you are aiming for.Im still kind of blank on RED, what im trying out now is to use the RED part of Jim diGriz'' (i think) script. It seems that a few packets are actually dropped when the link is getting full, but only about 5-10 in a couple of minutes.. Seems a bit low?> I am not convinced that just dropping ingress is best - a short queue > yes, then at least you don''t ever drop game packets.This is what im trying to do now, using IMQ for incoming traffic. However, it seems that my 2 root qdiscs are delaying packets a lot. According to tc -s qdisc etc etc about 100-500 packets are overlimits, even when dataflow is no more than around 5-10kb/s. Setting a ceil on the root classes seems to help it out a little, but not completely. This i dont understand. -- Patrick Petersen <lartc@schmakk.dk> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Patrick Petersen wrote:>>esfq won''t work hashing src on egress if you are doing NAT see the KPTD >>on www.docum.org - egress qos comes after NAT. Ingress with dst should >>be ok if you are NATing as long as you used the IMQ NAT patch. > > > I thought that with the NAT patch, imq would see incoming packets with > the real ip on the internal net?Yes - it is OK for incoming, but not for outbound AFAIK whether you use imq or the interface direct. The NAT patch for IMQ only changes the ingress hooks, not egress.> > >>The trouble with esfq hashing per user (unless you use limit) is that >>each user gets a 128 packet queue, which if they have many connections >>gets full and drops take a long time to be noticed. I have a modified >>esfq which overcomes this somewhat, but I also use classic hash and >>restrict the size of the queue. > > > I didnt thing a 128 packet queue would do any real difference, but im > testing with other qdiscs at the moment, since it seems that bandwidth > is being divided, but there is still latency problems.Are the problems brief or does it totally loose it. I just tested the ingress policer at 80% and with 8 tcps going it looses control.> > >>I can see why commercial bandwidth controllers use advertised window >>manipulation - often dropping is needed to get the sender to back off a >>bit and set its'' congestion window, but if you queue this may result in >>a resync burst later. Being able to reduce adv window on dups/sacks and >>increase slowly/randomly would be handy. > > > Ah yes, the holy grail it seems. Its a mystery that noone has started an > open source project for this. > > >>One thing that helps me is to give new bulk connections their own class >>with a short queue for the first 80000 bytes using connbytes (netfilter >>extra patch). This is limited to rate downrate/5 ceil /3 and stops tcp >>slowstart from overshooting. I have also tried connbytes just to drop >>early packets, but with browsers making many simultaneous connections, >>the resyncs cause a latency burst. > > > If im getting this right, you are using iptables to manage bandwidth > directly? Im real bad with iptables still, i dont think ive gotten to > know half of it yet.I did, just experementing try dropping an early packet from new connections. It was better in some cases, but not as good as putting new connections in their own limited bandwidth short queue. This still causes drops, but delays the packets aswell. I use the netfilter connbytes patch to mark < 80000 bytes then put them in their own htb class.> > >>I see you are trying RED - in theory this should be nice, but remember >>the settings/docs you read about don''t take into account that you are >>trying to shape behind a fifo (at ISP/teleco) that dequeues only 20% >>faster than what you are aiming for. > > > Im still kind of blank on RED, what im trying out now is to use the RED > part of Jim diGriz'' (i think) script. It seems that a few packets are > actually dropped when the link is getting full, but only about 5-10 in a > couple of minutes.. Seems a bit low?I guess it depends on how many connections - just queuing will throttle a low number without many drops. I tested your RED settings with 8 and it handled it OK and drops enough. It temporarily looses it when a new connection is started - but it''s hard to stop this. It also causes a blip when my link is otherwise empty - putting new connections into a restricted seperate class should stops this. I think it will be hard to get RED perfect as it really ought to live at your ISP, before the bottleneck, but it''s still worth seeing how well it can be tweaked.> > >>I am not convinced that just dropping ingress is best - a short queue >>yes, then at least you don''t ever drop game packets. > > > This is what im trying to do now, using IMQ for incoming traffic. > However, it seems that my 2 root qdiscs are delaying packets a lot. > According to tc -s qdisc etc etc about 100-500 packets are overlimits, > even when dataflow is no more than around 5-10kb/s. Setting a ceil on > the root classes seems to help it out a little, but not completely. This > i dont understand.This is OK, you want to see overlimits - tcp will send packets in bursts and these will come in at full link speed, and so be seen by HTB as overlimits. Andy. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/