Hi! Take a setup as follows: Clients---->[eth0]Proxy/Firewall[eth1]---->Internet The clients are using the proxy/firewall for various client like activities like browsing, email, ftp etc. Most of the traffic is "download" traffic, i.e. clients send very small requests and recieve very large replies. I want to do the most common thing...configure download limits/fairness of use among the clients. I understand that I can achieve this by traffic shaping the *outgoing* traffic on eth0, but that is a less than optimal solution. The problem is that the link to the Internet can still get choked, if too many clients start pulling too much data. Traffic shaping on eth0 would limit the bandwidth each client sees and TCPs congestion control mechanisms *might* indeed slow down the download requests from clients, but this does not seem like a elegant way to achieve the result. I am wondering if there is any implementation [or planned project] for TCP based rate control, like that provided by commercial solutions like PacketShaper and Sitara. Regards, -Varun _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Wednesday 14 May 2003 13:24, Varun Varma wrote:> Hi! > > Take a setup as follows: > > Clients---->[eth0]Proxy/Firewall[eth1]---->Internet > > The clients are using the proxy/firewall for various client like > activities like browsing, email, ftp etc. Most of the traffic is > "download" traffic, i.e. clients send very small requests and recieve > very large replies. > > I want to do the most common thing...configure download limits/fairness > of use among the clients. > > I understand that I can achieve this by traffic shaping the *outgoing* > traffic on eth0, but that is a less than optimal solution. The problem > is that the link to the Internet can still get choked, if too many > clients start pulling too much data. Traffic shaping on eth0 would limit > the bandwidth each client sees and TCPs congestion control mechanisms > *might* indeed slow down the download requests from clients, but this > does not seem like a elegant way to achieve the result.Why not? The only problem you will have is the Proxy. If you shape on eth0, you shape cached objects and objects fetched from the internet. But you don''t want to limit traffic that''s cached. So somehow you have to be able to separate cached and non cached traffic.> I am wondering if there is any implementation [or planned project] for > TCP based rate control, like that provided by commercial solutions like > PacketShaper and Sitara.You can do the same (and even more) with a linux box. Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> >I am wondering if there is any implementation [or planned project] for > >TCP based rate control, like that provided by commercial solutions like > >PacketShaper and Sitara.>You can do the same (and even more) with a linux box.Can you write how to do it ?
On Wednesday 14 May 2003 21:59, Brandis Jaroslav wrote:> > >I am wondering if there is any implementation [or planned project] for > > >TCP based rate control, like that provided by commercial solutions like > > >PacketShaper and Sitara. > > > >You can do the same (and even more) with a linux box. > > Can you write how to do it ?www.lartc.org But what exactly do you mean with tcp based rate control? Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Wednesday, 14 May 2003, at 22:09:58 +0200, Stef Coene wrote:> But what exactly do you mean with tcp based rate control? >I suppose he means the way products like PacketShaper (from Packeteer) work: instead of using queues to limit traffic, the use waht they call "TCP rate control". It consists of modifiying TCP "packets" on the fly, transparently, adapting TCP window sizes, shortly delaying ACKs, and so on, so the sender and receiver "agree" to send traffic at a lower (configured) rate. Seems to be a patented "idea" in the USA, but I remember someone on this list talked not long ago about he waaas implementing something like this for Linux, from outside the USA. Check the archives for the message: Subject: Re: [LARTC] How far can TC go? From: Patrick McHardy <kaber@trash.net> Date: Sat, 29 Mar 2003 11:08:30 +0100 Hope it helps. -- Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Sid (Linux 2.5.69) _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thursday 15 May 2003 01:06, Jose Luis Domingo Lopez wrote:> On Wednesday, 14 May 2003, at 22:09:58 +0200, > > Stef Coene wrote: > > But what exactly do you mean with tcp based rate control? > > I suppose he means the way products like PacketShaper (from Packeteer) > work: instead of using queues to limit traffic, the use waht they call > "TCP rate control". It consists of modifiying TCP "packets" on the fly, > transparently, adapting TCP window sizes, shortly delaying ACKs, and so > on, so the sender and receiver "agree" to send traffic at a lower > (configured) rate.You mean something like the gred or red qdisc. But I wonder if it will be a big different between "TCP rate contorl" and plain old shaping.> Seems to be a patented "idea" in the USA, but I remember someone on this > list talked not long ago about he waaas implementing something like this > for Linux, from outside the USA. Check the archives for the message: > Subject: Re: [LARTC] How far can TC go? > From: Patrick McHardy <kaber@trash.net> > Date: Sat, 29 Mar 2003 11:08:30 +0100 > > Hope it helps.ok. Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Stef Coene wrote:> > You mean something like the gred or red qdisc.The difference is RED drops the packets after they arrive basing it on the understanding that the TCP implementations would take this as a hint and slow down the transmission rates. Rate control on the other hand takes a more "hands on" approach to the problem. As Jose mentioned, ack pacing, window sizing etc...it actually exploits fundamentals in TCP flow control itself.> But I wonder if it will be a big different between "TCP rate contorl" and > plain old shaping.There are lot of comparisions on this. I don''t want to get into a debate here, since people from each group [rate control vs. queueing] feel very stongly about their stand. Simply put, there are dozens of queueing disciplines, because some might "behave better" than other in some cases. I *feel* that rate control would work better in some particular cases, so I was interested in knowing if a rate control like implementation is available under Linux.> >>Seems to be a patented "idea" in the USA, but I remember someone on this >>list talked not long ago about he waaas implementing something like this >>for Linux, from outside the USA. Check the archives for the message: >>Subject: Re: [LARTC] How far can TC go? >>From: Patrick McHardy <kaber@trash.net> >>Date: Sat, 29 Mar 2003 11:08:30 +0100Thanks for the reference...still need to check it out. About the patent thing...I know for sure that ack pacing has been published in paper before. Note: I am not suggesting how to circumvent this patent in any way. That said, I think it should be possible to work out other mechanisms, based around TCP flow control, which don''t work the way Packeteer''s patent works. If you google for "linux rate control", you come up with a link like: http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html Which is a master''s thesis, proposing another way to implement rate control. Regards, -Varun _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thursday 15 May 2003 19:41, Varun Varma wrote:> Stef Coene wrote: > > You mean something like the gred or red qdisc. > > The difference is RED drops the packets after they arrive basing it on > the understanding that the TCP implementations would take this as a hint > and slow down the transmission rates. > > Rate control on the other hand takes a more "hands on" approach to the > problem. As Jose mentioned, ack pacing, window sizing etc...it actually > exploits fundamentals in TCP flow control itself. > > > But I wonder if it will be a big different between "TCP rate contorl" and > > plain old shaping. > > There are lot of comparisions on this. I don''t want to get into a debate > here, since people from each group [rate control vs. queueing] feel > very stongly about their stand.But I''m still intested in the comparision :)> Simply put, there are dozens of queueing disciplines, because some might > "behave better" than other in some cases. I *feel* that rate control > would work better in some particular cases, so I was interested in > knowing if a rate control like implementation is available under Linux.Not in the default linux kernel.> >>Seems to be a patented "idea" in the USA, but I remember someone on this > >>list talked not long ago about he waaas implementing something like this > >>for Linux, from outside the USA. Check the archives for the message: > >>Subject: Re: [LARTC] How far can TC go? > >>From: Patrick McHardy <kaber@trash.net> > >>Date: Sat, 29 Mar 2003 11:08:30 +0100 > > Thanks for the reference...still need to check it out.I already did : Packeteer has various patents covering tcp rate control and everything else they do, including the "idea" to look at upper layers to detect the type of traffic. I live in germany so i don''t really care that much about their patents (they had none in europe last time i checked). last summer i started implementing tcp rate control as qdisc for linux. i haven''t worked on it for a couple of month now, but if anyone wishes to participate i would be glad to dig out my source again. it is basically working, the remaining problems are mostly how to detect and handle interactive traffic. Patrick> About the patent thing...I know for sure that ack pacing has been > published in paper before. Note: I am not suggesting how to circumvent > this patent in any way. That said, I think it should be possible to work > out other mechanisms, based around TCP flow control, which don''t work > the way Packeteer''s patent works. > > If you google for "linux rate control", you come up with a link like: > > http://www.postel.org/pipermail/end2end-interest/2001-April/000701.html > > Which is a master''s thesis, proposing another way to implement rate > control.Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Stef,>>There are lot of comparisons on this. I don''t want to get into a debate >> here, since people from each group [rate control vs. queuing] feel >>very strongly about their stand. > > But I''m still intested in the comparision :)It''s you quest for the truth that endears you to all of us. :) Googling for "tcp rate control" would lead you to a lot of material, including documents "proving" queuing is better than rate control. Here''s the test jig I plan to setup... Windows client running Gozilla, for crude throughput reporting | |(1) | ->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf, iptraf and MRTG-- | |(2) | ->Linux based FTP/HTTP Server I will try with sustained downloads, perhaps 1GB. The tbf would be set to limit at 100kbps. From what I expect, I would only get 100kbps on the windows client m/c, i.e. at point (1). But between the gateway and server, i.e. point (2), the bandwidth consumption would be higher. How much higher I don''t know - it might be anything from 100kbps to NIC speed. If it is within a 5% tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a great place and I am happy to be here...i.e. queuing would have achieved what I want. But, I suspect the difference would be much more... Would most probably do this on this weekend and publish my results. I don''t have too much experience with tc, so I might get the values wrong for the tbf. Has anyone done this kind of test? Where can I get some published results? All the tests I have seen are done from the client perspective...or is it just a case of RTFM? I am going to go ahead with the test in any case. This only tests one side of the story. I have tested two commercial solutions and can testify that they give very fine-grained control on the bandwidth, albeit at the cost of huge latency fluctuations. Don''t have access to these units right now, so won''t be able to include the results from there. I as see it, if the above test goes as I think it would, then that itself is a strong case for looking into other mechanisms.> >>Simply put, there are dozens of queuing disciplines, because some might >>"behave better" than other in some cases. I *feel* that rate control >>would work better in some particular cases, so I was interested in >>knowing if a rate control like implementation is available under Linux. > > Not in the default linux kernel.I have been tracking the tc development for quite some time, and have been hoping for someone to develop this.> >>>>Seems to be a patented "idea" in the USA, but I remember someone on this >>>>list talked not long ago about he waaas implementing something like this >>>>for Linux, from outside the USA. Check the archives for the message: >>>>Subject: Re: [LARTC] How far can TC go? >>>>From: Patrick McHardy <kaber@trash.net> >>>>Date: Sat, 29 Mar 2003 11:08:30 +0100 >> >>Thanks for the reference...still need to check it out. > > I already did : > > Packeteer has various patents covering tcp rate control and everything else > they do, including the "idea" to look at upper layers to detect the type > of traffic. > I live in germany so i don''t really care that much about their patents > (they had none in europe last time i checked). last summer i started > implementing tcp rate control as qdisc for linux. i haven''t worked on it for > a couple of month now, but if anyone wishes to participate i would be glad > to dig out my source again. it is basically working, the remaining problems > are mostly how to detect and handle interactive traffic. > > PatrickLike I said, it''s you quest for the truth that endears you to us! Regards, -Varun _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> > But I''m still intested in the comparision :) > > It''s you quest for the truth that endears you to all of us. :) > > Googling for "tcp rate control" would lead you to a lot of material, > including documents "proving" queuing is better than rate control. > > Here''s the test jig I plan to setup... > > Windows client running Gozilla, for crude throughput reporting > > |(1) > > ->"Gateway" m/c with two NICs [RTL8139 based] running nisnet and tbf, > iptraf and MRTG-- > > |(2) > > ->Linux based FTP/HTTP Server > > I will try with sustained downloads, perhaps 1GB. The tbf would be set > to limit at 100kbps. > > From what I expect, I would only get 100kbps on the windows client m/c, > i.e. at point (1). But between the gateway and server, i.e. point (2), > the bandwidth consumption would be higher. How much higher I don''t know > - it might be anything from 100kbps to NIC speed. If it is within a 5% > tolerance, i.e. from 95kbps to 100kbps, I would say that the world is a > great place and I am happy to be here...i.e. queuing would have achieved > what I want. > > But, I suspect the difference would be much more...Why? What will happen with the packets that are arrived too much? Do you really think they all will be lost? I''m no tcp specialist, but the traffic at 2 will be exactly 100kbps. I see it like this : client starts download and goes faster and faster. When it reaches 100kbps, all tokens in the bucket are used. After that, the gateway starts delaying packets. From the tbf manpage : If no tokens are available, packets are queued, up to a configured limit Packets are not dropped but delayed. At that time, the server noticed that the packets are not arriving at a speed > 100kbps (they are queued in the bucket) at the client and throttle downs. When the server throttles down below 100kbps, the queued packets can be send again because there are tokens available in the bucket. So after some time it will be stable at 100kbps. Again, I don''t know how tbf works. This is just the way I think it works. Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thursday 15 May 2003 21:17, Michael T. Babcock wrote:> > Packets are not dropped but delayed. At that time, the server noticed > > that > > > the packets are not arriving at a speed > 100kbps (they are queued in the > > bucket) at the client and throttle downs. When the server throttles down > > below 100kbps, the queued packets can be send again because there are > > tokens > > > available in the bucket. So after some time it will be stable at > > 100kbps. > > > > Again, I don''t know how tbf works. This is just the way I think it > > works. > > Its worth reading up on how RED works while you''re at it. There are many > good > whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff. As I > understand it, TBF may not work all the time because the TCP server > (sending the packets) may respond incorrectly to the delayed ACKs on the > part of the client. Dropping packets in a RED fashion as the queue builds > up instead of queuing the packets should get perfectly stable 100kbit/s.I think we can use some test results to prove this. Varun ?? Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Stef, Michael,>>Its worth reading up on how RED works while you''re at it. There are many >>good >>whitepapers on RED/GRED w.r.t. how TCP handles exponential backoff. As I >>understand it, TBF may not work all the time because the TCP server >>(sending the packets) may respond incorrectly to the delayed ACKs on the >>part of the client. Dropping packets in a RED fashion as the queue builds >>up instead of queuing the packets should get perfectly stable 100kbit/s. > > I think we can use some test results to prove this. Varun ??This mail caught me in the middle of writing a long theoritical explanation of why I think rate control might be better, which included my technical, philosophical, financial and ..er.. sexual reasons. But then I looked at this mail and realised that the best way to take this forward is tests. I would be game to testing both TBF and RED in the jig I have mentioned in my previous mail. Hopefully, should be able produce comprehensive documentation, including detailed explanation of the test setup, and the results over this weekend. Regards, -Varun _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thursday 15 May 2003 23:23, Michael T. Babcock wrote:> > But then I looked at this mail and realised that the best way to take > > this forward is tests. I would be game to testing both TBF and RED in > > the jig I have mentioned in my previous mail. Hopefully, should be able > > produce comprehensive documentation, including detailed explanation of > > the test setup, and the results over this weekend. > > I would be specifically interested in how an HTB + RED setup compares with > the same (effective) configuration performs with just HTB. > > 1) HTB class on 100mbit line allocating 1mbit queue. > 2) HTB class on 100mbit line allocating 1mbit RED queue. > 3) HTB class on 100mbit line allocating 100 1mbit queues. > 4) HTB class on 100mbit line allocating 100 1mbit RED queues. > > At near-saturation points, RED *should* give better performance (closer to > complete saturation of the link).And also how it reaches the saturation. How many packets are dropped? Does it go straight to a flat line or is "jumping" around the line before it stabelizes? And what happens if you change the rate? How fast does it react? How many packets are dropped? (I''m planning to create such a setup my self, but I have no spare NIC) Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/