I''m a little confused about the many ways I''ve read that can be used for traffic load balancing, that is, two or more interfaces to the outside world being used transparently and efficiently by the internal machines. I heard about: a) netfilter SNAT to more than one IP. If I''m correct, this is only a round robin, that is, one connection goes here, the other goes there, then the next goes here again, etc without much thought. b) multipath default route. Seems to do something similar, but it "caches" routes. What exactly does this mean in the long run? c) multipath default route, but with the equalize option. Again, seems to work, but the best description I could find about it was something along the lines of "packet randomization", whatever that means. What does it do in the long run? Is it better/worse than b)? d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed to understand how (no, I didn''t read that RFC thoroughly, it''s really high tech for me at this point). Does it use multipath routes to accomplish this? Any help is appreciated, thanks. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, since my masters thesis is also concerned with this, i thought of taking this opportunity to clarify/get clarified some of my own thougths. please see inline. On Thursday 21 November 2002 07:19 am, Andreas Hasenack wrote:> I''m a little confused about the many ways I''ve read that can be used > for traffic load balancing, that is, two or more interfaces to the > outside world being used transparently and efficiently by the internal > machines. > > I heard about: > a) netfilter SNAT to more than one IP. If I''m correct, this is only > a round robin, that is, one connection goes here, the other goes there, > then the next goes here again, etc without much thought. >i believe that SNAT is for connections initiated from outside world towards the internal network. and this would be reverse of what is intended here, that of using the links to the outside world efficiently. moreover if setup properly, the packets will go out back on the interface they came in.> b) multipath default route. Seems to do something similar, but it "caches" > routes. What exactly does this mean in the long run? >i''m still looking at the code here. AFAIK, the when you have multipath option set, then when looking up for a route to an address, among the multiple equal cost paths, one is selected according to some criteria (i read random somewhere in the kernel but am not able to locate it now). but once the route is found, it is cached in the route cache and all packets will follow that route. so it is kind of a route-based load-balancing.> c) multipath default route, but with the equalize option. Again, seems to > work, but the best description I could find about it was something along > the lines of "packet randomization", whatever that means. What does it do > in the long run? Is it better/worse than b)? >i read in one of the threads that ''equalize'' causes per-packet load balancing, i.e looks up the route for each packet. but i am doubtful about the performance when each packet causes a route-lookup in the FIB. (http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html)> d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed > to understand how (no, I didn''t read that RFC thoroughly, it''s really high > tech for me at this point). Does it use multipath routes to accomplish this? > > Any help is appreciated, thanks.ashok - ----------------------------------------------------------------------------- My public key: gpg --recv-keys --keyserver blackhole.pca.dfn.de DCB44F2E http://pgp.mit.edu:11371/pks/lookup?search=N+N+Ashok+%28CSE+MSU%29+%3Cnnashok%40yahoo.com%3E&op=index - ----------------------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE93RwTRhXpVty0Ty4RArc/AKCM7VWIK1CygSawQxDtPvcia/nclACeMqMI w5aTSJ7L1utzv0ePbUWsGxk=oALf -----END PGP SIGNATURE----- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu:> i believe that SNAT is for connections initiated from outside world towards > the internal network. and this would be reverse of what is intended here,That''s DNAT.> i''m still looking at the code here. AFAIK, the when you have multipath option > set, then when looking up for a route to an address, among the multiple equal > cost paths, one is selected according to some criteria (i read random > somewhere in the kernel but am not able to locate it now). but once the routeThis criteria I would like to understand.> route. so it is kind of a route-based load-balancing.Considering that the route cache is 60s by default, this should work I guess. I''m setting up such a setup for testing.> i read in one of the threads that ''equalize'' causes per-packet load balancing, > i.e looks up the route for each packet. but i am doubtful about the > performance when each packet causes a route-lookup in the FIB. > (http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html)Interesting, so ''equalize'' bypasses the cache, that''s about it? So we only need to know the criteria with which the route is choosed in a multipath route. Packet count? Round robin?> > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed > > to understand how (no, I didn''t read that RFC thoroughly, it''s really high > > tech for me at this point). Does it use multipath routes to accomplish this?Any thoughts on how OSPF does this? Thanks for your input _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thu, Nov 21, 2002 at 05:11:23PM -0200, Andreas Hasenack wrote:> > > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed > > > to understand how (no, I didn''t read that RFC thoroughly, it''s really high > > > tech for me at this point). Does it use multipath routes to accomplish this? > > Any thoughts on how OSPF does this?All OSPF does is installing equal cost routes in the FIB. The actual forwarding, making routing decision and/or load balancing has nothing to do with the OSPF protocol. Ramin _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thu, 2002-11-21 at 11:11, Andreas Hasenack wrote:> Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu: > > i believe that SNAT is for connections initiated from outside world towards > > the internal network. and this would be reverse of what is intended here, > > That''s DNAT. >Correct DNAT is outside in and SNAT is inside out.> > i''m still looking at the code here. AFAIK, the when you have multipath option > > set, then when looking up for a route to an address, among the multiple equal > > cost paths, one is selected according to some criteria (i read random > > somewhere in the kernel but am not able to locate it now). but once the route > > This criteria I would like to understand.From what I understand it''s like this. The whole load balancing thing works in two ways. load balancing outside in request and load balancing inside out request The multipath feature is there so you can have multiple paths from the inside out. Although without the equalize option each route would have to be specified and there really would be no balancing. Equalize along with the weights is what creates the balancing, from request coming from the inside going out. Multipath, equalize, and weights do not effect the incoming requests from the outside world. That are initiated from the outside. That type of load balancing typically is done at the DNS level. Balancing outside to in requests. As for the algorithm to balance from within using multipath and equalize its like this.If you have two routes with equal weights each one will be used half the time. If you have two routes one with weight 1, and the other with 2, then more than likely the first will be used 2 out of 3 times, with the second being used 1 out of 3 times. Or something like that.> > route. so it is kind of a route-based load-balancing. > > Considering that the route cache is 60s by default, this should work I guess. > I''m setting up such a setup for testing.I played around with the cache settings for like a week straight a while back. In the end I reverted to default values and they have worked out fine.> > i read in one of the threads that ''equalize'' causes per-packet load balancing, > > i.e looks up the route for each packet. but i am doubtful about the > > performance when each packet causes a route-lookup in the FIB. > > (http://www.uwsg.iu.edu/hypermail/linux/net/0107.3/0028.html) > > Interesting, so ''equalize'' bypasses the cache, that''s about it?No, it will first look to the catch, and then consult the multipath for the next route/gateway to use when using the equalize option if it does not already exist in the catch.> So we only need to know the criteria with which the route is choosed in a multipath > route. Packet count? Round robin?Also I do not believe the load balancing is packet based. Usually it''s more "connection" based. Meaning that if you request a file, more than likely all parts of that file will be transfered using the same route. If you request it again, it may take the same route or another. Once again a DNS thing if the request comes from the outside world. Now if the request was generated from the inside it would still work some what the same. If I send two emails out at once, the first will use gw1 and the other will use gw2. All packets for each will travel via the same route and use the same gateway from start to finish. If it was more on a packet level, the other end would be confused. It would be getting responses from an IP it was not expecting response from. I would imagine each side to send redirects, and all sorts of problems. Like it receiving every other packet and dropping the packets in between. I believe the cache also has effect on this. If during a file transfer the route cache is flushed, there is the possibility of the rest of the packets going out a different interface. But at that point all subsequent packets would be going out that interface as well. At some point I would further assume the client to catch on, and finish the transfer at some point. SSH and FTP most likely will not. Not to sure about HTTP, probably protocol dependent 1.0/1.1. This is also what would happen in case of link failure, so if redundancy is your main concern you may want to play with the catch settings. In the end load balancing is not exactly what you might expect it to be. From the inside world out, you do it via the kernel multipath equalize. The the outside world in you use DNS. Neither does it perfectly or with intelligent algorithms. Neither allow you to use all paths for a single transfer. So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs line. You do have one internal gateway for both, and if one goes down the other can be used. So you do have redundancy, and both lines can be used to serve difference requests to different places. You also have separate paths, so if ISP1 is closer client1, and ISP2 is closer to client2, each others traffic will not effect the other. Not to mention better transfers due to less distance. But that''s it. No bonding or true load balancing.> > > d) OSPF. I read in the RFC that OSPF can do "load balancing", but I failed > > > to understand how (no, I didn''t read that RFC thoroughly, it''s really high > > > tech for me at this point). Does it use multipath routes to accomplish this? > > Any thoughts on how OSPF does this?Never heard of it, but basically I am only aware of one way. Without having your own public ip block, and different provider routing for your own block. I believe then you would be using BGP or something.> Thanks for your input > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >-- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On 21 Nov 2002 14:20:57 -0800 "William L. Thomson Jr." <support@obsidian-studios.com> wrote:> Also I do not believe the load balancing is packet based. Usually it''s > more "connection" based. Meaning that if you request a file, more than > likely all parts of that file will be transfered using the same route. > If you request it again, it may take the same route or another.It is packet based. There is no connection concept in IP. From the IP Command Reference: equalize: allow packet by packet randomization on multipath routes. My understanding is, that for equalize to work, all lines must go to the same point and that must not be the end point. Also, this same point must implement the equalize very much the same way.> Now if the request was generated from the inside it would still work > some what the same. If I send two emails out at once, the first will use > gw1 and the other will use gw2. > > All packets for each will travel via the same route and use the same > gateway from start to finish.Not using equalize.> If it was more on a packet level, the other end would be confused.unless there is a routing cache which will hold the path. -- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help . _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On 21 Nov 2002 15:07:11 -0800 "William L. Thomson Jr." <support@obsidian-studios.com> wrote:> > From the IP Command Reference: > > > > equalize: allow packet by packet randomization on multipath > > routes. > > > > My understanding is, that for equalize to work, all lines must go to > > the same point and that must not be the end point. Also, this same > > point must implement the equalize very much the same way. > > Not necessarily. I have two lines going in completely different > directions, different private nets, and then via different ISP/Public > IPs.Two lines in different directions using equalize? What''s the point of this?> > > Now if the request was generated from the inside it would still work > > > some what the same. If I send two emails out at once, the first will > > > use gw1 and the other will use gw2. > > > > > > All packets for each will travel via the same route and use the same > > > gateway from start to finish. > > > > Not using equalize. > > Yes using equalize, because the path will already be known and used from > catch. Correct that if a further look up occurs the packets will use a > different interface, but that is rare. Most of the time during a > transfer the route is already known and kept in catch from the first > initial lookup.I didn''t play too much with equalize, but using it for load balancing and redundancy failed for reason by with equalize is defined. As far as I understood how weighting works: There are several identic routes so that the probability to pick one of them corresponds to the given weights. Still on a packet base. With equalize, the balancing should be perfect, which can never be the case with few connections, even if they imply massive data transfer. But if there is no NAT later, it just can''t work.> > unless there is a routing cache which will hold the path. > > That''s the problem. The route cache on the Linux load balancing router > will be correct, but the route cache on the client will not be.If "client" here is a computer reaching the internet through the linux router (it can''t be anything else, because you said that there are two independent lines out of your control), and unless that client uses source routing which might be ignored by the linux router, the client''s view of this situation really doesn''t matter.> The client will be caching the wrong path. So unless a redirect is sent > the client will site there and expect further communication using the > route in cache.The client will cache the route which leads to the Linux router and not bother with details happening later. -- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help . _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thu, 2002-11-21 at 14:55, Christoph Simon wrote:> On 21 Nov 2002 14:20:57 -0800 > "William L. Thomson Jr." <support@obsidian-studios.com> wrote: > > > Also I do not believe the load balancing is packet based. Usually it''s > > more "connection" based. Meaning that if you request a file, more than > > likely all parts of that file will be transfered using the same route. > > If you request it again, it may take the same route or another. > > It is packet based. There is no connection concept in IP.That''s why I put that word in quotes. I meant more like during a file transfer than with a connection in the true sense.> >From the IP Command Reference: > > equalize: allow packet by packet randomization on multipath > routes. > > My understanding is, that for equalize to work, all lines must go to > the same point and that must not be the end point. Also, this same > point must implement the equalize very much the same way.Not necessarily. I have two lines going in completely different directions, different private nets, and then via different ISP/Public IPs. Now for true load balancing or bonding yes. Both lines would need to be coming from and going to the same ISP.> > Now if the request was generated from the inside it would still work > > some what the same. If I send two emails out at once, the first will use > > gw1 and the other will use gw2. > > > > All packets for each will travel via the same route and use the same > > gateway from start to finish. > > Not using equalize.Yes using equalize, because the path will already be known and used from catch. Correct that if a further look up occurs the packets will use a different interface, but that is rare. Most of the time during a transfer the route is already known and kept in catch from the first initial lookup. I have not run into any problems along those lines.> > If it was more on a packet level, the other end would be confused. > > unless there is a routing cache which will hold the path.That''s the problem. The route cache on the Linux load balancing router will be correct, but the route cache on the client will not be. The client will be caching the wrong path. So unless a redirect is sent the client will site there and expect further communication using the route in cache. When I was experimenting with route cache timing in the Linux kernel, I had all kinds of problems on the client side. The client''s catch settings would need to be the same as the linux router. Which is pretty impossible on the Internet. On a lan it would be a different story. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Thu, 2002-11-21 at 15:41, Christoph Simon wrote:> On 21 Nov 2002 15:07:11 -0800 > "William L. Thomson Jr." <support@obsidian-studios.com> wrote: > > > > Not necessarily. I have two lines going in completely different > > directions, different private nets, and then via different ISP/Public > > IPs. > > Two lines in different directions using equalize? What''s the point of > this?Once again let me clarify myself. By different directions I mean different ISPs. Not incoming from one and outgoing on another. It''s more having two separate connections to the Internet with two separate ISPs. That''s what I mean by two different directions. One of my ISPs is North of me, the other is South. So if I were to say send out a mass mailer or god forbid spam, I could use both lines from the inside to send data out. Same page now? Sorry bout that.> > > > Now if the request was generated from the inside it would still work > > > > some what the same. If I send two emails out at once, the first will > > > > use gw1 and the other will use gw2. > > > > > > > > All packets for each will travel via the same route and use the same > > > > gateway from start to finish. > > > > > > Not using equalize. > > > > Yes using equalize, because the path will already be known and used from > > catch. Correct that if a further look up occurs the packets will use a > > different interface, but that is rare. Most of the time during a > > transfer the route is already known and kept in catch from the first > > initial lookup. > > I didn''t play too much with equalize, but using it for load balancing > and redundancy failed for reason by with equalize is defined.Yes for true redundancy the routes need to be removed and then added back or to stop being used. I believe that''s were Julian''s dead gateway detection stuff comes in.> As far > as I understood how weighting works: There are several identic routes > so that the probability to pick one of them corresponds to the given > weights. Still on a packet base. With equalize, the balancing should > be perfect, which can never be the case with few connections, even if > they imply massive data transfer. But if there is no NAT later, it > just can''t work.Correct based on weights using equalize. Because without equalize it will just travel the first route no matter what unless a route using the second is already know and existing in cache. From the inside balancing is basically perfect using equalize. It''s code I mean it does what it''s is supposed to do but is not smart. If one line is being used more than another from the outside, it will still balance things from the inside. Even thought one line is being bogged down. Even if you say have two rather large unequal files to send out, it will send the first file out one, and the second file out the other, and then keep going back and forth equally if the weights are equal. Even though the actual load is not equal. In my case I am doing NAT, and it is part of the reason why things work, along with Julian''s patches. But I have been informed I believe by Julian and others that the load balancing, multipath equalize feature can be used even without NAT but in a different situation that mine?> > > unless there is a routing cache which will hold the path. > > > > That''s the problem. The route cache on the Linux load balancing router > > will be correct, but the route cache on the client will not be. > > If "client" here is a computer reaching the internet through the linux > router (it can''t be anything else, because you said that there are two > independent lines out of your control), and unless that client uses > source routing which might be ignored by the linux router, the > client''s view of this situation really doesn''t matter.By client I mean someone in the outside world talking to say a server. The server is behind the Linux router. Not a client on the local lan looking out.> > The client will be caching the wrong path. So unless a redirect is sent > > the client will site there and expect further communication using the > > route in cache. > > The client will cache the route which leads to the Linux router and > not bother with details happening later.I am talking about the client caching the wrong public ip and communication being attempted on a separate public ip from the server. While the client sits there and times out on the first ip, when ever it''s cache expires. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 21 November 2002 05:20 pm, William L. Thomson Jr. wrote:> On Thu, 2002-11-21 at 11:11, Andreas Hasenack wrote: > > Em Thu, Nov 21, 2002 at 12:46:56PM -0500, Ashok N N escreveu: > > > i believe that SNAT is for connections initiated from outside worldtowards> > > the internal network. and this would be reverse of what is intendedhere,> > > > That''s DNAT. > > > > Correct DNAT is outside in and SNAT is inside out. >Yep..my mistake (i seem to always confuse between the two) ashok -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE93YWoRhXpVty0Ty4RArnnAKCXYk3nVU16mvVEHf/WSgALqBQDYgCggm1A QGctynuQzP9vOTdtof8bohU=hqBe -----END PGP SIGNATURE----- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Thu, Nov 21, 2002 at 02:20:57PM -0800, William L. Thomson Jr. escreveu:> Also I do not believe the load balancing is packet based. Usually it''s > more "connection" based. Meaning that if you request a file, more than > likely all parts of that file will be transfered using the same route. > If you request it again, it may take the same route or another.If I make many connections from one IP (inside) to a web server (outside), for example (like many simultaneous downloads, or a complex page), I think they will all go via the same route, because the originating IP and the destination are the same. It will hit the cache. Hmm, not good if your users use a proxy, but then again, the proxy would cache the page probably.> Now if the request was generated from the inside it would still work > some what the same. If I send two emails out at once, the first will use > gw1 and the other will use gw2.Unless they are sent to the same MTA in the outside, then it will get a cache hit (supposing the 60s haven''t gone by then). Or not?> All packets for each will travel via the same route and use the same > gateway from start to finish.Agreed.> If it was more on a packet level, the other end would be confused.Sure. When I said "packet count" before I was thinking about something along the lines of real traffic balancing, that is, the router somehow remembering how many packets it sent to each route and choosing the less used one.> It would be getting responses from an IP it was not expecting response > from. I would imagine each side to send redirects, and all sorts of > problems. Like it receiving every other packet and dropping the packets > in between.And breaking stateful firewalls.> If during a file transfer the route cache is flushed, there is the > possibility of the rest of the packets going out a different interface.Uh oh... It shouldn''t be that simple, what about that 60s timeout for the cache? It''s very likely to occur during a file transfer.> Neither does it perfectly or with intelligent algorithms. Neither allow > you to use all paths for a single transfer.Only things like MPPP I guess, for example, or channel bonding, or TQL.> So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs > line. You do have one internal gateway for both, and if one goes down > the other can be used. So you do have redundancy, and both lines can be > used to serve difference requests to different places.So it''s more like redundancy/HA with a best effort towards balancing. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Thu, Nov 21, 2002 at 08:55:05PM -0200, Christoph Simon escreveu:> My understanding is, that for equalize to work, all lines must go to > the same point and that must not be the end point. Also, this same > point must implement the equalize very much the same way.What is it that you call a "point" here (destination)? The same ISP? The same network? I understand that it should be the same ISP because of egress filtering, that is, one ISP should block packets with a source address that doesn''t belong to the ISP supplying the link. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Thu, Nov 21, 2002 at 04:24:06PM -0800, William L. Thomson Jr. escreveu:> But I have been informed I believe by Julian and others that the load > balancing, multipath equalize feature can be used even without NAT but > in a different situation that mine?I''m confused as well. Suppose you have two links to the internet, a DMZ, and an internal network, SNAT''ed. Suppose you have a public web server in the DMZ (the DMZ is not SNAT''ed). How would multipath route (with or without equalize) help here? I mean, it would only really help if there were connections starting from the inside (DMZ or SNAT''ed network) to the outside. But: - the internal network would probably do many downloads, and not uploads - the web server doesn''t originate traffic, it responds to requests from the outside world, and it will respond using the same link the request came in (or not?) _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Thu, Nov 21, 2002 at 04:06:26PM -0800, William L. Thomson Jr. escreveu:> Not necessarily. I have two lines going in completely different > directions, different private nets, and then via different ISP/Public > IPs.Do you have public servers that are accessed from the outside? How do you respond to requests comming down one link? I suppose via the same link, or using SNAT, otherwise you would have packets with a source IP from ISP1 traveling through the link to ISP2, right? _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 22 Nov 2002 10:30:35 -0200 Andreas Hasenack <andreas@conectiva.com.br> wrote:> Em Thu, Nov 21, 2002 at 08:55:05PM -0200, Christoph Simon escreveu: > > My understanding is, that for equalize to work, all lines must go to > > the same point and that must not be the end point. Also, this same > > point must implement the equalize very much the same way. > > What is it that you call a "point" here (destination)? The same ISP? The > same network? > > I understand that it should be the same ISP because of egress filtering, > that is, one ISP should block packets with a source address that doesn''t > belong to the ISP supplying the link.I was thinking of a configuration like: Satelite Main-Office -> ISP1 -m- LAN1 -> ISP2 -m- ISPx -> linux-router2 -> ISP-BIG -> -> ISP3 -m- If I have an office with one big Internet tube (e.g., MBs), I can put "linux-router2" there. Then all satelite offices could use one or more modems to reach this router. That linux-router2 would NAT everybody coming from the modems and forward the packets to ISP-BIG. Nobody in LAN1 would even attempt to reach linux-router2, which is the last point where their actually source addresses are known. With equalize this should allow for a packetwise loadbalancing to increase the effective bandwidth even for a single file transfer between a host in LAN1 and ISP-BIG. In this configuration, all modem lines go finally to one point (linux-router2) and that is not the endpoint. In fact, all internal clients would use linux-router2 as the default gateway. No cooperation of any ISP is required. This configuration makes sense if LAN1 and linux-router2 are in different cities and either LAN1 has no other reliable choice than modems or wants redundancy. ISP-BIG also needs to be cheap and reliable enough for this to make sense. As I said before, I didn''t play too much with equalize, but can''t see a essencially different scenario for a packet-wise loadbalancing. -- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help . _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 13:39, Andreas Hasenack wrote:> - the internal network would probably do many downloads, and not uploads > > - the web server doesn''t originate traffic, it responds to requests from > the outside world, and it will respond using the same link the request > came in (or not?) >No, replies packet will hit the multipath route, and thus may not take the same link to come back. But in such situation, the sraddr in the replies packets are likely to be set to the one on which the original request came on. In that case, we can add specific routing rules to ensure that the replies goes back through the right link. As far as I know, sraddr is likely to be set correctly for TCP servers, while for UDP servers it may not work as expected. Also, if our ISPs don''t do rp_filtering, then we don''t care what link is beeing used for the replies, as soon as the saddr is correct. Vincent.> _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/-- Vincent Jaussaud Kelkoo.com Security Manager email: tatooin@kelkoo.com "The UNIX philosophy is to design small tools that do one thing, and do it well." _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 04:28, Andreas Hasenack wrote:> > Sure. When I said "packet count" before I was thinking about something > along the lines of real traffic balancing, that is, the router somehow > remembering how many packets it sent to each route and choosing the > less used one.That would be one step closer to true load balancing but I do not thing would work for the other end the client. Instead it just splits requests/lookups and does not completely load balance traffic or packets. It is very easy to have uneven loads. For this one day I will look into QoS when the time is right, hoping it may be able to help even out the load? At least from the inside. Once again from the outside load balancing is done via DNS.> > It would be getting responses from an IP it was not expecting response > > from. I would imagine each side to send redirects, and all sorts of > > problems. Like it receiving every other packet and dropping the packets > > in between. > > And breaking stateful firewalls.Yep> > If during a file transfer the route cache is flushed, there is the > > possibility of the rest of the packets going out a different interface. > > Uh oh... It shouldn''t be that simple, what about that 60s timeout for > the cache? It''s very likely to occur during a file transfer.Well that''s more a worse case scenario like manually flushing the cache. I believe the cache gc algorithm takes in consideration routes that are currently in use. So lookups do not occur during a transfer? I have not really seen this to be a big problem. Although I do not have any large files being downloaded, and if I did it could cause problems there. But so far I have not had any problems along those lines.> > Neither does it perfectly or with intelligent algorithms. Neither allow > > you to use all paths for a single transfer. > > Only things like MPPP I guess, for example, or channel bonding, or TQL.Yes, but I believe each of those requires special configurations on both ends, client and ISP.> > So if you have two 1.5 mbs connection, you do not end up with a 3.0 mbs > > line. You do have one internal gateway for both, and if one goes down > > the other can be used. So you do have redundancy, and both lines can be > > used to serve difference requests to different places. > > So it''s more like redundancy/HA with a best effort towards balancing.Yes, or in other terms. My need was a single gateway for my servers although I have two ISPs. The amount of load balancing you get it about the same as the amount of redundancy. You get a partial solution to both, but not a complete solution. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 04:41, Andreas Hasenack wrote:> Em Thu, Nov 21, 2002 at 04:06:26PM -0800, William L. Thomson Jr. escreveu: > > Not necessarily. I have two lines going in completely different > > directions, different private nets, and then via different ISP/Public > > IPs. > > Do you have public servers that are accessed from the outside?Yes that''s basically all I have. I have three lines, two 1.1mps SDSL and one ADSL. My ADSL line is for surfing, and is my main inbound line for normal office traffic. My SDSL lines handle all requests to and from my public servers. So most of the traffic is initiated from the outside. Very little inbound traffic, mostly allot of outbound.> How do you > respond to requests comming down one link? I suppose via the same link, or > using SNAT, otherwise you would have packets with a source IP from ISP1 > traveling through the link to ISP2, right?That''s where it gets a little crazy. Each SDSL line terminates in a router that performs PAT. From the two SDSL routers doing PAT, the requests then make it to the Linux router, which does a second round of PAT, or DPAT. Now when inbound request leave then get SPAT as they leave the Linux router heading for the SDSL routers to be PAT again back to Public IPs. A little excessive, but having two firewalls back to back on either connection is nice. Pretty dam hard to get any packet through both that''s is not supposed to be going through them. Now the hardest part for me to grasp, was how ipchains is able to take a single internal IP and convert it into two different internal/external IPs. Internal in my case, since I do not use Public IPs until hitting and leaving the SDSL routers. There is a particular rule that does this, but I have not found an equivalent using iptables. Although I am sure one exists, or hope one does. Now this is also where Julian''s patches came into play. By using NAT and Julian''s patches, the NAT cache is looked at so the choice of what IP to map the servers IP to is based on the cache. If there is nothing there in the cache, then it goes to the multipath route. Sounds simple, but was a pain for me to grasp, and get working. But it has been working perfectly ever since, or at least as good as can be expected. Since I have implemented true dead gateway detection, nor have I dialed things in specifically with cache timeouts, gcs, etc. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 04:39, Andreas Hasenack wrote:> Em Thu, Nov 21, 2002 at 04:24:06PM -0800, William L. Thomson Jr. escreveu: > > But I have been informed I believe by Julian and others that the load > > balancing, multipath equalize feature can be used even without NAT but > > in a different situation that mine? > > I''m confused as well. Suppose you have two links to the internet, a DMZ, > and an internal network, SNAT''ed. Suppose you have a public web server > in the DMZ (the DMZ is not SNAT''ed). > > How would multipath route (with or without equalize) help here? I mean, > it would only really help if there were connections starting from > the inside (DMZ or SNAT''ed network) to the outside. But:Exactly> - the internal network would probably do many downloads, and not uploadsIn this case the multipath will help out. Splitting the downloads over the different interfaces. This way all lines will be used for downloads. Although the actual load will not be equal on all lines, as previously mentioned. But depending on the amount of routes and weights, the requests/lookups will be properly balanced.> - the web server doesn''t originate traffic, it responds to requests from > the outside world, and it will respond using the same link the request > came in (or not?)This is why I kept saying that you have to load balance from the outside in and the inside out. Inside out was above using multipath. Outside in is load balanced using DNS. Bind allows you to specify multiple IPs for a domain, and a similar weight value. So you can load balance DNS look ups. In my case where everything is even I have two Public IPs for each server in my DNS records. Do a dig on my domain. You will get two IPs, and each time a dns lookup occurs you should get on or the other. In theory every other request gets a different IP. So Bind is trying to load balance things from the outside world. But depending on allot of different variables, the client could end up doing another lookup or using an interface out of the load balancing order. So it''s not perfect. But to complete the load balancing scenario it''s absolutely necessary. Otherwise there is no way to load balance from the outside world without your ISP providing a service or using something like BGP. So load balancing inside out is done via multipath equalize, and load balancing from the outside in is done via DNS, in my case BIND. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 05:26, Vincent Jaussaud wrote:> On Fri, 2002-11-22 at 13:39, Andreas Hasenack wrote: > > - the internal network would probably do many downloads, and not uploads > > > > - the web server doesn''t originate traffic, it responds to requests from > > the outside world, and it will respond using the same link the request > > came in (or not?) > > > No, replies packet will hit the multipath route, and thus may not take > the same link to come back.Actually if you create the proper rules, there are gateways defined in a table prior to the multipath. So if the route is know via cache, it will take a known gateway. Otherwise if it is not, it will be compared to each table, and then finally hit the multipath equalize one and go from there.> But in such situation, the sraddr in the replies packets are likely to > be set to the one on which the original request came on.Which will use the rules if defined. They should be there.> In that case, we can add specific routing rules to ensure that the > replies goes back through the right link.Exactly, but you put them before the multipath route so they are looked at and compared to first.> As far as I know, sraddr is likely to be set correctly for TCP servers, > while for UDP servers it may not work as expected.Haven''t really seen and problems there. So far from my experience my DNS queries use both TCP and UDP. I have not seen and problems effecting UDP and not TCP. But DNS lookups are quick and short, as most UDP services, so there could be some problems there that are not effecting me in my current scenario.> Also, if our ISPs don''t do rp_filtering, then we don''t care what link is > beeing used for the replies, as soon as the saddr is correct.Not sure here, I never got into that aspect. I just assumed they did do rp filtering. I played with turing rp filtering on in the Linux router, but in the end left it on or what ever the default is. -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Em Fri, Nov 22, 2002 at 10:05:25AM -0800, William L. Thomson Jr. escreveu:> > So it''s more like redundancy/HA with a best effort towards balancing. > > Yes, or in other terms. My need was a single gateway for my servers > although I have two ISPs. The amount of load balancing you get it about > the same as the amount of redundancy. You get a partial solution to > both, but not a complete solution.I just found this patch, has anybody already played with it? ftp://sliepen.warande.net/pub/eql/patch-2.4.18-2.gz Excerpt: Load balancing needed a slight adjustment to the unpatched linux kernel, because of the route cache. Multipath is an option already found in the old 2.1.x kernels. However, once a packet arrives, and it matches a multipath route, a (quasi random) device out of the list of nexthops is taken for its destination. That''s okay, but after that the kernel puts everything into a hash table, and the next time a packet with the same source/dest/tos arrives, it finds it is in the hash table, and routes it via the same device as last time. The adjustment I made is as follows: If the kernel sees that the route to be taken has got the ''equalize'' flag set, it not only selects the random device, but also tags the packet with the RTCF_EQUALIZE flag. If another packet of the same kind arrives, it is looked up in the hash table. It then checks if our flag is set, and if so, it deletes the entry in the cache and has to recalculate the destination again. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 2002-11-22 at 19:47, William L. Thomson Jr. wrote:> Actually if you create the proper rules, there are gateways defined in a > table prior to the multipath. So if the route is know via cache, it will > take a known gateway. >Yes.> Otherwise if it is not, it will be compared to each table, and then > finally hit the multipath equalize one and go from there. > > > But in such situation, the sraddr in the replies packets are likely to > > be set to the one on which the original request came on. > > Which will use the rules if defined. They should be there.Yes.> Haven''t really seen and problems there. So far from my experience my DNS > queries use both TCP and UDP. I have not seen and problems effecting UDP > and not TCP. But DNS lookups are quick and short, as most UDP services, > so there could be some problems there that are not effecting me in my > current scenario. >That''s good news. Normally, TCP port 53 is only used for zone transferts, while UDP port is used for DNS queries. If you don''t see anything strange in your logs, such as "connection reset by peer", then it means it works fine. Well, you probably already figured this out :-) Did you tried with other UDP services ? (like timed for instance) If it works for simple UDP servers like timed, then I would definitly consider linux as the best OS in the world ;-)> > Also, if our ISPs don''t do rp_filtering, then we don''t care what link is > > beeing used for the replies, as soon as the saddr is correct. > > Not sure here, I never got into that aspect. I just assumed they did do > rp filtering. I played with turing rp filtering on in the Linux router, > but in the end left it on or what ever the default is.Anyway, if the do, it should be simple to ask them to disable rp_filtering. I had some talks with my ISP some days ago, and according to them, most ISPs don''t bother with rp_filtering, except for private netblocks trying to ''escape'' in case of a broken NAT configuration. -- Vincent Jaussaud Kelkoo.com Security Manager email: tatooin@kelkoo.com "The UNIX philosophy is to design small tools that do one thing, and do it well." _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi guys, How to delete the ip route cache. I have changed an IP for my T1 table (I did everything from http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html) and it was working fine till I cahnged my ISP (and of couse IP address). I tried to use ip route flush cache and ip route flush cache x.x.x.x (old IP address) but it doesn''t help me. I still cannot use the new one IP address. Thank you very much for help. Rimas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/