I have a dual-homed firewall. It has 2 Internet connections, provided by different ISPs (each with an associated IP address). The 2 Internet connections are connected to the same physical interface. The 2 Internet connections do NOT have equal bandwidth How do I configure the SNAT/MASQ and ensure sharing of the gateways with the correct ratio of usage and with the correct source IP address? I know how to use the ''ip'' commands to configure gateway sharing according to my defined ratios and ensure that packets go out of the correct gateway according to their source address. However, I need to configure SNAT-ed or MASQ-ed in such a way that it shares the gateways according to my defined ration and so that it sets the source address correctly for the gateway. How do I do this? Can someone please point me in the correct direction for the SNAT/MASQ setup with iptables/iproute2? Thanks! Simon _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Sunday, 29 September 2002, at 11:24:03 -0700, Simon Matthews wrote:> I have a dual-homed firewall. It has 2 Internet connections, provided by > different ISPs (each with an associated IP address). The 2 Internet > connections are connected to the same physical interface. The 2 Internet > connections do NOT have equal bandwidth >> How do I configure the SNAT/MASQ and ensure sharing of the gateways with > the correct ratio of usage and with the correct source IP address? >ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \ nexthop via $CONN2_IP dev $ETHX weight $Y weight(s) are the relative bandwidths of the connections. If the first is a 2 Mbps line and the second a 512 Kbps one, $=4 and $Y=1, for example. What matters is the ratio, not the actual bandwidth.> I know how to use the ''ip'' commands to configure gateway sharing according > to my defined ratios and ensure that packets go out of the correct gateway > according to their source address. >The above defines routing. And SNAT/MASQ is something done afterwards, so you could SNAT/MASQ traffic going to each connection differently. What I don''t know right now is how to determine to which Internet connection are packets going in your setup with only an ethernet card for both. With a card for each Internet connection it is simple. Hope this helps. -- Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Woody (Linux 2.4.19-pre6aa1) _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> > I have a dual-homed firewall. It has 2 Internet connections, provided by> > different ISPs (each with an associated IP address). The 2 Internet > > connections are connected to the same physical interface. The 2 Internet > > connections do NOT have equal bandwidth > > How do I configure the SNAT/MASQ and ensure sharing of the gateways with > > the correct ratio of usage and with the correct source IP address? > > > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \ > nexthop via $CONN2_IP dev $ETHX weight $Y Note that this only shapes outgoing traffic and also relies on your ISPs to NOT do the ingress filtering that they''re really supposed to do. If most of the traffic at your site is going out this might be reasonable. For most people, most of the traffic is coming in, e.g., behind your firewall are clients who want to download data. The bandwidth going out is mostly acks, which is very small compared to that coming in. Which interface traffic comes in on depends on which IP address it''s sent to, which depends on which address the original request came from. That''s related to the nat decision but there''s no way to make the "right" decision. You can divide the original requests in some given proportion, but you don''t know when you do that how much traffic each one will generate. So, for instance, the clients behind your firewall make 10 http requests. You nat 8 to the fast interface and 2 to the slow one. But then it turns out that the 8 you send to the fast one all result in small amounts of data while one of the two sent to the slow interface results in a large amount. You might be better off just using the fast one, or perhaps using the slow one only for stuff that you can be sure will fit comfortably in its bandwidth. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Sun, 29 Sep 2002, Don Cohen wrote:> > > I have a dual-homed firewall. It has 2 Internet connections, provided by > > > > > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \ > > nexthop via $CONN2_IP dev $ETHX weight $Y > > Note that this only shapes outgoing traffic and also relies on your > ISPs to NOT do the ingress filtering that they''re really supposed to do. > > If most of the traffic at your site is going out this might be > reasonable. For most people, most of the traffic is coming in, e.g., > behind your firewall are clients who want to download data. The > bandwidth going out is mostly acks, which is very small compared to > that coming in. > > off just using the fast one, or perhaps using the slow one only for > stuff that you can be sure will fit comfortably in its bandwidth.OK, this may be a reasonable approach, but how do I force it initiate connections from the "fast" interface, yet allow it to fail over to the slow interface if the sytem removes the route to the fast gateway because it has detected that it is not responding? Simon _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Simon Matthews writes: > OK, this may be a reasonable approach, but how do I force it initiate > connections from the "fast" interface, yet allow it to fail over to the > slow interface if the sytem removes the route to the fast gateway because > it has detected that it is not responding? Off hand I don''t know anything built in for this (I look forward to hearing an answer from someone who does), but I don''t think this is really what you want anyway. It''s not as if your link is the only one that could fail! If ISP1''s upstream link fails then you want to use ISP2 for all traffic other than that intended for ISP1 itself. And of course, problems further upstream prevent you from reaching certain addresses but not others, and you don''t really know which without a global view of the routing. I think the "right" solution involves monitoring the traffic. There''s a wide range of things you could do, the simplest being simply detecting that the link is not responding. You could also try to detect tcp retransmits, measure RTT, aggregate data to measure how well individual connections are working, further aggregate data to determine which addresses blocks are working well and which poorly, etc. Then use that data to decide which of your links to use for a given destination. I actually sent a proposal to this list that I think provides a good solution to the general problem: an extension to TCP (possibly even IP) that supports multiple addresses/ports. This would even allow you to switch addresses in the middle of a connection. I think what I described before applies more to the machine on the other side of your connection, which now would know both of your addresses. Whenever it does a tcp retransmit it switches the address. It therefore tends to stay on the one that works most reliably. (Perhaps this algorithm could be improved to take speed into account too.) This discussion points out that something similar should be done on your end: you should switch the output interface you use when you retransmit. Of course this is not yet implemented. It''s on my queue, but not close to the beginning. I''d be glad if someone out there could beat me to it. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Don Cohen wrote:>I actually sent a proposal to this list that I think provides a good >solution to the general problem: an extension to TCP (possibly even >IP) that supports multiple addresses/ports. This would even allow you >to switch addresses in the middle of a connection. I think what I > >SCTP actually supports this already; look it up -- its quite a bit different from TCP but allows you to do all the same types of things, with more options. That said, a Zebra (routing software) plugin that would run iptables scripts would be all you''d need in many cases. -- Michael T. Babcock C.T.O., FibreSpeed Ltd. http://www.fibrespeed.net/~mbabcock _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Sunday, 29 September 2002, at 22:18:30 -0700, Don Cohen wrote:> > ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \ > > nexthop via $CONN2_IP dev $ETHX weight $Y > > Note that this only shapes outgoing traffic and also relies on your > ISPs to NOT do the ingress filtering that they''re really supposed to do. >Just a note. The above routing doesn''t prevent you from applying SNAT/MASQ to the outgoing traffic, at least not when you have an ethernet card for each connection (not the case) and you can know through each one the traffic will go out. So adding another ethernet card and a couple of "iptables" rules can avoid problems with ISPs filtering "alien" incoming traffic :) -- Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Woody (Linux 2.4.19-pre6aa1) _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Mon, 30 Sep 2002, Don Cohen wrote:> Simon Matthews writes: > > OK, this may be a reasonable approach, but how do I force it initiate > > connections from the "fast" interface, yet allow it to fail over to the > > slow interface if the sytem removes the route to the fast gateway because > > it has detected that it is not responding? > > Off hand I don''t know anything built in for this (I look forward to > hearing an answer from someone who does), but I don''t think this is > really what you want anyway. It''s not as if your link is the only one > that could fail!Don, there are some kernel patches (already installed on my system) that support dead gateway detection and static routes. "Static" means that the routes are not forgotten when the system removes an interface because the gateway is not working. But the problem remains: how to handle this in iptables MASQ/SNAT commands? One can postulate that if the interface is removed because the gateway is dead, then the MASQ command will use the source related to the other gateway. However, the question now is: how to force the system to use the source address related to the "fast" gateway under normal operation while allowing a failover to the the slow gateway? Simon _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Mon, 30 Sep 2002, Michael T. Babcock wrote:> Don Cohen wrote: > > > That said, a Zebra (routing software) plugin that would run iptables > scripts would be all you''d need in many cases.The ISP that provides the "fast" connection won''t provide any IGP routing information (RIP, OSPF, etc), so I don''t think this is possible.> >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> ip route add default nexthop via $CONN1_IP dev $ETHX weight $X \ > nexthop via $CONN2_IP dev $ETHX weight $Y >Would this technique work for more than two gateways? How many nexthop clauses are allowed? Is there a limit? thanks - Greg Scott _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello, On Mon, 30 Sep 2002, Michael T. Babcock wrote:> Don Cohen wrote: > > >I actually sent a proposal to this list that I think provides a good > >solution to the general problem: an extension to TCP (possibly even > >IP) that supports multiple addresses/ports. This would even allow you > >to switch addresses in the middle of a connection. I think what IYes, we can implement it as separate IP protocol :) Of course, at the beginning the idea may sound too stupid, we have to change that. May be there is already solution for that? A "simple" tunnel without encryption that will support failover and balancing of the negotiated traffic, ability to negotiate multiple IPs for each endpoint. Of course, there should be some problems with the proper tunneling of this traffic in each end, see how difficult is routed the IPSec traffic. Each endpoint will do failover detection of all negotiated links and will do balancing (if desired) over these links, based on relative ratio. This tunnel should be transparent to the upper layers (TCP/UDP/ICMP/SCTP).> SCTP actually supports this already; look it up -- its quite a bit > different from TCP but allows you to do all the same types of things, > with more options.But this feature is only for SCTP. We want the traffic from one multihomed router to use multiple links when talking to another router, both understanding this "our new" IP tunneling protocol. I see it in this way: when such packet is received, we decapsulate it and place it on the expected interface. As result, the upper layers will see the packet on the right input interface even if it is received on another input interface (for example, if it is the only alive). Regards -- Julian Anastasov <ja@ssi.bg> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Simon, On Mon, 2002-09-30 at 00:06, Simon Matthews wrote:> > OK, this may be a reasonable approach, but how do I force it initiate > connections from the "fast" interface, yet allow it to fail over to the > slow interface if the sytem removes the route to the fast gateway because > it has detected that it is not responding?The way I see things it basically works like this, the out going traffic will get balanced via the next hop and weights. So based on the weight values, one will be used more than the other. Now to get the same thing to happen from the outside world, you need to load balance your dns. Look into setting up multiple dns records, and possible weight values as well. It''s pretty safe to say that if the fast connection goes down, it will be unavailable for the dns system to load balance, and all should divert to the line that is up. I would assume it to be best if using internal dns servers, but may work better with outside dns servers if possible? Now that''s from the outside. You mentioned using dead gateway detection to remove, and hopefully add back the route when it becomes available again. Which will be needed so that the route is no longer used from the inside. Causing the route the is available to be the only one used from the inside. This switch over can be seamless, but if you are say streaming media, it will be interrupted, as an example. Other things like http, will most likely be fine, stateless things won''t care, others will. You real problem is for say if a large download or something is diverted from a fast line to a slow line. I would imagine it to be somewhat unlikely since, the fast one may but in during the lookup process. You may want to look into some sort of traffic control or shaping, which may be a good idea in case that the fast line is down. You may really want to control the left over bandwidth for important stuff. Someone else also mentioned this. TC and QoS is not an area I have much experience in, so you will have to look to other there. In the end the over all load balancing logic is very general, and not extremely smart/advanced. It would be nice if it was. You have the next hops from the inside, and dns for the outside. Not to much beyond that. Beyond that you will have to look to other things to really dial things in, in my opinion. Not sure if this helps or not, but hopefully it will. Good luck -- Sincerely, William L. Thomson Jr. Support Group Obsidian-Studios Inc. 439 Amber Way Petaluma, Ca. 94952 Phone 707.766.9509 Fax 707.766.8989 http://www.obsidian-studios.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/