Hi There, We currently using iproute2 for load balancing. However we need more upload speed as we load balance over 3 dsl lines. I''ve been looking for a way to combine the upload speed to more faster. Found a site called www.upstreaminter.net where these guys bond the adsl channels to improve uploads, Since downloading is problem as its need to know the ip address of the downloader they use VPN to their existing infrastruture to improve download speeds to a single IP. Im not interested in doing that though, Our company uses alot of mail to our smarthost and the upload is only 256k on the 4Mbit Line These guys talking about using USB modems , some researching says your ISP must support MLPPP , Anyone with any expierencing bonding adsl modems with a standard kernel and distro? Most nix''s come with RP-PPPOE, will this work? Thanks Sew
On 10/11/07 10:17, the sew wrote:> We currently using iproute2 for load balancing. However we need more > upload speed as we load balance over 3 dsl lines.*nod*> I''ve been looking for a way to combine the upload speed to more > faster. Found a site called www.upstreaminter.net where these guys > bond the adsl channels to improve uploads, Since downloading is > problem as its need to know the ip address of the downloader they use > VPN to their existing infrastruture to improve download speeds to a > single IP. Im not interested in doing that though, Our company uses > alot of mail to our smarthost and the upload is only 256k on the > 4Mbit LineNow you are entering some more complex and interesting territory. The problem you have with traditional load balancing that the LARTC suggests for SOHO users is that you have multiple external IP addresses. Thus any single stream is limited to the speed of the link that it is using. If you are using multiple links from a single provider who is willing to work with you, you could do something like I did with the two 1.1 Mbps SDSL links that I have aggregated in to one 2.2 Mbps link. I can easily, and routinely do, push and / or pull 1.9+ Mbps on a single ftp stream. I have a small globally routable DMZ behind the two globally routable IPs for my SDSL links. My ISP has a router here on my premise and is using OSPF to load balance across the SDSL links. (A little ASCII drawing comes to mind.) : : +--------+ +--------+ +-----+ +-----+ | SDSL 1 +-----+ SDSL 1 | +------+ | LAN +-----| NAT +-----+ CPERTR | | ISPRTR +-----+ INet | +-----+ +-----+ | SDSL 2 +-----+ SDSL 2 | +------+ : +--------+ +--------+ Your domain : ISP domain : To recap, the following interfaces all have globally routable IP addresses: - NAT outside - CPE Router inside - CPE Router SDSL 1 outside - CPE Router SDSL 2 outside - ISP Router SDSL 1 inside - ISP Router SDSL 2 inside - ISP Router INET outside The only interfaces that have private IPs are the LAN and the NAT inside. As such the CPE Router and ISP Router load balance across the two SDSL links. The world sees my office as being from the NAT outside and does not give a darn about any thing in between. We even have OSPF set up to realize if one of the SDSL links goes down to re-route across the remaining link.> These guys talking about using USB modems , some researching says > your ISP must support MLPPP ,I don''t think the technology of the modem really matters all that much. However you will require support from your provider, be it PPP Multi-Link, or load balancing like I''m doing, or something similar.> Anyone with any expierencing bonding adsl modems with a standard > kernel and distro?Not speciffically. Though, given a scenario like mine, I don''t see why it would matter as long as you can get Equal Cost Multi-Path to work correctly. My provider required that their own equipment be used on customer premise because they were using OSPF an interior routing protocol to my premise. So my ISP extended their premise to be located at my office so that the IRP stays with in their control. Heh, it works and I get aggregated speeds and redundancy, so I''m not going to complain. Seeing as how the CPE Router above is a Cisco 3640 I can''t say any thing for sure about Linux, but it should be able to do the job just fine.> Most nix''s come with RP-PPPOE, will this work?I don''t know if RP-PPPoE will do PPP Multi-Linking or not. Though I don''t think that it is an aspect of RP-PPPoE, but rather the underlying PPP implementation on your system. It is my understanding that RP-PPPoE just passes traffic from "over Ethernet" to your PPP daemon. So it is probably really up to your PPP daemon. Granted you will have to have your providers support to be able to do this too. Another option you could look in to if you have access to a system with high speed (read OC-1 or better) internet access somewhere is tunnels. That is create an IP in IP or GRE tunnel from your external IP to a co-located system and then load balance across the tunnels and NAT at the co-located system. Your router and the co-located system would need to know how to spread the bandwidth across the tunnels in to you. : : +--------+ +--------+ : +-----+ | ADSL 1 +-----+ ADSL 1 | +------+ +------+ | LAN +-----+ CPENAT | | ISPRTR +-----+ COLO |-----+ INet | +-----+ | ADSL 2 +-----+ ADSL 2 | +------+ +------+ +--------+ +--------+ : : : Your Domain : Shared Domain : Public Domain : : In this scenario, you would set up a tunnel from the CPE NAT Router ADSL 1 outside to the COLO inside and from the CPE NAT Router ADSL 2 outside to the COLO inside. You would add a route to the COLO box via the ISPs upstream gateway to each ADSL connection and use the COLO box as your default gateways. I.e. the CPD NAT Router would have a routing table similar to this: 0.0.0.0/0 <Tunnel 1 internal IP> 0.0.0.0/0 <Tunnel 2 internal IP> <COLO/32> <ADSL 1 GW> <COLO/32> <ADSL 2 GW> This way computers on your LAN will use the CPE NAT Router as their default gateway. The CPE NAT Router will really have two default gateways, the far end of the tunnels. The Co-Located box is what will really be doing the NATing for the LAN. As far as subnets are concerned, this is what the network will look like. +--------+ +--------+ +--------+ | ...1.2 +-----| ...1.1 | +------+ | ...0.x +-----+ ...0.1 | | Global +-----+ INet | +--------+ + ...2.2 +-----| ...2.1 | +------+ |--------+ +--------+ The CPE Router (that is not NATing) will use the Co-Located box tunnels as its two default gateways and know about the LAN and tunnel subnets. The CPE Router will have routes to the globally routable IP of the Co-Located box via the ISPs upstream gateways but not use the ISPs upstream gateways as default routes. The Co-Located box will use the tunnels as multiple routes to get to the LAN subnet and know about the tunnels and LAN subnet. I know that this type of set up is extremely convoluted but will work and it will work with out the support of the ISP(s). Heck, you could even use multiple different ISPs each with different layer 1 technology. Thus you would really have redundancy in case one of the links goes down. You could even expand this modle to include a second different co-located box to allow for even more redundancy. (I''m going to shut up now.) Grant. . . .
On 10/11/07 11:18, Grant Taylor wrote:> Another option you could look in to if you have access to a system > with high speed (read OC-1 or better) internet access somewhere is > tunnels. That is create an IP in IP or GRE tunnel from your external > IP to a co-located system and then load balance across the tunnels > and NAT at the co-located system.I would recommend that you not use an encrypting tunnel to reach the Co-Located box as you are dealing with traffic that is going to go across the internet at large any way, just let it go out. However if you want to take this idea to connect multiple sites together, you obviously would want to consider encrypting VPNs. Grant. . . .
Hi Grant, Thanks for all the info, I like your SDSL option with the ospf, exactly what I would like Our situation is quite simple, Our ISP is telkom, one main Provider for our Country, they will not touch anything other than the standard services they provide. No SDSL, only ADSL. with standard pppoe. I guess i''m bit stuck there. I could do it to our co-location like u suggest, but that will add more costs to line rental where lines here is quite pricey. That would be a last resort for me I''ve seen the bonding in action,but havent seen the configs as they kept it quite secret as its a "new" thing. Looks like I''m gonna have to try this route on my own. I could not find much info on this. I will give it a try with 2 x USB modems and tell pppd to use multilinking and see if I can attach the device as one, I''m sure downloads will be speed to a single dsl only, but upload should be double. I think I might get a bit stuck as they might do the bonding to a co-location like you said and just resell it, will give it a #!/bin/bash anyway Thanks again for your info Sew . On 10/11/07, Grant Taylor <gtaylor@riverviewtech.net> wrote:> On 10/11/07 11:18, Grant Taylor wrote: > > Another option you could look in to if you have access to a system > > with high speed (read OC-1 or better) internet access somewhere is > > tunnels. That is create an IP in IP or GRE tunnel from your external > > IP to a co-located system and then load balance across the tunnels > > and NAT at the co-located system. > > I would recommend that you not use an encrypting tunnel to reach the > Co-Located box as you are dealing with traffic that is going to go > across the internet at large any way, just let it go out. However if > you want to take this idea to connect multiple sites together, you > obviously would want to consider encrypting VPNs. > > > > Grant. . . . > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >
On 10/11/07 13:12, the sew wrote:> Thanks for all the info, I like your SDSL option with the ospf, > exactly what I would like*nod* I like it too. :)> Our situation is quite simple, Our ISP is telkom, one main Provider > for our Country, they will not touch anything other than the standard > services they provide. No SDSL, only ADSL. with standard pppoe. I > guess i''m bit stuck there. I could do it to our co-location like u > suggest, but that will add more costs to line rental where lines here > is quite pricey. That would be a last resort for meYou can do the tunneling like I was referring to across any thing that will carry IP, it does not have to be SDSL or any other form of connection in particular. What do you mean by "... that will add more costs to line rental ..."? Are you trying to say that you pay for bandwidth, so you would be doubling what passes through your co-located box?> I''ve seen the bonding in action,but havent seen the configs as they > kept it quite secret as its a "new" thing. Looks like I''m gonna have > to try this route on my own. I could not find much info on this. I > will give it a try with 2 x USB modems and tell pppd to use > multilinking and see if I can attach the device as one, I''m sure > downloads will be speed to a single dsl only, but upload should be > double. I think I might get a bit stuck as they might do the bonding > to a co-location like you said and just resell it, will give it a > #!/bin/bash anywayI don''t think you need to worry about using a USB modem persay, if the provider is using standard RFC 1483 (2684) (Multiprotocol Encapsulation over ATM Adaptation Layer 5) encapsulation to carry the ethernet frames back to the router. You could easily use an external ADSL ATM to Ethernet bridging modem with out a problem. Now if the provider is wanting to do something more special such that there are two different ATM connections with the ATM stack on the Linux system using the USB ADSL ATM modems, then yes you would need to use the USB modems. However I think it would be much more complex to try to multiplex across two different ATM connections with out an intermediary IP layer. With regards to PPP Multi-Linking, things have been standardized for a long time. Well as standardized as things can be when there is little call for it. Basically what is done is two separate PPP connections are made between the PPP daemons on each end where the PPP daemon knows that some traffic for a given logical interface will pass down both connections. The PPP daemon splits the traffic that it sends and combines the traffic that it receives. In short this is much like striping data (RAID 0) across two drives except that it is used for networking. I think you will need to find out some more information to know what direction to go to get this to work. Ideally if your provider does support PPP Multi-Linking there is no reason (that I know of) why you could not use this across PPPoE. If more providers supported this, this would be a very good thing. Heck I think you can even get the PPP daemon to realize that one of the links is down and to fall back to the single link, thus you have some redundancy.> Thanks again for your infoYou are welcome. Grant. . . .
Hi Grant,
You were right, the usb modem mehtod was just a way for me to see how
usb modems works in linux, u basicly add new firmware and load the
driver and it sees your usb modem also as a ethernet device and u use
pppd with that :-)
Im trying your option below to get increased upload, but no luck yet
from your ascii , mine looks similar
+--------+ +--------+
+--------+ | ...245.18 +-----| ...77.1 | +------+
| ...0.x +-----+ ...0.1 | | COLO +-----+ 101.x |
+--------+ + ...245.19 +-----| ...78.1 | +------+
|--------+ +--------+
the background how I got this setup is that 245.18 and 19 has pppoe
into my ISP, then I made a pptp tunnel to COLO on 2 diffrent ips and
got my new ips which is 245.18 Ptp 77.1 and 245.19 Ptp 78.1.
All this works fine, I''ve even add multiple routes to 101.x via dev
ppp0 nexthop dev ppp1, but this just load balances the links, not
increase uploads
Is it possible at all to route all upload via both links at the same
time, maybe disable the cached routes, let it just change routes the
whole time?
Thanks agian for your help
Sew
On 10/25/07 00:49, the sew wrote:> You were right, the usb modem mehtod was just a way for me to see how > usb modems works in linux, u basicly add new firmware and load the > driver and it sees your usb modem also as a ethernet device and u use > pppd with that :-)*nod* That''s the experience that I had too. I will say that I perceived performance to be a bit better with the USB ADSL modem than I did with external bridging modems. I attributed this perceived performance improvement to be related to more intelligent caching of data on the system with the IP stack with comparison to external modems that did not know if there would be any more packets coming or not thus the external would have to wait longer before transmitting packets. I have no numbers to back this up, but I did perceive a performance improvement both under Linux and Windows with my USB ADSL modem and under Windows with my PCI ADSL modem.> Im trying your option below to get increased upload, but no luck yet > > from your ascii , mine looks similar+-----------+ +---------+ +--------+ | ...245.18 +-----+ ...77.1 | +-------+ | ...0.x +-----+ ...0.1 | | COLO +-----+ 101.x | +--------+ | ...245.19 +-----+ ...78.1 | +-------+ +-----------+ +---------+ (Slight clean up.)> the background how I got this setup is that 245.18 and 19 has pppoe > into my ISP, then I made a pptp tunnel to COLO on 2 diffrent ips and > got my new ips which is 245.18 Ptp 77.1 and 245.19 Ptp 78.1. > > All this works fine, I''ve even add multiple routes to 101.x via dev > ppp0 nexthop dev ppp1, but this just load balances the links, not > increase uploadsWhere are you doing your NATing to the world? Are you NATing on your PPtP tunnels or on the COLO system?> Is it possible at all to route all upload via both links at the same > time, maybe disable the cached routes, let it just change routes the > whole time?If you are doing your NATing on the COLO system and you add two routes to your internal network via the two PPtP tunnels, you should be able to equal cost multipath route across both PPtP tunnels to achieve increased bandwidth. The key part is that both tunnels have to appear to the world as a single external IP.> Thanks agian for your helpYou are welcome. Grant. . . .
+-----------+ +---------+
+--------+ | ...245.18 +-----+ ...77.1 | +-------+
| ...0.x +-----+ ...0.1 | | COLO +-----+ 101.x |
+--------+ | ...245.19 +-----+ ...78.1 | +-------+
+-----------+
+---------+>
> (Slight clean up.)
>
>
> Where are you doing your NATing to the world? Are you NATing on your
> PPtP tunnels or on the COLO system?
Currently I''m natting on 0.1 ( -o ppp+ -j MASQUERADE ) , This was
setup by default as I did not want 0.x to be routed. I''ve however
taken off the natting, and added a route for 0.20/32 dev ppp62 nexthop
dev ppp32 ( the 2 vpn interfaces) at COLO and obviously same nexthop
routes at 0.1 for 101.x
testing from 0.20, I scp a tar file over to 101.20 , still goes via
one line at time, the route cache which I disabled, just reroute it
the whole time( about every 5 sec) via the diffrent uplink, but not to
our result we want
I use sysstat to check the speeds and tcpdump verified I its from 0.20
-> 101.20 ssh
>
> If you are doing your NATing on the COLO system and you add two routes
> to your internal network via the two PPtP tunnels, you should be able to
> equal cost multipath route across both PPtP tunnels to achieve increased
> bandwidth. The key part is that both tunnels have to appear to the
> world as a single external IP.
I understand now very clearly the key part.
My problem must be the tunnel, im sure im messing up, the equal cost
multipath routing , am I using the right utitily? , still iproute2
right, or is iptables gonna play big part here as well?
S
On 10/25/2007 1:06 PM, the sew wrote> +-----------+ +---------+ > +--------+ | ...245.18 +-----+ ...77.1 | +-------+ > | ...0.x +-----+ ...0.1 | | COLO +-----+ 101.x | > +--------+ | ...245.19 +-----+ ...78.1 | +-------+ > +-----------+ +---------+ > > Currently I''m natting on 0.1 ( -o ppp+ -j MASQUERADE ) , This was > setup by default as I did not want 0.x to be routed. I''ve however > taken off the natting, and added a route for 0.20/32 dev ppp62 > nexthop dev ppp32 ( the 2 vpn interfaces) at COLO and obviously same > nexthop routes at 0.1 for 101.xOk, good. The NATing at 0.1 was part of the problem.> testing from 0.20, I scp a tar file over to 101.20 , still goes via > one line at time, the route cache which I disabled, just reroute it > the whole time( about every 5 sec) via the diffrent uplink, but not > to our result we wantHum. I would question the routes then.> I use sysstat to check the speeds and tcpdump verified I its from > 0.20 -> 101.20 sshOk.> I understand now very clearly the key part. > > My problem must be the tunnel, im sure im messing up, the equal cost > multipath routing , am I using the right utitily? , still iproute2 > right, or is iptables gonna play big part here as well?To the best of my knowledge you will still be using the "ip" command from iproute2. Will you please show us your routing table and / or the ip route command you are using to set up your ecmp route? Grant. . . .