Colin Anderson
2006-Apr-06 14:44 UTC
[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]
I was given the challenge recently of creating a LAN-LAN bridge between two buildings several hundred metres from each other, using only existing Cat 3 wiring and without having to resort to an expensive and finicky 5 Ghz wireless link. I was able to create a 90 megabit link for about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black Box VDSL Ethernet Extender, which supports 30 megabits over a single twisted pair. This is relevant to the list because I have seen many posts with people facing the same kind of challenge deploying Asterisk in remote locations. In my case, I am running ~40 Snom 360's from the remote building to where my Asterisk server is, and it's working fine. Hope this helps someone. Yay Linux. ---------------------------------------------------------------------------- --------------- HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable up to 1.9 km ---------------------------------------------------------------------------- --------------- Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders and Linux bonding, it is possible to create a protocol-independent, redundant, high speed Ethernet bridge using common Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9 km (6,233 feet), with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of the link is that of a regular bridged LAN, and is suitable for high speed or latency sensitive applications such as Voice over IP. HARDWARE: 2 X 4-PCI slot PC's 6 X Black Box LB300A VDSL Ethernet Extenders 8 X PCI Ethernet cards http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946 RJ-12 crimper RJ-12 6-conductor plugs (will work with RJ-11) SOFTWARE: CentOS 4.1 or any Linux distro that supports bonding and bridging 1. Install hardware on both machines In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX cards. The reason I selected this board was because there is 5 avaliable PCI slots. Install the cards 0-3 in slots 0-3. 2. Disable onboard hardware Because the NIC's will require an IRQ each, disable the following resources: a) Onboard NIC b) Onboard sound c) Onboard USB d) Onboard legacy ports e) Set "F1 on error" or "halt on errors" to off in the BIOS so you can boot headless 3. Install your OS I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a "server" with everything disabled except "development". I also set the detected NIC's to "unconfigured" - no DHCP, no start at boot. 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's are running OK 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface to the LAN and eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface will be bridged together. Easy enough - plug in a network cable into a NIC and run dhclient. Once you get an IP from your DHCP server you can use ifconfig to determine whihc one got the IP. Repeat for each NIC. Once you have detemined which physical device is which, I used a sharpie to label each port so I wouldn't get confused. 6. Install bridging support, if your distro does not support it OOB. Fortunately, CentOS does so no problem there. 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum install bridge-utils" 8. Configure the bonding config In /etc/sysconfig/network-scripts make valid entries for your bonded device and each slave. ifcfg-bond0: DEVICE=bond0 ONBOOT=no BOOTPROTO=none ifcfg-eth1/2/3: DEVICE=eth1/2/3 (replace as nessisary) USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none 9. Startup script /etc/rc.d/rc.local: modprobe bonding miimon=1* brctl addbr br0 brctl addif br0 eth0 sleep 10s** brctl addif br0 bond0 *miimon=1 means disable the port after one second on link down. This is the redundant part. If one of your links fail, the other two will keep working. ** The 'sleep' is to allow the bridge to stabilize. In testing, the bridge did not work if each brctl statement immediately followed each other. 10. Test without VDSL extenders Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the source LAN. Plug the 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd PC's eth0 with a crossover cable, or plug the second PC into a switch then plug your devices into the same switch. Make sure both PC's have a keyboard and monitor so you can see what's going on. Finally, plug crossover cables into each PC's eth1, eth2 & eth3 On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic right away. Do some test pings from and to your devices, and try to get an IP address (dhcp) from the "remote" LAN. Troubleshooting: *Link up at 100base-T FD? *ifconfig eth1, eth2, eth3 shows as SLAVE? *same ifconfig shows TX and RX values > 0 *try mii-tool -F 100baseT-FD 11. Rack up and plug in the PC's in the source and destination LAN's. Terminate the Cat 3 with the RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely brief manual that comes with the LB-300 so straight-through on all pins was used. Undoubtedly, it follows the telco standard of the middle pins, but this is untested. Plug the freshly-terminated plugs into the RJ12 port on the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45 ports on the LB-300 into eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to "Loc" (LOCAL) on the source LAN's LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300. 12. Power up and watch for the blinkenlights Troubleshooting: *Same as troubleshooting above? *Look at the lights on the top of the LB-300's. They should give you an indication as to what's happening to the LB-300 according to the troubleshooting chart on the LB-300 manual. 13. Enjoy your link!
David Antognini
2006-Apr-06 15:31 UTC
[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]
Or, you could use a Corinex Phone Line Bridge which runs 128Mbits up to 2000 feet. They also have a co-ax version which is 200mbits and goes 4000 feet... About $300 for both ends. http://www.corinex.com/web/docx.nsf/(w)/eng-corinex_av_phoneline_ethernet_bridge> I was given the challenge recently of creating a LAN-LAN bridge between two > buildings several > hundred metres from each other, using only existing Cat 3 wiring and without > having to resort > to an expensive and finicky 5 Ghz wireless link. I was able to create a 90 > megabit link for > about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black > Box VDSL Ethernet > Extender, which supports 30 megabits over a single twisted pair. > > This is relevant to the list because I have seen many posts with people > facing the same > kind of challenge deploying Asterisk in remote locations. In my case, I am > running > ~40 Snom 360's from the remote building to where my Asterisk server is, and > it's working > fine. > > Hope this helps someone. Yay Linux. > > ---------------------------------------------------------------------------- > --------------- > HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable > up to 1.9 km > ---------------------------------------------------------------------------- > --------------- > > Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders > and Linux bonding, > it is possible to create a protocol-independent, redundant, high speed > Ethernet bridge using common > Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9 > km (6,233 feet), > with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of > the link is that of > a regular bridged LAN, and is suitable for high speed or latency sensitive > applications such as Voice > over IP. > > HARDWARE: > > 2 X 4-PCI slot PC's > 6 X Black Box LB300A VDSL Ethernet Extenders > 8 X PCI Ethernet cards > http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946 > RJ-12 crimper > RJ-12 6-conductor plugs (will work with RJ-11) > > SOFTWARE: > > CentOS 4.1 or any Linux distro that supports bonding and bridging > > > 1. Install hardware on both machines > > In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX > cards. The reason I selected > this board was because there is 5 avaliable PCI slots. Install the cards 0-3 > in slots 0-3. > > 2. Disable onboard hardware > > Because the NIC's will require an IRQ each, disable the following resources: > > a) Onboard NIC > b) Onboard sound > c) Onboard USB > d) Onboard legacy ports > e) Set "F1 on error" or "halt on errors" to off in the BIOS so you > can boot headless > > 3. Install your OS > > I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a > "server" with everything > disabled except "development". I also set the detected NIC's to > "unconfigured" - no DHCP, > no start at boot. > > 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's > are running OK > > 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface > to the LAN and > eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface > will be bridged together. > > Easy enough - plug in a network cable into a NIC and run dhclient. Once you > get an IP from your > DHCP server you can use ifconfig to determine whihc one got the IP. Repeat > for each NIC. Once you > have detemined which physical device is which, I used a sharpie to label > each port so I wouldn't > get confused. > > 6. Install bridging support, if your distro does not support it OOB. > Fortunately, CentOS does so > no problem there. > > 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum > install bridge-utils" > > 8. Configure the bonding config > > In /etc/sysconfig/network-scripts make valid entries for your bonded device > and each slave. > > ifcfg-bond0: > > DEVICE=bond0 > ONBOOT=no > BOOTPROTO=none > > ifcfg-eth1/2/3: > > DEVICE=eth1/2/3 (replace as nessisary) > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > > 9. Startup script /etc/rc.d/rc.local: > > modprobe bonding miimon=1* > brctl addbr br0 > brctl addif br0 eth0 > sleep 10s** > brctl addif br0 bond0 > > *miimon=1 means disable the port after one second on link down. This is the > redundant part. If > one of your links fail, the other two will keep working. > > ** The 'sleep' is to allow the bridge to stabilize. In testing, the bridge > did not work if each > brctl statement immediately followed each other. > > 10. Test without VDSL extenders > > Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the > source LAN. Plug the > 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd > PC's eth0 with a > crossover cable, or plug the second PC into a switch then plug your devices > into the same switch. > Make sure both PC's have a keyboard and monitor so you can see what's going > on. Finally, plug > crossover cables into each PC's eth1, eth2 & eth3 > > On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic > right away. Do some > test pings from and to your devices, and try to get an IP address (dhcp) > from the "remote" LAN. > > Troubleshooting: > > *Link up at 100base-T FD? > *ifconfig eth1, eth2, eth3 shows as SLAVE? > *same ifconfig shows TX and RX values > 0 > *try mii-tool -F 100baseT-FD > > 11. Rack up and plug in the PC's in the source and destination LAN's. > Terminate the Cat 3 with the > RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely > brief manual that comes > with the LB-300 so straight-through on all pins was used. Undoubtedly, it > follows the telco standard of > the middle pins, but this is untested. Plug the freshly-terminated plugs > into the RJ12 port on > the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45 > ports on the LB-300 into > eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to "Loc" > (LOCAL) on the source LAN's > LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300. > > 12. Power up and watch for the blinkenlights > > Troubleshooting: > > *Same as troubleshooting above? > *Look at the lights on the top of the LB-300's. They should give you an > indication as to what's > happening to the LB-300 according to the troubleshooting chart on the LB-300 > manual. > > 13. Enjoy your link! > > _______________________________________________ > --Bandwidth and Colocation provided by Easynews.com -- > > Asterisk-Users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >
Ronan Mullally
2006-Apr-07 01:41 UTC
[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]
Hi Colin, Nice post - thanks, but:> HARDWARE: > > 2 X 4-PCI slot PC's > 6 X Black Box LB300A VDSL Ethernet Extenders > 8 X PCI Ethernet cardsWhy not just install an ethernet switch on both ends that supports trunking / etherchannel? Less configuration, less chance for operator error, and no hard disks. You'll most likely also need a switch on each end *anyway*... -Ronan
BJ Weschke
2006-Apr-07 04:54 UTC
[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]
On 4/6/06, Colin Anderson <ColinA@landmarkmasterbuilder.com> wrote:> I was given the challenge recently of creating a LAN-LAN bridge between two > buildings several > hundred metres from each other, using only existing Cat 3 wiring and without > having to resort > to an expensive and finicky 5 Ghz wireless link. I was able to create a 90 > megabit link for > about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black > Box VDSL Ethernet > Extender, which supports 30 megabits over a single twisted pair. > > This is relevant to the list because I have seen many posts with people > facing the same > kind of challenge deploying Asterisk in remote locations. In my case, I am > running > ~40 Snom 360's from the remote building to where my Asterisk server is, and > it's working > fine. > > Hope this helps someone. Yay Linux. > > ---------------------------------------------------------------------------- > --------------- > HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable > up to 1.9 km > ---------------------------------------------------------------------------- > --------------- > > Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders > and Linux bonding, > it is possible to create a protocol-independent, redundant, high speed > Ethernet bridge using common > Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9 > km (6,233 feet), > with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of > the link is that of > a regular bridged LAN, and is suitable for high speed or latency sensitive > applications such as Voice > over IP. > > HARDWARE: > > 2 X 4-PCI slot PC's > 6 X Black Box LB300A VDSL Ethernet Extenders > 8 X PCI Ethernet cards > http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946 > RJ-12 crimper > RJ-12 6-conductor plugs (will work with RJ-11) > > SOFTWARE: > > CentOS 4.1 or any Linux distro that supports bonding and bridging > > > 1. Install hardware on both machines > > In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX > cards. The reason I selected > this board was because there is 5 avaliable PCI slots. Install the cards 0-3 > in slots 0-3. > > 2. Disable onboard hardware > > Because the NIC's will require an IRQ each, disable the following resources: > > a) Onboard NIC > b) Onboard sound > c) Onboard USB > d) Onboard legacy ports > e) Set "F1 on error" or "halt on errors" to off in the BIOS so you > can boot headless > > 3. Install your OS > > I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a > "server" with everything > disabled except "development". I also set the detected NIC's to > "unconfigured" - no DHCP, > no start at boot. > > 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's > are running OK > > 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface > to the LAN and > eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface > will be bridged together. > > Easy enough - plug in a network cable into a NIC and run dhclient. Once you > get an IP from your > DHCP server you can use ifconfig to determine whihc one got the IP. Repeat > for each NIC. Once you > have detemined which physical device is which, I used a sharpie to label > each port so I wouldn't > get confused. > > 6. Install bridging support, if your distro does not support it OOB. > Fortunately, CentOS does so > no problem there. > > 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum > install bridge-utils" > > 8. Configure the bonding config > > In /etc/sysconfig/network-scripts make valid entries for your bonded device > and each slave. > > ifcfg-bond0: > > DEVICE=bond0 > ONBOOT=no > BOOTPROTO=none > > ifcfg-eth1/2/3: > > DEVICE=eth1/2/3 (replace as nessisary) > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > > 9. Startup script /etc/rc.d/rc.local: > > modprobe bonding miimon=1* > brctl addbr br0 > brctl addif br0 eth0 > sleep 10s** > brctl addif br0 bond0 > > *miimon=1 means disable the port after one second on link down. This is the > redundant part. If > one of your links fail, the other two will keep working. > > ** The 'sleep' is to allow the bridge to stabilize. In testing, the bridge > did not work if each > brctl statement immediately followed each other. > > 10. Test without VDSL extenders > > Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the > source LAN. Plug the > 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd > PC's eth0 with a > crossover cable, or plug the second PC into a switch then plug your devices > into the same switch. > Make sure both PC's have a keyboard and monitor so you can see what's going > on. Finally, plug > crossover cables into each PC's eth1, eth2 & eth3 > > On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic > right away. Do some > test pings from and to your devices, and try to get an IP address (dhcp) > from the "remote" LAN. > > Troubleshooting: > > *Link up at 100base-T FD? > *ifconfig eth1, eth2, eth3 shows as SLAVE? > *same ifconfig shows TX and RX values > 0 > *try mii-tool -F 100baseT-FD > > 11. Rack up and plug in the PC's in the source and destination LAN's. > Terminate the Cat 3 with the > RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely > brief manual that comes > with the LB-300 so straight-through on all pins was used. Undoubtedly, it > follows the telco standard of > the middle pins, but this is untested. Plug the freshly-terminated plugs > into the RJ12 port on > the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45 > ports on the LB-300 into > eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to "Loc" > (LOCAL) on the source LAN's > LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300. > > 12. Power up and watch for the blinkenlights > > Troubleshooting: > > *Same as troubleshooting above? > *Look at the lights on the top of the LB-300's. They should give you an > indication as to what's > happening to the LB-300 according to the troubleshooting chart on the LB-300 > manual. > > 13. Enjoy your link! >Along the same lines, we've had good success at one client where the local ILEC leased dry pair at this property management client and we were able to apply HDSL adapters from http://www.rad-direct.com/ to these lines and get about 3Mb/s in a distance of over 2 miles on some of the links. We've got IP501's hung off these connections and the really nice thing (that we really hadn't even planned on until we saw it happening) was that these adapters passed 802.1q tagged packets right through so we can use the QoS on the managed switches as if it were just part of the rest of the LAN. -- Bird's The Word Technologies, Inc. http://www.btwtech.com/
Andrew Kohlsmith
2006-Apr-07 05:53 UTC
[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]
On Thursday 06 April 2006 17:44, Colin Anderson wrote:> I was given the challenge recently of creating a LAN-LAN bridge between two > buildings several > hundred metres from each other, using only existing Cat 3 wiring and > without having to resort > to an expensive and finicky 5 Ghz wireless link. I was able to create a 90 > megabit link for > about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black > Box VDSL Ethernet > Extender, which supports 30 megabits over a single twisted pair.I have to ask, what was wrong with a pair of media converters ($200/pair new, $50/pair on ebay) and some cheap-as-dirt multimode fiber? Isolated, 100mbit and easily, easily gangable. Was the goal simply to get as fast as possible with regular copper wire, or was there a bigger objective? I do appreciate the effort put into this, though, and more than anything I appreciate your posting it here for others. I sincerely thank you for that. -A.
Seemingly Similar Threads
- OT: VoIP over bonded link
- OT: HOWTO: Create a 90mbit bonded link 600 metres away with Cat 3 or telco wire [long]
- OT: HOWTO: Create a 90mbit bonded link 600 m etre s away with Cat 3 or telco wire [long]
- OT: HOWTO: Create a 90mbit bonded link 600 m etres away with Cat 3 or telco wire [long]
- Configure Ubuntu Server 16.04 for icecast2