asterisk users - Apr 2006 - OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]

I was given the challenge recently of creating a LAN-LAN bridge between two
buildings several
hundred metres from each other, using only existing Cat 3 wiring and without
having to resort
to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
megabit link for 
about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black
Box VDSL Ethernet 
Extender, which supports 30 megabits over a single twisted pair. 

This is relevant to the list because I have seen many posts with people
facing the same 
kind of challenge deploying Asterisk in remote locations. In my case, I am
running 
~40 Snom 360's from the remote building to where my Asterisk server is, and
it's working
fine. 

Hope this helps someone. Yay Linux. 

----------------------------------------------------------------------------
---------------
HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable
up to 1.9 km 
----------------------------------------------------------------------------
---------------

Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders
and Linux bonding,
it is possible to create a protocol-independent, redundant, high speed
Ethernet bridge using common
Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9
km (6,233 feet), 
with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of
the link is that of
a regular bridged LAN, and is suitable for high speed or latency sensitive
applications such as Voice
over IP.

HARDWARE:

2 X 4-PCI slot PC's
6 X Black Box LB300A VDSL Ethernet Extenders 
8 X PCI Ethernet cards
http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946
RJ-12 crimper
RJ-12 6-conductor plugs (will work with RJ-11)

SOFTWARE:

CentOS 4.1 or any Linux distro that supports bonding and bridging


1. Install hardware on both machines

In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX
cards. The reason I selected 
this board was because there is 5 avaliable PCI slots. Install the cards 0-3
in slots 0-3. 

2. Disable onboard hardware

Because the NIC's will require an IRQ each, disable the following resources:

	a) Onboard NIC
	b) Onboard sound
	c) Onboard USB
	d) Onboard legacy ports 
	e) Set "F1 on error" or "halt on errors" to off in the BIOS
so you
can boot headless
	
3. Install your OS

I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a
"server" with everything 
disabled except "development". I also set the detected NIC's to
"unconfigured" - no DHCP, 
no start at boot.

4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's
are running OK

5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface
to the LAN and 
eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface
will be bridged together.

Easy enough - plug in a network cable into a NIC and run dhclient. Once you
get an IP from your 
DHCP server you can use ifconfig to determine whihc one got the IP. Repeat
for each NIC. Once you 
have detemined which physical device is which, I used a sharpie to label
each port so I wouldn't 
get confused.

6. Install bridging support, if your distro does not support it OOB.
Fortunately, CentOS does so 
no problem there.

7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum
install bridge-utils"

8. Configure the bonding config

In /etc/sysconfig/network-scripts make valid entries for your bonded device
and each slave.

ifcfg-bond0:

DEVICE=bond0
ONBOOT=no
BOOTPROTO=none

ifcfg-eth1/2/3:

DEVICE=eth1/2/3 (replace as nessisary)
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

9. Startup script /etc/rc.d/rc.local:

modprobe bonding miimon=1*
brctl addbr br0
brctl addif br0 eth0
sleep 10s**
brctl addif br0 bond0

*miimon=1 means disable the port after one second on link down. This is the
redundant part. If
one of your links fail, the other two will keep working. 

** The 'sleep' is to allow the bridge to stabilize. In testing, the
bridge
did not work if each 
brctl statement immediately followed each other.

10. Test without VDSL extenders

Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the
source LAN. Plug the 
3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd
PC's eth0 with a 
crossover cable, or plug the second PC into a switch then plug your devices
into the same switch. 
Make sure both PC's have a keyboard and monitor so you can see what's
going
on. Finally, plug
crossover cables into each PC's eth1, eth2 & eth3

On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic
right away. Do some 
test pings from and to your devices, and try to get an IP address (dhcp)
from the "remote" LAN. 

Troubleshooting:

*Link up at 100base-T FD?
*ifconfig eth1, eth2, eth3 shows as SLAVE?
*same ifconfig shows TX and RX values > 0
*try mii-tool -F 100baseT-FD

11. Rack up and plug in the PC's in the source and destination LAN's.
Terminate the Cat 3 with the
RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely
brief manual that comes 
with the LB-300 so straight-through on all pins was used. Undoubtedly, it
follows the telco standard of
the middle pins, but this is untested. Plug the freshly-terminated plugs
into the RJ12 port on
the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45
ports on the LB-300 into
eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to
"Loc"
(LOCAL) on the source LAN's 
LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300. 

12. Power up and watch for the blinkenlights

Troubleshooting:

*Same as troubleshooting above?
*Look at the lights on the top of the LB-300's. They should give you an
indication as to what's
happening to the LB-300 according to the troubleshooting chart on the LB-300
manual.

13. Enjoy your link!

Or, you could use a Corinex Phone Line Bridge which runs 128Mbits up to 2000
feet. They also have a co-ax version which is 200mbits and goes 4000 feet...
About $300 for both ends.

http://www.corinex.com/web/docx.nsf/(w)/eng-corinex_av_phoneline_ethernet_bridge


> I was given the challenge recently of creating a LAN-LAN bridge between two
> buildings several
> hundred metres from each other, using only existing Cat 3 wiring and
without
> having to resort
> to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
> megabit link for 
> about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable
Black
> Box VDSL Ethernet 
> Extender, which supports 30 megabits over a single twisted pair. 
> 
> This is relevant to the list because I have seen many posts with people
> facing the same 
> kind of challenge deploying Asterisk in remote locations. In my case, I am
> running 
> ~40 Snom 360's from the remote building to where my Asterisk server is,
and
> it's working
> fine. 
> 
> Hope this helps someone. Yay Linux. 
> 
>
----------------------------------------------------------------------------
> ---------------
> HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable
> up to 1.9 km 
>
----------------------------------------------------------------------------
> ---------------
> 
> Abstract: Using the newly-available Black Box LB300A VDSL Ethernet
Extenders
> and Linux bonding,
> it is possible to create a protocol-independent, redundant, high speed
> Ethernet bridge using common
> Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9
> km (6,233 feet), 
> with bandwidth ranging from 90 megabits to 3 megabits. The characteristic
of
> the link is that of
> a regular bridged LAN, and is suitable for high speed or latency sensitive
> applications such as Voice
> over IP.
> 
> HARDWARE:
> 
> 2 X 4-PCI slot PC's
> 6 X Black Box LB300A VDSL Ethernet Extenders 
> 8 X PCI Ethernet cards
> http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946
> RJ-12 crimper
> RJ-12 6-conductor plugs (will work with RJ-11)
> 
> SOFTWARE:
> 
> CentOS 4.1 or any Linux distro that supports bonding and bridging
> 
> 
> 1. Install hardware on both machines
> 
> In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX
> cards. The reason I selected 
> this board was because there is 5 avaliable PCI slots. Install the cards
0-3
> in slots 0-3. 
> 
> 2. Disable onboard hardware
> 
> Because the NIC's will require an IRQ each, disable the following
resources:
> 
> 	a) Onboard NIC
> 	b) Onboard sound
> 	c) Onboard USB
> 	d) Onboard legacy ports 
> 	e) Set "F1 on error" or "halt on errors" to off in the
BIOS so you
> can boot headless
> 	
> 3. Install your OS
> 
> I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a
> "server" with everything 
> disabled except "development". I also set the detected NIC's
to
> "unconfigured" - no DHCP, 
> no start at boot.
> 
> 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the
NIC's
> are running OK
> 
> 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the
interface
> to the LAN and 
> eth1,2,3 will be the bonded link. Both the bonded link and the LAN
interface
> will be bridged together.
> 
> Easy enough - plug in a network cable into a NIC and run dhclient. Once you
> get an IP from your 
> DHCP server you can use ifconfig to determine whihc one got the IP. Repeat
> for each NIC. Once you 
> have detemined which physical device is which, I used a sharpie to label
> each port so I wouldn't 
> get confused.
> 
> 6. Install bridging support, if your distro does not support it OOB.
> Fortunately, CentOS does so 
> no problem there.
> 
> 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum
> install bridge-utils"
> 
> 8. Configure the bonding config
> 
> In /etc/sysconfig/network-scripts make valid entries for your bonded device
> and each slave.
> 
> ifcfg-bond0:
> 
> DEVICE=bond0
> ONBOOT=no
> BOOTPROTO=none
> 
> ifcfg-eth1/2/3:
> 
> DEVICE=eth1/2/3 (replace as nessisary)
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
> 
> 9. Startup script /etc/rc.d/rc.local:
> 
> modprobe bonding miimon=1*
> brctl addbr br0
> brctl addif br0 eth0
> sleep 10s**
> brctl addif br0 bond0
> 
> *miimon=1 means disable the port after one second on link down. This is the
> redundant part. If
> one of your links fail, the other two will keep working. 
> 
> ** The 'sleep' is to allow the bridge to stabilize. In testing, the
bridge
> did not work if each 
> brctl statement immediately followed each other.
> 
> 10. Test without VDSL extenders
> 
> Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port
into the
> source LAN. Plug the 
> 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into
the 2nd
> PC's eth0 with a 
> crossover cable, or plug the second PC into a switch then plug your devices
> into the same switch. 
> Make sure both PC's have a keyboard and monitor so you can see
what's going
> on. Finally, plug
> crossover cables into each PC's eth1, eth2 & eth3
> 
> On both PC's run tcpdump -i br0 from the console . In my case, I saw
traffic
> right away. Do some 
> test pings from and to your devices, and try to get an IP address (dhcp)
> from the "remote" LAN. 
> 
> Troubleshooting:
> 
> *Link up at 100base-T FD?
> *ifconfig eth1, eth2, eth3 shows as SLAVE?
> *same ifconfig shows TX and RX values > 0
> *try mii-tool -F 100baseT-FD
> 
> 11. Rack up and plug in the PC's in the source and destination
LAN's.
> Terminate the Cat 3 with the
> RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the
extremely
> brief manual that comes 
> with the LB-300 so straight-through on all pins was used. Undoubtedly, it
> follows the telco standard of
> the middle pins, but this is untested. Plug the freshly-terminated plugs
> into the RJ12 port on
> the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45
> ports on the LB-300 into
> eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to
"Loc"
> (LOCAL) on the source LAN's 
> LB-3090's and "Rmt" (REMOTE) on the destination LAN's
LB-300.
> 
> 12. Power up and watch for the blinkenlights
> 
> Troubleshooting:
> 
> *Same as troubleshooting above?
> *Look at the lights on the top of the LB-300's. They should give you an
> indication as to what's
> happening to the LB-300 according to the troubleshooting chart on the
LB-300
> manual.
> 
> 13. Enjoy your link! 
> 
> _______________________________________________
> --Bandwidth and Colocation provided by Easynews.com --
> 
> Asterisk-Users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>

Hi Colin,

Nice post - thanks, but:
> HARDWARE:
>
> 2 X 4-PCI slot PC's
> 6 X Black Box LB300A VDSL Ethernet Extenders
> 8 X PCI Ethernet cards
Why not just install an ethernet switch on both ends that supports
trunking / etherchannel?  Less configuration, less chance for operator
error, and no hard disks.  You'll most likely also need a switch on
each end *anyway*...


-Ronan

On 4/6/06, Colin Anderson <ColinA@landmarkmasterbuilder.com>
wrote:
> I was given the challenge recently of creating a LAN-LAN bridge between two
> buildings several
> hundred metres from each other, using only existing Cat 3 wiring and
without
> having to resort
> to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
> megabit link for
> about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable
Black
> Box VDSL Ethernet
> Extender, which supports 30 megabits over a single twisted pair.
>
> This is relevant to the list because I have seen many posts with people
> facing the same
> kind of challenge deploying Asterisk in remote locations. In my case, I am
> running
> ~40 Snom 360's from the remote building to where my Asterisk server is,
and
> it's working
> fine.
>
> Hope this helps someone. Yay Linux.
>
>
----------------------------------------------------------------------------
> ---------------
> HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable
> up to 1.9 km
>
----------------------------------------------------------------------------
> ---------------
>
> Abstract: Using the newly-available Black Box LB300A VDSL Ethernet
Extenders
> and Linux bonding,
> it is possible to create a protocol-independent, redundant, high speed
> Ethernet bridge using common
> Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9
> km (6,233 feet),
> with bandwidth ranging from 90 megabits to 3 megabits. The characteristic
of
> the link is that of
> a regular bridged LAN, and is suitable for high speed or latency sensitive
> applications such as Voice
> over IP.
>
> HARDWARE:
>
> 2 X 4-PCI slot PC's
> 6 X Black Box LB300A VDSL Ethernet Extenders
> 8 X PCI Ethernet cards
> http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946
> RJ-12 crimper
> RJ-12 6-conductor plugs (will work with RJ-11)
>
> SOFTWARE:
>
> CentOS 4.1 or any Linux distro that supports bonding and bridging
>
>
> 1. Install hardware on both machines
>
> In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX
> cards. The reason I selected
> this board was because there is 5 avaliable PCI slots. Install the cards
0-3
> in slots 0-3.
>
> 2. Disable onboard hardware
>
> Because the NIC's will require an IRQ each, disable the following
resources:
>
>        a) Onboard NIC
>        b) Onboard sound
>        c) Onboard USB
>        d) Onboard legacy ports
>        e) Set "F1 on error" or "halt on errors" to off
in the BIOS so you
> can boot headless
>
> 3. Install your OS
>
> I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a
> "server" with everything
> disabled except "development". I also set the detected NIC's
to
> "unconfigured" - no DHCP,
> no start at boot.
>
> 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the
NIC's
> are running OK
>
> 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the
interface
> to the LAN and
> eth1,2,3 will be the bonded link. Both the bonded link and the LAN
interface
> will be bridged together.
>
> Easy enough - plug in a network cable into a NIC and run dhclient. Once you
> get an IP from your
> DHCP server you can use ifconfig to determine whihc one got the IP. Repeat
> for each NIC. Once you
> have detemined which physical device is which, I used a sharpie to label
> each port so I wouldn't
> get confused.
>
> 6. Install bridging support, if your distro does not support it OOB.
> Fortunately, CentOS does so
> no problem there.
>
> 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum
> install bridge-utils"
>
> 8. Configure the bonding config
>
> In /etc/sysconfig/network-scripts make valid entries for your bonded device
> and each slave.
>
> ifcfg-bond0:
>
> DEVICE=bond0
> ONBOOT=no
> BOOTPROTO=none
>
> ifcfg-eth1/2/3:
>
> DEVICE=eth1/2/3 (replace as nessisary)
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
>
> 9. Startup script /etc/rc.d/rc.local:
>
> modprobe bonding miimon=1*
> brctl addbr br0
> brctl addif br0 eth0
> sleep 10s**
> brctl addif br0 bond0
>
> *miimon=1 means disable the port after one second on link down. This is the
> redundant part. If
> one of your links fail, the other two will keep working.
>
> ** The 'sleep' is to allow the bridge to stabilize. In testing, the
bridge
> did not work if each
> brctl statement immediately followed each other.
>
> 10. Test without VDSL extenders
>
> Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port
into the
> source LAN. Plug the
> 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into
the 2nd
> PC's eth0 with a
> crossover cable, or plug the second PC into a switch then plug your devices
> into the same switch.
> Make sure both PC's have a keyboard and monitor so you can see
what's going
> on. Finally, plug
> crossover cables into each PC's eth1, eth2 & eth3
>
> On both PC's run tcpdump -i br0 from the console . In my case, I saw
traffic
> right away. Do some
> test pings from and to your devices, and try to get an IP address (dhcp)
> from the "remote" LAN.
>
> Troubleshooting:
>
> *Link up at 100base-T FD?
> *ifconfig eth1, eth2, eth3 shows as SLAVE?
> *same ifconfig shows TX and RX values > 0
> *try mii-tool -F 100baseT-FD
>
> 11. Rack up and plug in the PC's in the source and destination
LAN's.
> Terminate the Cat 3 with the
> RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the
extremely
> brief manual that comes
> with the LB-300 so straight-through on all pins was used. Undoubtedly, it
> follows the telco standard of
> the middle pins, but this is untested. Plug the freshly-terminated plugs
> into the RJ12 port on
> the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45
> ports on the LB-300 into
> eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to
"Loc"
> (LOCAL) on the source LAN's
> LB-3090's and "Rmt" (REMOTE) on the destination LAN's
LB-300.
>
> 12. Power up and watch for the blinkenlights
>
> Troubleshooting:
>
> *Same as troubleshooting above?
> *Look at the lights on the top of the LB-300's. They should give you an
> indication as to what's
> happening to the LB-300 according to the troubleshooting chart on the
LB-300
> manual.
>
> 13. Enjoy your link!
>

 Along the same lines, we've had good success at one client where the
local ILEC leased dry pair at this property management client and we
were able to apply HDSL adapters from http://www.rad-direct.com/ to
these lines and get about 3Mb/s in a distance of over 2 miles on some
of the links. We've got IP501's hung off these connections and the
really nice thing (that we really hadn't even planned on until we saw
it happening) was that these adapters passed 802.1q tagged packets
right through so we can use the QoS on the managed switches as if it
were just part of the rest of the LAN.

--
Bird's The Word Technologies, Inc.
http://www.btwtech.com/

On Thursday 06 April 2006 17:44, Colin Anderson wrote:
> I was given the challenge recently of creating a LAN-LAN bridge between two
> buildings several
> hundred metres from each other, using only existing Cat 3 wiring and
> without having to resort
> to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
> megabit link for
> about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable
Black
> Box VDSL Ethernet
> Extender, which supports 30 megabits over a single twisted pair.
I have to ask, what was wrong with a pair of media converters ($200/pair new, 
$50/pair on ebay) and some cheap-as-dirt multimode fiber?  Isolated, 100mbit 
and easily, easily gangable. Was the goal simply to get as fast as possible 
with regular copper wire, or was there a bigger objective?

I do appreciate the effort put into this, though, and more than anything I 
appreciate your posting it here for others.  I sincerely thank you for that.

-A.