Xen users - Feb 2006 - XenNetworking - Reason for NOARP on eth1/br1?

http://wiki.xensource.com/xenwiki/XenNetworking

I''m wondering about this requirement:

before you connect a physical interface to a bridge, remember to reset
it''s mac and turn arp off. For example:

# ip link set eth1 down
# ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off
# brctl addif br1 eth1
# ip link set eth1 up

NOARP does not seem to be set by default (netrwork-bridge) on eth0. Is
there a reason for this?



--
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, 2006-02-20 at 10:53 +1300, Nicholas Lee wrote:
> http://wiki.xensource.com/xenwiki/XenNetworking
> 
> I''m wondering about this requirement:
> 
> before you connect a physical interface to a bridge, remember to reset
> it''s mac and turn arp off. For example:
> 
> # ip link set eth1 down
> # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off
> # brctl addif br1 eth1
> # ip link set eth1 up
> 
> NOARP does not seem to be set by default (netrwork-bridge) on eth0. Is
> there a reason for this?
If you look carefully at network-bridge, you will find the line in the
function "op_start" that runs "ip link set ${pdev} down arp
off".  This
command shuts down the physical interface and disables ARP at the same
time.

ARP is the way a host learns an ethernet MAC address associated with an
IP address.  The reason ARP is disabled on the physical interface is
that once the xenbr0 bridge is brought up, the physical interface is
simply being used as a transparent virtual connection between the
external network and the xenbr0 bridge.  It does not have any IP address
associated with it, so there is no reason for it to support ARP.

Dom0''s IP address and MAC address get moved from that physical
interface
to a virtual interface (and the confusing part is that the virtual
interface also assumes the OS device name "eth0").  ARP does get
enabled
on the virtual interface "eth0", since that is the interface with the
dom0''s IP address on it.

So, the same thing needs to occur whenever you create an internal bridge
and attach a physical interface to it.  You disable ARP on that physical
interface and set it''s MAC address to a generic bridge MAC that
isn''t
really used by IP at all.  You then put IP addresses and enable ARP on
the domU''s virtual ethernet interfaces that connect to the internal
bridge.

I hope that clears things up a little.

-- 

Patrick Wolfe

email:   pwolfe@employease.com



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 21/02/06, Patrick Wolfe <pwolfe@employease.com>
wrote:
> external network and the xenbr0 bridge.  It does not have any IP address
> associated with it, so there is no reason for it to support ARP.
Right, but if ARP is enabled will it wont cause problems?

> So, the same thing needs to occur whenever you create an internal bridge
> and attach a physical interface to it.  You disable ARP on that physical
> interface and set it''s MAC address to a generic bridge MAC that
isn''t
> really used by IP at all.  You then put IP addresses and enable ARP on
> the domU''s virtual ethernet interfaces that connect to the
internal
> bridge.
Thing is a virtual ethernet device is only created for eth0 by xend.
AFAICT with the default scripts if you setup a second bridge network
with either a real physical or a dummy interface a second virtual
veth/vif is not created.

So that bit of instruction seems out of place without additional
instruction on how to create a second virtual ethernet pair to split
eth1 in a similar way to eth0 into peth and (v)eth.


Main reason I''m trying to figure this out is on my problem host,
I''m
struggling with what seems to be a problem with bridging and working
correct. (See and early email.) Pings work, tcp (ssh) doesn''t.  eth0
bridge works locally, eth1 bridge doesn''t work locally. Only remotely.

I upgraded to 3.0.1 yesterday, and I''m sure previously eth0 bridge
didn''t work locally, but now it does.  So I''m trying to make
the two
bridge on the same machine exactly the same to see if I can figure out
why and what is causing the problem.


Thanks.
--
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ignore my previous comment. I just had some help from Ewen and
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=332 in
xen-devel.

I''ll try update XenNetworking with a clearer example for using xenbr1.

--
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

I''ve had a similar problem. I found it to be due to crc checksum
errors.

I solved that with "ethtool -K ethN tx off" on all domU''s.

I found some related info on the following thread:

http://lists.xensource.com/archives/html/xen-users/2006-01/msg00088.html

rgrds,
Braulio Gergull

On Feb 20, 2006 04:26 PM, Nicholas Lee <emptysands@gmail.com> wrote:
> On 21/02/06, Patrick Wolfe <pwolfe@employease.com> wrote:
> > external network and the xenbr0 bridge. It does not have any IP
> > address
> > associated with it, so there is no reason for it to support ARP.
>
> Right, but if ARP is enabled will it wont cause problems?
>
>
> > So, the same thing needs to occur whenever you create an internal
> > bridge
> > and attach a physical interface to it. You disable ARP on that
> > physical
> > interface and set it''s MAC address to a generic bridge MAC
that
> > isn''t
> > really used by IP at all. You then put IP addresses and enable ARP
> > on
> > the domU''s virtual ethernet interfaces that connect to the
internal
> > bridge.
>
> Thing is a virtual ethernet device is only created for eth0 by xend.
> AFAICT with the default scripts if you setup a second bridge network
> with either a real physical or a dummy interface a second virtual
> veth/vif is not created.
>
> So that bit of instruction seems out of place without additional
> instruction on how to create a second virtual ethernet pair to split
> eth1 in a similar way to eth0 into peth and (v)eth.
>
>
> Main reason I''m trying to figure this out is on my problem host,
I''m
> struggling with what seems to be a problem with bridging and working
> correct. (See and early email.) Pings work, tcp (ssh) doesn''t.
eth0
> bridge works locally, eth1 bridge doesn''t work locally. Only
remotely.
>
> I upgraded to 3.0.1 yesterday, and I''m sure previously eth0 bridge
> didn''t work locally, but now it does. So I''m trying to
make the two
> bridge on the same machine exactly the same to see if I can figure out
> why and what is causing the problem.
>
>
> Thanks.
> --
> Nicholas Lee
> http://stateless.geek.nz
> gpg 8072 4F86 EDCD 4FC1 18EF 5BDD 07B0 9597 6D58 D70C
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 2006-02-21 at 08:26 +1300, Nicholas Lee wrote:
> On 21/02/06, Patrick Wolfe <pwolfe@employease.com> wrote:
> > external network and the xenbr0 bridge.  It does not have any IP
address
> > associated with it, so there is no reason for it to support ARP.
> 
> Right, but if ARP is enabled will it wont cause problems?
If ARP is enabled on a physical ethernet interface that is setup with
the mac address FE:FF:FF:FF:FF:FF, I doubt any problem will occur, but
why risk it?
> Thing is a virtual ethernet device is only created for eth0 by xend.
> AFAICT with the default scripts if you setup a second bridge network
> with either a real physical or a dummy interface a second virtual
> veth/vif is not created.
There are two sets of virtual ethernet interface pair that Xen creates.
The first set are for use by dom0.  That''s the eight connected pairs
from veth0<->vif0.0, veth1<->vif0.1 through veth7<->vif0.7.

That''s all that exists when you first boot up dom0.  When you create a
new domU, a new virtual ethernet interface pair is created out of thin
air.  One end is the ethernet interface of the domU (eth0 for linux, xn0
for bsd), the other end is the vif#.0 in dom0.  The "#" character is
actually the virtual machine number.

If you want to create a second bridge, attach a physical interface to
it, and attach a domU to that bridge, without dom0 having an interface
with it, you will need to write some script somewhere that will create
the bridge and set it up, setup the physical interface and attach it to
the bridge, and then you can "xm create" the domU, passing the second
bridge''s name in the vif statement.  For example:

#
#       xen virtual machine configuration file
#
name = "fire1"
memory = 64
kernel = "/boot/vmlinuz-2.6.12.6-xen"
ramdisk = "/boot/initrd.img-2.6.12.6-xen"
root = "/dev/sda1"
disk=[  ''phy:/dev/vg0/fire1root,sda1,w'',
        ''phy:/dev/vg0/fire1swap,sda2,w'' ]
vif=[   ''mac=00:16:3E:70:01:01,bridge=br0eth0'',
        ''mac=00:16:3E:70:02:01,bridge=br1dmz'' ]


This is the fire1.sxp config file I use on my desktop.  It specifies
that the fire1 system connects to two different bridges - br0eth0 which
happens to be connected to the physical interface eth0, and br1dmz,
which is where all the rest of my domU''s connect their eth0 interface.

You don''t have to connect dom0 to the second bridge.  You
don''t have to
rename the physical interface as "peth1" either.  I think the only
reason they did that renaming thing, is that some scripts and people
would get confused if their dom0''s ethernet interface was named
different than the default interface of each of their domUs.
> Main reason I''m trying to figure this out is on my problem host,
I''m
> struggling with what seems to be a problem with bridging and working
> correct. (See and early email.) Pings work, tcp (ssh) doesn''t. 
eth0
> bridge works locally, eth1 bridge doesn''t work locally. Only
remotely.
That sounds like the well known issue with checksum offloading not
working in the xen virtual ethernet drivers.  I had the same problem, at
first.  You just need to run:

	ethtool -K eth0 tx off

on every one of your interfaces in every domU and dom0 whenever they are
brought up.

If you''re using debian, just add:

	pre-up ethtool -K eth0 tx off

to every domU''s /etc/network/interfaces file, right after the
"iface
eth0 inet dhcp" line.  If you''re using some other distro, like
redhat or
suse, well, I''m not intimately familiar with how they set their network
interfaces up to know where to add that command.

-- 

Patrick Wolfe

email:   pwolfe@employease.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users