Xen users - Oct 2007 - RESOLVED: Debian Xen + Broadcom NetXtreme II (IBM x3655 7985-AC1)

I have a new x3655 IBM and whenever the /etc/xen/scripts/network-bridge
script starts, the ethernet would no longer work.  This post isn''t
about
the cause, but the fix.  I did a lot of tcpdumps and Googling, but I''ll
spare you that.  All they do is prove that yes, there is an issue. :)

I tried Debian 4.0 i386/amd64 and Ubuntu 7.04 Server i386/amd64 and they
all exhibit the problem.

Upgrading the bnx2 driver is the key.  I came to find bnx2 is included
in the initrd.  Debian comes with 1.4.44 and Ubuntu 1.4.45 of the
driver.  As of this writing 1.5.10c is the latest from Broadcom and is
what I ended up using.

Summary:
 * Install kernel source, make config/prepare the distro''s config file
 * Download bnx2 source, compile and install
 * Update initrd

Here are step by step instructions for clean install of Debian 4.0 amd64:

aptitude install linux-source unzip
cd /usr/src
tar jxf linux-source-2.6.18.tar.bz2
ln -s linux-source-2.6.18 linux
cd linux
cp /boot/config-2.6.18-5-xen-amd64 ./.config
make oldconfig (<enter> through all the prompts to take defaults)
make prepare
cd /tmp
wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip
unzip linux-1.5.10c.zip
cp Server/Linux/Driver/bnx2-1.5.10c.tar.gz /usr/src
cd /usr/src
tar zxf bnx2-1.5.10c.tar.gz
cd bnx2-1.5.10c/src
make
make install
mv /boot/initrd.img-2.6.18-5-xen-amd64 \
/boot/initrd.img-2.6.18-5-xen-amd64.orig
update-initramfs -k `uname -r` -c
reboot

Verify:
zcat /boot/initrd.img-2.6.18-5-xen-amd64.orig|strings|grep "bnx2 v"
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44 (August 10, 2006)

zcat /boot/initrd.img-2.6.18-5-xen-amd64|strings|grep "bnx2 v"
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.5.10c (May 4, 2007)


Other keywords: Dell DELL

-- 
Kris S. Amundson
Founder, CIO                   GPG Key: D6D39F2C
OpenSourcery, LLC.             http://www.opensourcery.com/



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

Kris S. Amundson schreef:
> Upgrading the bnx2 driver is the key.  I came to find bnx2 is included
> in the initrd.  Debian comes with 1.4.44 and Ubuntu 1.4.45 of the
> driver.  As of this writing 1.5.10c is the latest from Broadcom and is
> what I ended up using.

Just updated the driver, verified it was loaded (by version), changed
the config of xend back to bridged. Restarted Xend, bang, bridge kills
the interface.


So my basic assumption:
- - A bridge should be placed automatically by the default scripts over
something like peth0/eth0 and the vif interface between the virtual
hosts. The peth0/eth0 is the interface you receive your netwerk
connections on.


For me the ''remove management functions'' doesn''t
work, and this new
driver doesn''t give me a working bridge.



Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHCEhEYH1+F2Rqwn0RCuVLAJ9x/QeYJnKliDENW9p0bWiUCuhrVgCeLAZh
lwp2caIeoLbZPpe9IPaeVZA=0UQg
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha schreef:
> Stefan de Konink wrote:
>> Fajar A. Nugraha schreef:
>>> One question. On your configuration, does bridge works on native
Linux?
>> Nope, any bridge that touches the NIC kills the NIC.
>>
> Okay, now we''re getting somewhere :)
> 
> First step will be to get it working on native linux first.
> 
> I use :
> - RHEL5
> - HP ProLiant DL380 G5
> - Broadcom NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Using Gentoo, with the same driver (now). The only thing I can think of
now is that the ''bridge'' iptables stuff in /proc/net are set
wrong again
after boot. That kill the traffic, with a working driver. I only changed
these values with the old driver while testing with a Xen host on eth1.

I''ll investigate that possibility too.

Stefan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> Hi,
>
> Kris S. Amundson schreef:
> > Upgrading the bnx2 driver is the key.  I came to find bnx2 is included
> > in the initrd.  Debian comes with 1.4.44 and Ubuntu 1.4.45 of the
> > driver.  As of this writing 1.5.10c is the latest from Broadcom and is
> > what I ended up using.
>
Strange. I thought network drivers were not supposed to be included in
the initrd.
>
> Just updated the driver, verified it was loaded (by version), changed
> the config of xend back to bridged. Restarted Xend, bang, bridge kills
> the interface.
>
One question. On your configuration, does bridge works on native Linux?
>
> So my basic assumption:
> - A bridge should be placed automatically by the default scripts over
> something like peth0/eth0 and the vif interface between the virtual
> hosts. The peth0/eth0 is the interface you receive your netwerk
> connections on.
>
>
I had problems with some models of Broadcom NICs (both tg3 and bnx2
drivers). Other than that, I didn''t like the way xend start script was
changing network device names, so what I did was :
- upgrade the drivers
- changed network-script to /bin/true on xend-config.sxp
- setup bridge using standard RHEL config files
(/etc/sysconfig/network-scripts/ifcfg-br0)

I find this approach works best on my setup. Using RHEL5, which includes
Xen-3.0.3.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> Fajar A. Nugraha schreef:
> > One question. On your configuration, does bridge works on native
Linux?
>
> Nope, any bridge that touches the NIC kills the NIC.
>
Okay, now we''re getting somewhere :)

First step will be to get it working on native linux first.

I use :
- RHEL5
- HP ProLiant DL380 G5
- Broadcom NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

This server works fine with bridge and VLAN (using both native Linux and
Xen).
The default RH driver works fine, but I prefer to use updated bnx2
driver (1.5.10c). I don''t have Dell 1950 to test, but if your NIC is
the
same model (see lspci) then updated driver should work.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ok there are several issues with bridging:


*BUG* The bridging script doesn''t take in account an ipv6 address can
be
present in the ip addr command. This results in the (parent) network
interface not to come up again, because of a scripting error. It can be
fixed for IPv4 by adding a ''space'' in the script after
''inet'', I posted
some details in the Gentoo Bugtracker.

*STUPIDNESS* Broadcom nics in Dells get disabled if PXE boot is not
selected at boot. This results in an interface that cannot get a link,
because it does exists, but is not enabled.

*STUPIDNESS* You cannot set an IP address on the endpoint of a NIC. If
you want to manipulate a random subnet trough the bridge you should set
the bridge with an IP address in this subnet and not one of the ''to be
bridged'' IP addresses.


*IDEA* I did disable the management code, I don''t know if it is really
required.


So

eth1:0.0.0.0  ----  bridge  ---- tap0:0.0.0.0
                  192.168.0.1

and NOT:

eth1:192.168.0.1 ---- bridge ---- tap0:0.0.0.0


Interesting enough, if you put the IP address on the tap0 for example,
you would get a working routing configuration, but this is probably not
what you want to do.


I checked this config on the Xen machine and on an external machine
connected via eth1.



Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHC+cVYH1+F2Rqwn0RCiS2AJ9o8KQxQHYAbNGCzPw7TRCCsIALowCdFtpZ
0G75+e234zV9spK5EErAhng=WB9d
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> Ok there are several issues with bridging:
> *STUPIDNESS* You cannot set an IP address on the endpoint of a NIC. If
> you want to manipulate a random subnet trough the bridge you should set
> the bridge with an IP address in this subnet and not one of the
''to be
> bridged'' IP addresses.
>
>
> So
>
> eth1:0.0.0.0  ----  bridge  ---- tap0:0.0.0.0
>                   192.168.0.1
>
> and NOT:
>
> eth1:192.168.0.1 ---- bridge ---- tap0:0.0.0.0
>
>
This is how linux bridge works.
Solaris bridge uses a different bridging configration, in which you put
IP address on the NIC.

What I like to do is this :
eth1 --- bridge --- tap0,vif0.1

veth1:192.168.0.1

That is :
- disabled Xen''s network-bridge script
- put vif0.1 on the bridge
- setup ip address on veth1

vif0.1 and veth1 is created by xen''s netloop driver (loaded by xend. I
also added it to /etc/modprobe.conf). I prefer this setup because :
- there''s no IP address on the bridge
- All networking setup is done by redhat''s standard config files.

Regards,

Fajar


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> Ok there are several issues with bridging:
> *STUPIDNESS* You cannot set an IP address on the endpoint of a NIC. If
> you want to manipulate a random subnet trough the bridge you should set
> the bridge with an IP address in this subnet and not one of the
''to be
> bridged'' IP addresses.
>
>
> So
>
> eth1:0.0.0.0  ----  bridge  ---- tap0:0.0.0.0
>                   192.168.0.1
>
> and NOT:
>
> eth1:192.168.0.1 ---- bridge ---- tap0:0.0.0.0
>
>
This is how linux bridge works.
Solaris bridge uses a different bridging configration, in which you put
IP address on the NIC.

What I like to do is this :
eth1 --- bridge --- tap0,vif0.1

veth1:192.168.0.1

That is :
- disabled Xen''s network-bridge script
- put vif0.1 on the bridge
- setup ip address on veth1

vif0.1 and veth1 is created by xen''s netloop driver (loaded by xend. I
also added it to /etc/modprobe.conf). I prefer this setup because :
- there''s no IP address on the bridge
- All networking setup is done by redhat''s standard config files.

Regards,

Fajar


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, 10 Oct 2007, Fajar A. Nugraha wrote:
> This is how linux bridge works.
> Solaris bridge uses a different bridging configration, in which you put
> IP address on the NIC.
Interesting, than it was not stupidness ;)
> What I like to do is this :
> eth1 --- bridge --- tap0,vif0.1
>
> veth1:192.168.0.1
>
> That is :
> - disabled Xen''s network-bridge script
> - put vif0.1 on the bridge
> - setup ip address on veth1
>
> vif0.1 and veth1 is created by xen''s netloop driver (loaded by
xend. I
> also added it to /etc/modprobe.conf). I prefer this setup because :
> - there''s no IP address on the bridge
> - All networking setup is done by redhat''s standard config files.
When I put a tcpdump on the interface the only thing data came out was
tap0. What is vif0.1 doing for Xen?


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stefan de Konink wrote:
> When I put a tcpdump on the interface the only thing data came out was
> tap0. What is vif0.1 doing for Xen?
>
>   
tap interface is created for HVM domains. A PV domain will only create
vifx.y.

x -> domain ID
y -> interface index

vif0.1 is the netback for dom0''s veth1
vif1.0 is the netback for domU''s (domain ID 1) eth0

vethy and vif0.y is only created (in RH''s case) after netloop driver is
loaded (which is the case after starting xend). Don''t know about gentoo
though.

In the default xend network-bridge script,
1. a bridge is created
xenbr0 -- vif0.0, eth0
2. transfer mac, ip, and routes to veth0
3. eth0 is renamed into peth0, veth0 renamed into eth0

The order is not excactly like that (see
/etc/xen/script/network-bridge), but you get the idea.
I think the purpose is compatibility, to allow applications that uses
eth0 continue to do so.

In practice the ip/route transfer and interface-renaming might cause
some problems for scripts that start before xend (early relsease of
RHEL5''s cluster comes to mind), which is part of the reason why
disabled
the default network-bridge and setup bridge/ip address using standard RH
config files.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users