CentOS virt - Apr 2008 - Xen networking - pulling out my hair

I just cant get a Centos 5.1 domU to install under Centos 5.1 dom0. It
is maddening! I'm trying to replace a single server that currently uses
simple ethernet aliases (eth0:0, eth0:1, etc) with a new server that
uses Xen virtualization.

Maybe I dont understand bridging properly or something.

My dom0 has the following:
eth0      Link encap:Ethernet  HWaddr 00:00:44:11:22:33
          inet addr:192.168.144.6  Bcast:192.168.144.7 Mask:255.255.255.252

Notice that it is a 192.168.144.4/30 address. That is all I have for my
dom0. I want two different domU's. All the examples I see online have
/24 ranges.

domU #1 will be a single 192.168.139.4
domU #2 will be a subnet of 192.168.139.128/29
 At this point that domU will be using 139.129, 130, and 131.

The router that I am connected to routes/forwards those subnets to my dom0.

At this stage I have modified xend-config.sxp and changed a line to
(network-script /bin/true). This was based on various postings telling
people to let Centos handle things instead of Xen.

A also did the following:
ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=none
NETWORK=192.168.144.4
NETMASK=255.255.255.252
IPADDR=192.168.144.6
GATEWAY=192.168.144.5
ONBOOT=yes
Type=Bridge

ifcfg-xenbr0:
DEVICE=xenbr0
ONBOOT=yes
BRIDGE=eth0
DELAY=0
HWADDR=00:00:44:11:22:33

One problem with this is that eth0 comes up first and complains:
Bringing up interface eth0:  Device eth0 does not seem to be present,
delaying initialization.
[FAILED]
Bringing up interface xenbr0:  [  OK  ]

So I ended up adding ifup eth0 to /etc/rc.local. What a hack. If I tried
to swap xenbr0 and eth0 files (like some online examples) there are
complaints about bridging operation not supported.

At this point, am I supposed to do anything special? I just run
virt-manager and for shared network it only shows me eth0 (shouldn't it
be xenbr0 - since they are bridged, maybe it doesnt matter)

In anaconda, I select manual/static IP address. I enter 192.168.139.4
with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5 (my
router). On the next screen anaconda says that it cannot find my
stage2.img. When I look at my server logs and even with tcpdump,
anaconda is not making any attempt to fetch this file despite the
'unable to retrieve.' message. So there must be some networking issue
between the dom0 and domU. I just have no clue. The only attempts I see
via tcpdump is if I try to select DHCP server, I see the broadcast packets.

I'm not running selinux and I've tried disabling iptables as well as
using 'iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT'

Initially from reading I thought I may want a 'routed' xen setup instead
of bridged, but it seems none of the tools virt-manager, virt-install,
koan, etc support 'routed'.

Ken Bass wrote on Mon, 28 Apr 2008 15:08:22 -0400:
> In anaconda, I select manual/static IP address. I enter 192.168.139.4
> with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5
AFAIK, there is no route from 192.168.139.4 to 192.168.144.5. You need to 
specify a gateway in that net!
Your earlier explanation looks a bit like you think your packets somehow 
"go thru" your Dom0 and rely on the network settings on it. That's
not the
case. Once the initial bridge is setup you can use *any* IP number on the 
DomU that you like, there is no connection to the IP numbers that are 
configured for the Dom0.
In other words, if you have a router with 192.168.1.1 and 192.168.2.1 and 
a Dom0 with 192.168.1.2 and a DomU with 192.168.2.2 you can reach that 
router from both with the standard (older) xen bridging that comes with 
CentOS 5. No problem. Also, you do not need any Iptables rules.
I think you needed nothing to change in your networking setup. I have been 
using static IP addresses with CentOS 5/5.1 before upgrading to Xen 3.2 
and they worked just fine. The only problems I had (and still have) were 
with DHCP.

Kai

-- 
Kai Sch?tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com