Xen users - Feb 2008 - question on routed configuration & public IP addresses

Hello.

I have been working on this for a few weeks now and am at a wall.  I am 
looking at replacing some aging equipment with some virtual servers.  We 
have public IP addresses on all of our servers.  I need to set up Xen in 
the following manner (sample, not actual, IPs given):

dom0 =	199.199.199.200

domUa =	199.199.199.219

domUb =	199.199.199.220

domUc =	199.199.199.221
	199.199.199.222
	199.199.199.223
	199.199.199.224
	199.199.199.225

KEY POINTS:
1) Each of the domU guests are HVMs, as opposed to paravirtualized. Most
will be CentOS, while 1-2 may be Gentoo/Debian.

2) I have looked and looked and can find zero/zip real world examples 
for network routing (as opposed to the bridge style). I''m presuming
that
this is what I must have since all machines need public/routable addresses.

3) In addition, the card has two NICs, and it might be best some of the
traffic to be on one NIC and some on the other for security.

4) Another key point is that, for domUc, there are several IPs listed.
The reason that is there is for our web server which has numerous IP
addresses bound to it. HTTPS likes having its own IP addresses, and we
need to be able to bind multiple IPs to that guest. Is THIS possible,
or are we prevented somehow from binding multiple IPs to a server?

5) The machine was set up ORIGINALLY using the GUI Xen tool on CentOS 
dom0 which resulted in a virtual bridge interface setup (the default) 
being configured.  That still exists even though I have replaced the 
xend-config.xsp with route statement rather than bridge statements.

Could someone help me getting the guts of a working config for the dom0 
and domU as well as any other changes I must make for this to work?  I 
would greatly appreciate it!

I''ve been banging my head on this for days, even after reading the
networking section and searching the various resource sites, list
archives and wikis.

I may have missed a resource somewhere, but I did try.

Thank you.

LT


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 8 Feb 2008, xenlist@localmail.com wrote:
> Hello.
>
> I have been working on this for a few weeks now and am at a wall.  I am 
> looking at replacing some aging equipment with some virtual servers.  We
have
> public IP addresses on all of our servers.  I need to set up Xen in the 
> following manner (sample, not actual, IPs given):
>
> dom0 =	199.199.199.200
>
> domUa =	199.199.199.219
>
> domUb =	199.199.199.220
>
> domUc =	199.199.199.221
> 	199.199.199.222
> 	199.199.199.223
> 	199.199.199.224
> 	199.199.199.225
>
> KEY POINTS:
> 1) Each of the domU guests are HVMs, as opposed to paravirtualized. Most
> will be CentOS, while 1-2 may be Gentoo/Debian.
>
> 2) I have looked and looked and can find zero/zip real world examples for 
> network routing (as opposed to the bridge style). I''m presuming
that this is
> what I must have since all machines need public/routable addresses.
>
Use the bridging (not the nat).  The machines will have public/routable.
We are doing this now.

> 3) In addition, the card has two NICs, and it might be best some of the
> traffic to be on one NIC and some on the other for security.
>
It is possible to set up two bridges, giving each xen a virtual eth0
and eth1, and map the various IP''s to each as needed.  We are doing
this.

> 4) Another key point is that, for domUc, there are several IPs listed.
> The reason that is there is for our web server which has numerous IP
> addresses bound to it. HTTPS likes having its own IP addresses, and we
> need to be able to bind multiple IPs to that guest. Is THIS possible,
> or are we prevented somehow from binding multiple IPs to a server?
This is possible but you only list one IP in the xen config itself
and then start up the others once the machine starts up.
We have one xen instance that has four IP''s right now and it is
working fine.

Steve Timm

>
> 5) The machine was set up ORIGINALLY using the GUI Xen tool on CentOS dom0 
> which resulted in a virtual bridge interface setup (the default) being 
> configured.  That still exists even though I have replaced the 
> xend-config.xsp with route statement rather than bridge statements.
>
> Could someone help me getting the guts of a working config for the dom0 and
> domU as well as any other changes I must make for this to work?  I would 
> greatly appreciate it!
>
> I''ve been banging my head on this for days, even after reading the
> networking section and searching the various resource sites, list
> archives and wikis.
>
> I may have missed a resource somewhere, but I did try.
>
> Thank you.
>
> LT
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
-- 
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Steven Timm wrote:
>> 2) I have looked and looked and can find zero/zip real world examples 
>> for network routing (as opposed to the bridge style). I''m
presuming
>> that this is what I must have since all machines need public/routable 
>> addresses.
>>
> Use the bridging (not the nat).  The machines will have public/routable.
> We are doing this now.
OK.  What I had read previously had point me in the routed direction, so 
I had not tried the bridging as much.
>> 4) Another key point is that, for domUc, there are several IPs listed.
>> The reason that is there is for our web server which has numerous IP
>> addresses bound to it. HTTPS likes having its own IP addresses, and we
>> need to be able to bind multiple IPs to that guest. Is THIS possible,
>> or are we prevented somehow from binding multiple IPs to a server?
> 
> This is possible but you only list one IP in the xen config itself
> and then start up the others once the machine starts up.
> We have one xen instance that has four IP''s right now and it is
> working fine.
I''m guessing I''ll need to turn on proxy arp on dom0 for this
to work, right?

BTW, do you mind attaching or e-mailing me directly a copy of your domU 
config so I can check it versus what I have?  I have tried so many 
different things, I''d like to see a good, clean example that is known
to
be working.  Feel free to mask/change anything you need to for privacy.

Thanks.

LT

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Steven Timm wrote:
> Use the bridging (not the nat).  The machines will have public/routable.
> We are doing this now.
I''m still having trouble with this.  I started with a clean install of 
CentOS 5.1.  I am testing it on an internal network currently until I 
get the problems worked out.

Here''s what I have:

1) I have set the dom0 server to have an IP address of 192.168.1.200/24 
on eth0.  This machine sees the actual network default gateway of 
192.168.1.1.

2) CentOS installs the following default.xml file under 
/etc/libvirt/qemu/networks:

<network>
   <name>default</name>
   <uuid>cut-out</name>
   <bridge name="virbr0" />
   <forward/>
   <ip address="192.168.122.1"
netmask="255.255.255.0">
     <dhcp>
       <range start="192.168.122.2" end="192.168.122.254"
/>
     </dhcp>
   </ip>
</network>

3) I changed the above file to use the IP address 192.168.1.199 with the 
same netmask.  I am, after all, wanting to verify that I can have dom0 
and domU on the same network as the gateway and other systems.

4) I created a logical volume for my domU.

5) I created a config file for my hardware virtual machine which looks 
like this:

****************
imprt os, re
arch = os.uname ()[4]
if re.search(''64'', arch):
	arch_libdir = ''lib64''
else:
	arch_libdir = ''lib''
kernel = "/usr/lib/xen/boot/hvmloader"
builder=''hvm''
memory = 1024
shadow_memory = 8
name = "servername"
pae=1
vif = [ ''type=ioemu, bridge=virbr0, ip=192.168.1.201'' ]
disk = [''phy:/dev/VolGroup00/lvguest,hda,w'',
''phy:/dev/hdb,hdc:cdrom,r'']
device_model = ''/usr/'' + arch_libdir +
''/xen/bin/qemu-dm''
boot=''dc''
sdl=0
vnc=1
vnclisten=''0.0.0.0''
vncunused=1
stdvga=0
serial=''pty''
*****************

6) I installed a copy of CentOS on this HVM.

7) I configured the network on this domU to use 192.168.1.201 with a /24 
netmask.

8) I turned OFF iptables on both machines to ensure there are not 
blockages there.

SO, at this point, from the domU, I can:

	ping 192.168.1.201 (theh domU itself)
	ping 192.168.1.199 (the virbr0 IP address)
	ping 192.168.1.200 (the dom0)

BUT, I cannot ping beyond the dom0.

ALSO, from dom0, I cannot ping 192.168.1.1, the default gateway, even 
though I could before hand.

If I do a netstat -rn, it appears that I have TWO network routes:

Destination	Gateway		Genmask		Iface
192.168.1.0	0.0.0.0		255.255.255.0	virbr0
192.168.1.0	0.0.0.0		255.255.255.0	eth0
169.254.0.0	0.0.0.0		255.255.0.0	eth0
0.0.0.0		192.168.1.1	0.0.0.0		virbr0

So the default route is pointing to the bridge.  I''m not sure if that
is
correct or incorrect behavior, as I have not gotten this to work yet.

I''m also guessing that at least ONE of these 192.168.1.0 network routes
is unnecessary, but I''m not sure which.

I''m also unclear as to how to fix it and what configuration files need 
to be changed in order to maintain proper behavior across reboots.

Any ideas or pointers would be appreciated.  Working examples get bonus 
points.  8^)

Thanks.

LT


-- 
Craig Thompson, President
Caldwell Global Communications, Inc.
423.559.5465 (v)
423.559.5145 (f)
"Why Surf When You Can Fly?" (TM)

Visit http://www.thompsonreviews.com for family friendly reviews

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

CENTOS USERS BEWARE!

Interestingly enough, I''ve made some headway.

After beating my head into the wall repeatedly, I got an inspiration, 
from the Lord no doubt.

In my ifconfig, I consistently saw a xenbr0 listed even though CentOS 
does not USE that bridge.  It sets up and uses virbr0 by default.

Don''t ask me why the good folks at RedHat chose to do this.

I thought, "Why don''t I pretend that virbr0 does not exist? 
Pretend
that xenbr0 is really the bridge to use."

So I changed my /etc/libvirt/qemu/networks/default.xml file to use the 
name xenbr0 instead of virbr0 (default).

I also changed my guest config file in /etc/xen to explicitly use 
''xenbr0'' in the vif statement.

I rebooted, brought up my guest, and voila!  It worked.

I can ping my default gateway from dom0.  I can ping my guest OS from dom0.

I can ping dom0 from domU.  I can ping the Internet from domU.

NOW, I beg the question, WHY did RedHat do this?  Why use virbr0 by 
default if it doesn''t work?

My routing table still looks the same as it did before:

192.168.1.0	0.0.0.0		255.255.255.0	U	0 0 0 xenbr0
192.168.1.0	0.0.0.0 	255.255.255.0	U	0 0 0 eth0
169.254.0.0	0.0.0.0 	255.255.255.0	U	0 0 0 eth0
0.0.0.0		192.168.1.1	0.0.0.0		UG	0 0 0 xenbr0

So that routing table WORKS if the proper bridge is utilized.

I hope this helps someone else.

LT

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Quoting xenlist@localmail.com:

Try this on Dom0 command line and see if you can ping out with the  
DomU beyond Dom0:


iptables -L

Does it show the FORWARD chain as DENY? If so, then issue this on the  
command line:

iptables -P FORWARD ACCEPT

Hope that helps,

Jon





_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 13 February 2008 05:42:29 pm Craig Thompson
wrote:
> NOW, I beg the question, WHY did RedHat do this?  Why use virbr0 by
> default if it doesn''t work?
As sadique explained recently (and if I remember it correctly!), virbr0 
doesn''t have peth0 attached, so there is no way it can talk to the
external
world. Do a ''brctl show'' to see where peth0 is attached. It
may be xenbr0, or
in recent xen versions, eth0. In my experience, using virt-manager/install 
usually uses the right bridge for the config, no matter what is 
in  /etc/libvirt/qemu/networks/default.xml.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users