I have refined my howto a bit. I fixed a few typos and had a few things in the
wrong order.
Although they work, I still have a few questions on these steps.
1) Do we need to create a VIF on DOM0 in order to set up the management
interface? I am starting to think not.
2) When setting up the VLAN should I connect to the BOND (as I indicated
below) or to the network that initially gets connected to the bond?
Any help on these would be great.
Thanks;
James
While I am still in the process of working through this here are the steps I am
taking to set up each machine.
1) Since each of my servers have multiple NIC ports I bond them together
(see page 34 of the Administrators Guide) This is best done from the physical
server and not via the console. If you do it remotely you will then have to
reconfigure the management port before the next step. I think it is cleaner if
you do not have to do that.
a. Shutdown all VM''s (this is easy since I am pretty much a new
installation)
i. xe
vm-list
ii. xe
vm-shutdown uuid
b. Create the network (this is like a virtual switch), write down the
first part of the UUID that is returned after this command since the next step
may cause it to scroll off the page
i. xe
network-create name-label=bond0
c. Create the actual bond (keep track of the uuid of this bond since it
will be used in step 2)
i. xe
pif-list
ii. xe
bond-create network-uuid=<UUID from above> pif-uuids=<UUID of the first
interface from the last step>,<UUID of the second interface from the last
step>
2) Next for the VLAN work. First, of course, each NIC port must be on a
trunked switch port. IN our case, we forced the port encapsulation to dot1q and
the mode to trunk with a native VLAN of 1. We did this using ranges so that we
know everything is configured the same. I am not sure if allowing the ports to
negotiate the encapsulation or making the trunk mode dynamic would have worked
but, we were not taking any chances. This way we had two fewer things to
troubleshoot if we had problems.
a. Create a new network. This will be like a VLAN specific switch for all
of your DOM''s
i. xe
network-create name-label=vlan103
b. Next tie the network to the network interface on the physical server
(in our case, the bonded interfaces) and specify the VLAN. Keep track of the
UUID returned as you will need it when we create VIF''s for the
DOM''s
i. xe
vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the bond
in step 1.c.ii> vlan=103
c. Reconfigure the IP address info. I do not use DHCP for my servers so I
make this static. Also, I do not know for sure how to enter multiple DNS
servers. You may just separate them with a comma but I have not tested that yet.
Finally, the case seems to be important here. I noticed it on the DNS keyword.
If it is lowercase it does not seem to work.
i. xe
pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static
gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK>
DNS=<DNS SERVER ADDRESS>
3) The next step is to get the management port back up and running. I am
not sure if you have to actually set up a VIF (steps a through d. Step e may be
enough.) These are the same steps for setting up VIF''s on other
VM''s.
a. get the UUID of DOM0
i. xe
vm-list
b. Create a VIF to tie DOM0 to the management VLAN (this is the VLAN I
created above). In this case I had one interface (BOND0) so I set the device to
1. Keep track of this UUID as you will need it to configure the management
interface.
i. xe
vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network
created just above in 2.b.i> device=1
c. As long as a VM is not running, you can just start it and the new
interface will be ready. For DOM0 we will have to plug it in or reboot the
server.
i. xe
vif-plug uuid=<UUID of the VIF created above>
d. Configure IP information of the VIF. First run ifconfig to get the
device and then run it again to configure it
i. ifconfig
ii. ifconfig
xapi2 <IP ADDRESS> netmask <NETMASK>
e. Configure the VLAN PIF as the management interface
i. xe
host-management-reconfigure pif-uuid=<UUID of the PIF created above 2.b.i>
f. Clean up after yourself. This will help to keep you from getting
confused later when you look at the settings.
i. xe
pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None
Notes
1) The XenNetworking WiKi page helped me wrap my head around what was
going on. http://wiki.xensource.com/xenwiki/XenNetworking
2) I found this thread on the forums that basically became my template.
http://forums.xensource.com/thread.jspa?messageID=15451㱛
3) The above thread led me to the knowledge that there is more about VLANS
in the admin guide than the index would have you believe. Page 33 - 34 tells you
how to set them up. I hope that this omission can be fixed in the next document
release.
4) I found that the UUID''s work with tab completion. This may be
obvious to everyone else but I never saw it mentioned in the manual (it may be
there but I never saw it).
5) One problem I had with configuring bridges manually in Linux instead of
doing it the Xensource way above, was that Xensource automatically removes
interfaces it does not know about. Since I was not sure how to make it aware of
things I had manually created, my interfaces and bridges would stay up for 30 -
60 seconds or so before being torn down.
6) Once you get the management interface setup, you can create links to
other VLANS using the XenCenter.
7) I have not tested yet but, my hope is that this information will travel
to each of the other machines I add to the resource pool. That is why I took the
above steps on the master server.
8) If you dig through this entire thread you will find a nugget of
information that sounds like it could cause problems if set incorrectly.
"The secret seems to be that the native VLAN on the trunkport must be 1
(which is the default)."
http://forums.xensource.com/thread.jspa?messageID=15780
9) Before you can take all of the above steps you must install your
license. This presents a Catch-22 since you would normally install it using the
XenCenter, however, this requires the network already be working. To install via
the command line you need to:
1. copy the license to a thumbdrive, take it to the server and mount it.
2. mv /etc/xensource/license /etc/xensource/license.old
3. copy the license from the thumb drive to /etc/xensource/license
4. If necessary, chmod a+x /etc/xensource/license to make the permissions
match the original file.
5. reboot the box
James Alspach
Systems Analyst II
Shasta County Office of Education
________________________________
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of James Alspach
Sent: Monday, June 09, 2008 3:37 PM
To: augusto lopes
Cc: xen-users@lists.xensource.com
Subject: RE: RE: [Xen-users] VLAN and BRIDGE HELP
You are very welcome.
One thing that I am working on now involves removing the step toward the
beginning:
xe pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static
gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK>
DNS=<DNS SERVER ADDRESS>
I do not believe that this step is necessary since we will not use the pif
directly. We will use vif''s connected to this pif (actually to the
network connected to the pif) and the vif''s will each get their own IP
addresses.
I am working on this right now and will let you know how it goes.
Thanks;
James
James Alspach
Systems Analyst II
Shasta County Office of Education
________________________________
From: augusto lopes [mailto:nhanonme@yahoo.com.br]
Sent: Monday, June 09, 2008 3:29 PM
To: James Alspach
Cc: xen-users@lists.xensource.com
Subject: RE: RE: [Xen-users] VLAN and BRIDGE HELP
Thank you VERY, VERY MUCH JAMES!
This is a great detail oriented document that answers all my noobie''s
doubts: now I should be able to set without issues. I will let you know how it
goes for me, but I truly have a good feeling about it now that you provided
me with this feed...
Thanks again for you help,
Augusto Lopes
Intern Systems Admin
Apollo System
James Alspach <jalspach@shastacoe.org> escreveu:
While I am still in the process of working through this here are the steps I am
taking to set up each machine.
Since each of my servers have multiple NIC ports I bond them together (see page
34 of the Administrators Guide) first. This is best down from the physical
server and not via the remote console. If you do it remotely you will then have
to reconfigure the management port before the next step. I think it is cleaner
if you do not have to do that.
Shutdown all VM''s (this is easy since I am pretty much a new
installation)
xe vm-list
xe vm-shutdown uuid
Create the network (this is like a virtual switch), write down the first part of
the UUID that is returned after this command since the next step may cause it to
scroll off the page
xe network-create name-label=bond0
Create the actual bond (keep track of the uuid of this bond since it will be
used in step 2)
xe pif-list
xe bond-create network-uuid=<UUID from above> pif-uuids=<UUID if the
first interface from the last step>,<UUID if the second interface from the
last step>
Reconfigure the IP address info. I do not use DHCP for my servers so I make
this static. Also, I do not know for sure how to enter multiple DNS servers.
You may just separate them with a comma but I have not tested that yet. Finally,
the case seems to be important here. I noticed it on the DNS keyword. If it is
lowercase it does not seem to work.
xe pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static
gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK>
DNS=<DNS SERVER ADDRESS>
Next for the VLAN work (see page 33 of the Administrators Guide). First, of
course, each NIC port must be on a trunked switch port. IN our case, we forced
the port encapsulation to dot1q and the mode to trunk with a native VLAN of 1.
We did this using ranges so that we know everything is configured the same. I am
not sure if allowing the ports to negotiate the encapsulation or making the
trunk mode dynamic would have worked but, we were not taking any chances. This
way we had two fewer things to troubleshoot if we had problems.
Create a new network. This will be like a VLAN specific switch for all of your
DOM''s
xe network-create name-label=vlan103
Next tie the network to the network interface on the physical server (in our
case, the bonded interfaces) and specify the VLAN. Keep track of the UUID
returned as you will need it when we create VIF''s for the
DOM''s
xe vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the
bond in step 1.c.ii> vlan=103
The next step is to get the management port back up and running. These are the
same steps for setting up VIF''s on other VM''s.
Get the UUID of DOM0
xe vm-list
Create a VIF to tie DOM0 to the management VLAN (this is the VLAN I created
above). In this case I had one interface (BOND0) so I set the device to 1.
Keep track of this UUID as you will need it to configure the management
interface.
xe vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network
created just above in 2.b.i> device=1
Configure IP information of the VIF. First run ifconfig to get the device name
and then run it again to configure it
ifconfig
ifconfig eth1 <IP ADDRESS> netmask <NETMASK>
Configure it as the management interface
xe host-management-reconfigure pif-uuid=<UUID of the VIF created above>
Clean up after yourself. This will help to keep you from getting confused later
when you look at the settings.
xe pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None
As long as a VM is not running, you can just start it and the new interface will
be ready. For DOM0 we will have to plug it in.
xe vif-plug uuid=<UUID of the VIF created above>
Notes
The XenNetworking WiKi page helped me wrap my head around what was going on.
http://wiki.xensource.com/xenwiki/XenNetworking
I found this thread on the forums that basically became my template.
http://forums.xensource.com/thread.jspa?messageID=15451㱛
The above thread led me to the knowledge that there is more about VLANS in the
admin guide than the index would have you believe. Page 33 - 34 tells you how to
set them up. I hope that this omission can be fixed in the next document
release.
I found that the UUID''s work with tab completion from the command line.
This may be obvious to everyone else but I never saw it mentioned in the manual
(it may be there but I never saw it).
One problem I had with configuring bridges manually in Linux instead of doing it
the Xensource way above was that Xensource automatically removes interfaces it
does not know about. Since I was not sure how to make it aware of things I had
manually created, my interfaces and bridges would stay up and running for 30 -
60 seconds or so before being torn down.
Once you get the management interface setup, you can create links to other VLANS
using the XenCenter and skip much of the above.
I have not tested yet but, my hope is that this information will travel to each
of the other machines I add to the resource pool. That is why I took the above
steps on the master server.
I also attached the above info as a PDF. If anyone sees any problems with what
I have described or better ways to go about it, please let me know so that I can
update this in the hope that it helps somebody somewhere.
James Alspach
P.S. Thank you for everyone who provided suggestions and help while I was (and
still am) trying to figure this all out.
James Alspach
Systems Analyst II
Shasta County Office of Education
________________________________
From: augusto lopes [mailto:nhanonme@yahoo.com.br]
Sent: Monday, June 09, 2008 9:17 AM
To: James Alspach
Subject: En: RE: [Xen-users] VLAN and BRIDGE HELP
Hello James;
I was just checking all the recent help emails and found this particular one
which is basically referring to a similar environment I would like to set up.
Basically, in my scenario I am asked to set up three guest domains (domU1 -
domU3). The first tow will provide web and mail services sequentially. And the
last one will basically be the DBM server.
Since in a normal network environment domU3 should be on a protected subnet and
the other tow on a dmz subnet, I have not been able to set up VLANs
appropriately for them. I have practically set up netfilter (iptables firewall )
on the Dom0, but do not understand VLAN concept in xen''s virtual
environment well enough to accomplish the main goal of separating each service
to its own domain.
After reading your help, I can picture the all thing better, but I am still not
sure how to bring it all about. Would you please give me some ropes regarding
this topic?
I am working with RHEL5 as the Dom0 and guest domains will be various win server
2k3 as well as rhel5.
Thank you in advance for you help.
Augusto Lopes
Intern System Admin
Apollo System
James Alspach <jalspach@shastacoe.org> escreveu:
Data: Thu, 5 Jun 2008 09:24:04 -0700
De: "James Alspach" <jalspach@shastacoe.org>
Para: "Emil Palm" <empa@cardium.se>,
<xen-users@lists.xensource.com>
Assunto: RE: [Xen-users] VLAN help
Emil -
Thank you! This gives me another avenue to explore.
So what you are saying is that one does not have to explicitly create the new
interface on the VLAN by making changes in /etc/sysconfig/network-scripts/ ? By
configuring the vlan in vconfig it builds the new interface automagically? Will
this setup survive a reboot or should I build a script to set this up each time?
In my case I have bonded two NICs (well actually two ports on the same NIC but
no need to split hairs) so I would imagine I would replace eth0 in your example
with bond0.
The host OS I am using is, I believe, CentOS. I am using the Xen Enterprise
version installed right off of the CD which uses some RedHat derived distro.
Thank you for your help
James
James Alspach
Systems Analyst II
Shasta County Office of Education
________________________________
From: Emil Palm [mailto:empa@cardium.se]
Sent: Thursday, June 05, 2008 12:35 AM
To: James Alspach; xen-users@lists.xensource.com
Subject: RE: [Xen-users] VLAN help
Hi!
I''v just recently set that kind of eviroment up. What Host OS are you
running becouse i wrote 2 diffrent howtos for Red Hat and one for SuSe.
But in theory you just have 1 interface for ex: eth0
then you do vconfig add eth0 VLANID
when that is done you create a bridge something like this:
brctl addbr brVLANID
brctl addif brVLANID eth0.VLANDID
when that is setup you should change "network-script=network-bridge"
to "network-script=network-dummy" within your xend-config.sxp so Xend
doesnt screw up your real physical interface.
When that is done just put:
vif=["mac=XX:XX:XX:XX:XX:XX,bridge=brVLANID",] in your domU
configuration file.
If you want more info just give me a mail and i will help you as the best as i
can.
Emil Palm
Cardium AB
Sweden
-----Original message-----
From: James Alspach <jalspach@shastacoe.org>
Sent: Wed 06/04/08 19:56:40
To: xen-users@lists.xensource.com;
Subject: [Xen-users] VLAN help
We are in the process of setting up a few Xensource servers whose initial
function will be to run Exchange 2007. As part of this (and for future
VM''s) I need to be able to provide access to various VLANS to the
various DOM''s.
In theory this sounds fairly straight forward: DOM0 gets a PIF for each VLAN.
This PIF connects to a VLAN specific bridge and then, for each DOM that needs
one, a VIF is created and connected to the bridge.
Does this sound correct?
If so, my question is how to specify the VLAN for a PIF. I can list it but I
am not able to set it since it is read only.
How do virtual networks fit into the above and how is a virtual network
different from a virtual bridge?
Any help or pointers to information are greatly appreciated.
Thank you for your help;
James
James Alspach
Systems Analyst II
Shasta County Office of Education
1644 Magnolia avenue
Redding, California
96003
jalspach@shastacoe.org <mailto:jalspach@shastacoe.org>
(530) 225-0293
IT Hotline: 225-0279
hotline@shastacoe.org <mailto:hotline@shastacoe.org>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
________________________________
Abra sua conta no Yahoo! Mail
<http://br.rd.yahoo.com/mail/taglines/mail/*http:/br.mail.yahoo.com/> , o
único sem limite de espaço para armazenamento!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
________________________________
Abra sua conta no Yahoo! Mail
<http://br.rd.yahoo.com/mail/taglines/mail/*http:/br.mail.yahoo.com/> , o
único sem limite de espaço para armazenamento!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users