Hello List,
How can we integrate fibre channel SAN storage like VMware does with
VMFS in Ovirt
So what would be the purpose of creating such a infrastructure?
Using luns in a virtual environment is nasty because you have to control
which server the lun is attached. We only have one SAN so I can't tell
for other systems how it works.
In the SAN management you have to assign disk space to a lun which is
bounded to a FC card or multiple FC cards.
When using one lun to one host it is realy simple and you could not
screw up the filesystem on it because only one machine is using the
specified lun. But when you assign one lun to multiple machines you have
to find a way that not all machines are writing to the disk at the same
time.
Cluster filesystems
This can be done using cluster filesystems where you create an gfs or
ocfs filesystem which or both cluster aware. The thing is that you need
to setup central locking ability's on these filesystems to make sure
everything stays in sync and one node cannot access files which are
already opened on another node accessing the files.
LVM
When using LVM on a SAN lun gives you flexibility because you can resize
logical volumes in a volume group if the lun is big enough. The problem
of locking stays when you say I can open the same logical volume on
multiple machines but that would be the same for iscsi. Our experience
is very good using a volume group on a lun of 1TB which is attached to
multiple nodes. The only thing you have to make sure is don't start a
virtual machine on multiple nodes. So you have to find a way that when a
node is unreachable which had vm's running on it you would have to kill
the server before starting the vm on a new node. Only this way you can
make sure that your vm's stay consistent otherwise you will end up with
a node running on mutiple servers crashing the entire file system. (our
experience: rebuild the entire machine). The better way would be to let
the administrator login to the node from the normal lan and bring the
systems down manualy and start them on another node.
When you are going to look how VMware uses a lun then it is almost the
same as what LVM can do for us. I have contacted a storage specialist
with dell, and he confirmed my guessing. VMware uses two types of SAN
storage within its products. You can assign a lun raw to a vm so you can
give it priority on SAN level (QOS for high io workloads), or you can
assign one big lun to the nodes which shares the load on all vm's that
are running from it. The VMFS is a cluster aware filesystem so it can
lock the files which are already in use. This is no problem when the
node dies and the file has no locking anymore so you can start the vm on
another node. It still is a problem when for some reason the management
lan goes down for one node and the vm's are still running. The
management does not know if the vm's are running so it does nothing
because the file on VMFS are still locked. So you could manualy login to
the node and see what is the problem and you can decide as an
administrator what you gonna do with the vm's, live migration is not
possible so you should bring the system down manually.
So how can we implement this within Ovirt?
My thinking is that LVM could do the job. It is mostly implemented in
libvirt so we can make use of that. There are several options for this.
1. The easy method would be to have a LVM already on the lun so you
don't have to create one. The only thing would be to add a
volume group to Ovirt with the possibility to create LVM's on
it. It also would be nice to give administrators an option for
resizing the logical volume. So in technical terms you should
run the command vgdisplay on the nodes and capture all the
volume groups and present that to the administrator. It would be
nice to see which Volume group is available in what nodes so you
can assign the storage specifically to a group of vm's which are
administered by another department. The administrator can then
select the volume group to use and add it to the storage pool.
When you have assigned this to the Virtual Machine Pool you can
create new logical volumes within the Volume Group and assign
this to a virtual machine. This is mostly the same for iscsi
with logical volumes.
2. The second method would be to discover the luns on nodes and use
that lun to create a Volume group on it. This is more
complicated then one but is how VMware uses SAN luns so we
should also be able to implement this. The rest would be the
same as for method 1. probably roadmap?
3. The third method would be to assign a lun to a specific vm so
you could make use of the SAN tools for snapshotting and QOS.
This is apparently important for systems under heavy load. I
can't say because we haven't got any. This will be harder to
implement and should be on the roadmap I guess.
The only thing would be to keep the LVM metadata correct on all nodes so
you would have to run a command on all nodes so the metadata is updated.
Also when a node goes down first the nodes which have the Volume group
attached to it should have the LVM metadata updated so Ovirt manager
would know if complete node is down and if it is safe to start the
virtual machine on another node. Another way could be using CLVM but
then you should have multiple nodes before you could create the cluster
lvm.
The pro's for an LVM storage are:
It is very quick when creating a disk for a virtual machine.
It has the ability to snapshot the volume.
It is easy to use migration and it is realy fast
and did I already mention that it is realy fast?
The con's are
I cannot think off con's over using iscsi or nfs it is just faster.
I have tested a bit on my local disk and here are the results:
other linux dist so it could be a little bit different
1. Create a local logical volume in a volume group
lvcreate -n testlvm -L+1G VGname
2. show the logical volume with lvdisplay: here it says "# open 0"
3. start virt manager and assign the lvm as a raw
device /dev/mapper/VGname_testlvm to a new vm
4. start the vm
5. issue the command lvdisplay and it will say "# open 1" for the
testlvm
this could be the locking mechanism i guess but i'm not sure.
A clvm tech guy could maybe give an answer to that.
With kind regards
William van de Velde