sseago and I (and variously, other folks) had a somewhat longish conversation on IRC today about carving up storage with LVM. This is the second time we've beaten this horse, so hopefully we are somewhat OK now. The basic idea is that, given an iSCSI LUN (and SCSI and FC LUNs in the future), we want to either: 1) Assign the entire LUN to a guest (this is the way that ovirt works right now) 2) Carve up the LUN using LVM, and then hand out individual logical volumes to guests. Libvirt handles this case sort of implicitly; that is, you first build a storage pool for iscsi, and find all of the volumes on it. Then you build an LVM storage pool out of the iSCSI LUN, and then you can create volumes on top of that. We can follow the same sort of model for ovirt. That is, we currently have a StoragePool defined in the model, which contains 0 or more StorageVolumes. So the idea is that we now add a new type of StoragePool, LVM, which consists of one or more iSCSI StorageVolumes, and on top of that, you have a new type of StorageVolume, LVM, which is what you eventually assign to guests. Note that the above model should eventually support binding multiple iSCSI LUNs into a single LVMPool, although we won't expose that functionality to the user for the moment. Once those model changes are in place, then we just need the backend taskomatic code to handle this (I've worked on that somewhat today, so I have a pretty good handle on what it will require), and the frontend WUI pieces. This latter can get somewhat complex, so for the time being, we will have pretty rudimentary support. That is, during VM creation time, we'll allow a user to either choose an existing whole LUN for the guest, choose an already existing (but not in use) LVM volume for the guest, or carve out a new LVM volume for the guest (subject to physical disk space and user quota, naturally). Later on we'll add more complicated handling of LVM, such as deletion, resizing, etc. etc. -- Chris Lalancette
On Tue, Sep 16, 2008 at 06:37:50PM +0200, Chris Lalancette wrote:> sseago and I (and variously, other folks) had a somewhat longish conversation on > IRC today about carving up storage with LVM. This is the second time we've > beaten this horse, so hopefully we are somewhat OK now. The basic idea is that, > given an iSCSI LUN (and SCSI and FC LUNs in the future), we want to either: > > 1) Assign the entire LUN to a guest (this is the way that ovirt works right now) > 2) Carve up the LUN using LVM, and then hand out individual logical volumes to > guests. > > Libvirt handles this case sort of implicitly; that is, you first build a storage > pool for iscsi, and find all of the volumes on it. Then you build an LVM > storage pool out of the iSCSI LUN, and then you can create volumes on top of that. > > We can follow the same sort of model for ovirt. That is, we currently have a > StoragePool defined in the model, which contains 0 or more StorageVolumes. So > the idea is that we now add a new type of StoragePool, LVM, which consists of > one or more iSCSI StorageVolumes, and on top of that, you have a new type of > StorageVolume, LVM, which is what you eventually assign to guests. > > Note that the above model should eventually support binding multiple iSCSI LUNs > into a single LVMPool, although we won't expose that functionality to the user > for the moment. > > Once those model changes are in place, then we just need the backend taskomatic > code to handle this (I've worked on that somewhat today, so I have a pretty good > handle on what it will require), and the frontend WUI pieces. This latter can > get somewhat complex, so for the time being, we will have pretty rudimentary > support. That is, during VM creation time, we'll allow a user to either choose > an existing whole LUN for the guest, choose an already existing (but not in use) > LVM volume for the guest, or carve out a new LVM volume for the guest (subject > to physical disk space and user quota, naturally). Later on we'll add more > complicated handling of LVM, such as deletion, resizing, etc. etc.This looks good to me. One of the outstanding questions I have is what (if anything) to do about deleting storage volumes when a user deletes a VM, but that question goes beyond LVM specifically so I'm willing to table it for now. Take care, --Hugh
Chris Lalancette wrote:> We can follow the same sort of model for ovirt. That is, we currently have a > StoragePool defined in the model, which contains 0 or more StorageVolumes. So > the idea is that we now add a new type of StoragePool, LVM, which consists of > one or more iSCSI StorageVolumes, and on top of that, you have a new type of > StorageVolume, LVM, which is what you eventually assign to guests. > >Just to clarify above part, as I think the above gives a bit of a wrong idea. Right now, each StoragePool has 0 or more StorageVolumes. The volumes must be of the same 'type' as the pool. Thus an NfsStoragePool as 0 or more NfsStorageVolumes, an IscsiStoragePool has 0 or more IscsiStorageVolumes. Following the same pattern, an LvmStoragePool has 0 or more LvmStorageVolumes. For all 3 types, the StorageVolumes can potentially be assigned to guests. In additon, for LvmStoragePools, we have a new association defined between it and StorageVolumes. an LvmStoragePool has 1 or more "source storage volumes" (I'm open to different names here -- I just didn't want the name to be iSCSI-specific), which for the moment must be IscsiStorageVolumes. When determining which storage volumes are available for guests, we'll have to filter out storage volumes which are connected to LvmStoragePools.> Note that the above model should eventually support binding multiple iSCSI LUNs > into a single LVMPool, although we won't expose that functionality to the user > for the moment. > >Scott
Chris Lalancette wrote:> sseago and I (and variously, other folks) had a somewhat longish conversation on > IRC today about carving up storage with LVM. This is the second time we've > beaten this horse, so hopefully we are somewhat OK now. The basic idea is that, > given an iSCSI LUN (and SCSI and FC LUNs in the future), we want to either: > > 1) Assign the entire LUN to a guest (this is the way that ovirt works right now) > 2) Carve up the LUN using LVM, and then hand out individual logical volumes to > guests.How do you plan to distinguish between LVM PVs/LVs created by ovirt/libvirt on iscsi LUNs from LVM PVs/LVs created by guest OSes on directly connected iscsi LUNs? Just blindly running pvscan will run into all sorts of trouble. Steve
On Tue, Sep 16, 2008 at 06:37:50PM +0200, Chris Lalancette wrote:> sseago and I (and variously, other folks) had a somewhat longish conversation on > IRC today about carving up storage with LVM. This is the second time we'veAh, sorry I missed it.> beaten this horse, so hopefully we are somewhat OK now. The basic idea is that, > given an iSCSI LUN (and SCSI and FC LUNs in the future), we want to either: > > 1) Assign the entire LUN to a guest (this is the way that ovirt works right now) > 2) Carve up the LUN using LVM, and then hand out individual logical volumes to > guests.> > Libvirt handles this case sort of implicitly; that is, you first build a storage > pool for iscsi, and find all of the volumes on it. Then you build an LVMWhen you say volumes, do you mean PVs/VGs? or just logical disks?> storage pool out of the iSCSI LUN, and then you can create volumes on top of that.Creating the storage pool implies then that uninitialized LUNs would go through the 'pvcreate/vgcreate/vgchange' process? This would also imply that for FC you would need to create a storage pool for the FC.. HBA and from there get the SCSI LUNs? What about the asynchronous nature of the FC wherein the remote ports can show up and new disks would appear without the need to setup a storage pool? With multipath in this picture this is going to get a bit tricky. It would seem as it would sit on top of the storage pool for iSCSI. And that would mean the LVM operations MUST be disallowed on the iSCSI/FC layer but "moved" up to the multipath storage pool?
Some additional clarification based on a conversation I just had with pmyers. No changes from the basic model side of things, but we do want to explicitly exclude any LVM bits in LUNs assigned directly to VMs. So if a VM carves up one of its assigned LUNs into multiple LVs, oVirt doesn't care -- we only show the whole LUN assigned to a VM. The other thing is that we don't want to explicitly scan all unused LUNs for PVs/LVs -- rather we should do this on-demand. If we think of it in terms of the "new VM" ui. There are several types of storage available, but selecting storage will basically be a two-step process. Initially the user will be presented with a list of NFS storage pools (i.e. NFS mounts), and iSCSI storage volumes (i.e. LUNs). If an NFS pool is chosen, the user can pick an existing unallocated NFS Storage Volume (image file) or create a new NFS Volume/image file with a specified size (within the user's quota) to be 1) added to the NFS pool; and 2) attached to the VM. If an iSCSI volume (LUN) is chosen, there are a couple of options. First of all, the LUN will be scanned for PVs/LVs. These PVs/LVs will be inserted into the oVirt DB as LVM storage pools/volumes. The user will be able to choose an existing LV, create a new LV, or choose the whole LUN to attach to the VM. The latter "whole LUN" option only exists if there are not any LVs currently attached to other VMs. When a whole LUN is attached to a VM, the individual PVs/LVs are removed from the ovirt DB, since they can be overwritten at any time by the VM anyway. The other aspect of this is what is seen on the Hardware Pool/Storage tab. Right now we see NFS/iSCSI pools in the top-level list, and NFS/iSSCSI storage volumes when we drill down. For NFS, we need to allow creation/deletion of NFS Volumes (image files). For iSCSI, we need to allow the user to drill down from iSCSI Volumes (LUNs) to LVM Pools and Volumes (VGs and LVs). Two things here we need to work out: 1) drilling down will require refreshing the LVM Pool, as we only scan these on demand -- at any given time, oVirt will not have a complete picture of all LVM Pools for every LUN in the db 2) The UI is already fairly crowded with the Target -> LUN drilldown -- now we're adding another level. I'm guessing that at the moment, the best way to show this will be via another facebox popup, but we still need to work through the UI implications. Perry -- does this cover most of what we discussed? Scott Chris Lalancette wrote:> sseago and I (and variously, other folks) had a somewhat longish conversation on > IRC today about carving up storage with LVM. This is the second time we've > beaten this horse, so hopefully we are somewhat OK now. The basic idea is that, > given an iSCSI LUN (and SCSI and FC LUNs in the future), we want to either: > > 1) Assign the entire LUN to a guest (this is the way that ovirt works right now) > 2) Carve up the LUN using LVM, and then hand out individual logical volumes to > guests. > > Libvirt handles this case sort of implicitly; that is, you first build a storage > pool for iscsi, and find all of the volumes on it. Then you build an LVM > storage pool out of the iSCSI LUN, and then you can create volumes on top of that. > > We can follow the same sort of model for ovirt. That is, we currently have a > StoragePool defined in the model, which contains 0 or more StorageVolumes. So > the idea is that we now add a new type of StoragePool, LVM, which consists of > one or more iSCSI StorageVolumes, and on top of that, you have a new type of > StorageVolume, LVM, which is what you eventually assign to guests. > > Note that the above model should eventually support binding multiple iSCSI LUNs > into a single LVMPool, although we won't expose that functionality to the user > for the moment. > > Once those model changes are in place, then we just need the backend taskomatic > code to handle this (I've worked on that somewhat today, so I have a pretty good > handle on what it will require), and the frontend WUI pieces. This latter can > get somewhat complex, so for the time being, we will have pretty rudimentary > support. That is, during VM creation time, we'll allow a user to either choose > an existing whole LUN for the guest, choose an already existing (but not in use) > LVM volume for the guest, or carve out a new LVM volume for the guest (subject > to physical disk space and user quota, naturally). Later on we'll add more > complicated handling of LVM, such as deletion, resizing, etc. etc. > >