I wonder if someone might be so kind as to sort of "validate" this layout and confirm a few assumptions... I want to deploy a number of Linux servers, probably RHEL5/CentOS5 (as both DOM0 and multiple DOMU roles) and already have a FC RAID and some FC cards. I''ve toyed around with this on a stand-alone machine and am pretty sure the concept will translate over to a shared storage system on FC. Oh and yes I''ve read a tremendous amount of info, blog posts, how-to''s, etc.. Many thanks to all who have shared info which has helped me (and others) better understand. I''ve configured and maintained GFS before. It''s great technology and I''m trying to decide if I need it for this server virtualization project or not. If the VM''s live inside "files" then obviously those files would need to be accessible by any dom0 physical node right (hence GFS or similar)? GFS can have serious performance issues though, especially with very large files. The alternative is to carve out a chunk of disk space, a LUN or possibly an LV as seen by the DOM0''s and just expose it to the DOMU''s. The latter is what I''m leaning towards, it would seem to offer a lot of advantages. If it will work as I think it will that is... What I see is servers with FC cards attached to a switch and to an FC RAID device. All 3 have local disks which they boot from. If CLVM is installed and configured I should be able to make the FC RAID into a PV. Via CLVM all 3 boxes see the PV and can work with it with changes staying in lockstep (from the LVM perspective I mean). On any node I create a VG and within that create a few LVs of various sizes. In virt-manager I then create a VM pointing to physical storage as /dev/mapper/name-of-lv-created. Testing this totally on local storage appeared to work fine. Extending that to FC/SAN storage, is this the proper approach? Will CLVM do the necessary magic to make this work without issue? I''m hoping live migration should also be possible? What I''ve also played with on a single local test setup is to pretend disk space was exhausted in the DOMU and I needed to assign it some more. From the DOM0 I just created a second LV named similarly to the main one that DOMU uses as it''s hard drive. From virt-manager I went into devices and added that LV as virtual storage. A "tail -f" of /var/log/messages in the booted DOMU instance showed xvdb just suddenly appeared. I did pvcreate/vgextend/lvextend/resize2fs to allow the DOMU VM to add that extra space and extend a filesystem onto it. The idea of a full-fledged LVM living insize of an LV was a bit strange at first. The ability to create new LVs at the DOM0 level and expose those to DOMU VM''s is simply fantastic though. Is it truly this easy? On a local internal 500GB SATA drive on a single server this appears to work prefectly. If CLVM will allow it to work on an FC shared hard drive thoguh, absolutely fantastic... Are there pitfalls or limitations I''ve not thought of here though? Is this approach a "best practices" or is some other method considered "better"? Thanks! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Sep 15, 2008 at 11:02 AM, Wendell Dingus <wendell@bisonline.com> wrote:> Are there pitfalls or limitations I''ve not thought of here though? Is this > approach a "best practices" or is some other method considered "better"?AFAICS, you''re right and ready to go. the only thing i don''t like too much is using LVM inside DomU''s. in theory some scanning tools could confuse the LVM metadata inside those volumes with the ''outer'' LVM metadata and wreck the whole thing. i don''t think it really happens in practice... if you want to avoid LVM inside DomU''s, expanding a DomU drive become a little harder: you have to expand the LV from Dom0, and then somehow make that resizing known to the DomU kernel. usually that means (at least) unmounting the device, or even restarting. after that, just resize the FS. if you can tolerate a few minutes of downtime to resize a DomU, consider avoiding the ''inner'' LVM. if not, or if you''re confident no CLVM tool would confuse ''inner'' and ''outer'' LVM''s, go with your plan. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Just complementing Javier''s email, I have tested the "online" resizing inside domU''s. It does not work without reboot or (re)-mount the drive / partition. Use LVM inside a domU seems to solve this problem, but you DO need to consider Javier''s statements regarding "inner" and "outer" LVM''s. IF we could find ANY documentation about it, and make sure this nested logical volume Will work witouht problem, I would say this is the *perfect* solution. We have *exactly* same scenario here : 03 hosts (dual Quad-Core cpus, 32Gb of RAM and EMC drive with several luns). I am on the SAME research of "Best practices" to use this combination. I think we find the 02 possibilities to GO. 1) LVM inside domU (downtime 0) 2) reboot the guest (domU) to "see" the changes (I would prefer this solution, since I personally don''t like the Idea to "add" more LV''s instead of resize them. Regards Bruno Bertechini -----Mensagem original----- De: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] Em nome de Javier Guerra Enviada em: segunda-feira, 15 de setembro de 2008 13:16 Para: Wendell Dingus Cc: xen-users@lists.xensource.com Assunto: Re: [Xen-users] SAN + XEN + FC + LVM question(s) On Mon, Sep 15, 2008 at 11:02 AM, Wendell Dingus <wendell@bisonline.com> wrote:> Are there pitfalls or limitations I''ve not thought of here though? Is this > approach a "best practices" or is some other method considered "better"?AFAICS, you''re right and ready to go. the only thing i don''t like too much is using LVM inside DomU''s. in theory some scanning tools could confuse the LVM metadata inside those volumes with the ''outer'' LVM metadata and wreck the whole thing. i don''t think it really happens in practice... if you want to avoid LVM inside DomU''s, expanding a DomU drive become a little harder: you have to expand the LV from Dom0, and then somehow make that resizing known to the DomU kernel. usually that means (at least) unmounting the device, or even restarting. after that, just resize the FS. if you can tolerate a few minutes of downtime to resize a DomU, consider avoiding the ''inner'' LVM. if not, or if you''re confident no CLVM tool would confuse ''inner'' and ''outer'' LVM''s, go with your plan. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
"Javier Guerra" <javier@guerrag.com> writes:> On Mon, Sep 15, 2008 at 11:02 AM, Wendell Dingus <wendell@bisonline.com> wrote: >> Are there pitfalls or limitations I''ve not thought of here though? Is this >> approach a "best practices" or is some other method considered "better"? > > AFAICS, you''re right and ready to go. > > the only thing i don''t like too much is using LVM inside DomU''s. in > theory some scanning tools could confuse the LVM metadata inside those > volumes with the ''outer'' LVM metadata and wreck the whole thing. i > don''t think it really happens in practice...You can easily sidestep this by setting up appropriate "filter" rules in lvm.conf in you dom0s. That way pvscan won''t look into your LVs thus it won''t discover the embedded guest LVs. Still, if you shut down a guest and want to access its LVs from dom0, you are perfectly able to: just modify your filters, pvscan, and there you go. Just don''t forget to undo that before starting up the guest again. One thing I contemplated over was using "multipath" in the guest. That may give you a chance to propagate the resize into the guest without rebooting it or permanently introducing another PV in it. Like this: resize the LV in the dom0, pass it to the guest as another block device, replace the LVM mappings so that they point to the new device, then detach the original device, pvextend etc. I''m not sure multipath-tools would work in this setup, but dmsetup surely would. I never tried it, though. -- Cheers, Feri. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Sep 16, 2008 at 11:08:39AM +0200, Ferenc Wagner wrote:> "Javier Guerra" <javier@guerrag.com> writes: > > > On Mon, Sep 15, 2008 at 11:02 AM, Wendell Dingus <wendell@bisonline.com> wrote: > >> Are there pitfalls or limitations I''ve not thought of here though? Is this > >> approach a "best practices" or is some other method considered "better"? > > > > AFAICS, you''re right and ready to go. > > > > the only thing i don''t like too much is using LVM inside DomU''s. in > > theory some scanning tools could confuse the LVM metadata inside those > > volumes with the ''outer'' LVM metadata and wreck the whole thing. i > > don''t think it really happens in practice... > > You can easily sidestep this by setting up appropriate "filter" rules > in lvm.conf in you dom0s. That way pvscan won''t look into your LVs > thus it won''t discover the embedded guest LVs. Still, if you shut > down a guest and want to access its LVs from dom0, you are perfectly > able to: just modify your filters, pvscan, and there you go. Just > don''t forget to undo that before starting up the guest again. > > One thing I contemplated over was using "multipath" in the guest. > That may give you a chance to propagate the resize into the guest > without rebooting it or permanently introducing another PV in it. > Like this: resize the LV in the dom0, pass it to the guest as another > block device, replace the LVM mappings so that they point to the new > device, then detach the original device, pvextend etc. I''m not sure > multipath-tools would work in this setup, but dmsetup surely would. > I never tried it, though.Yes, dm-multipath should allow that. Check https://www.redhat.com/archives/dm-devel/2008-August/msg00033.html -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra Giraldez
2008-Sep-16 12:49 UTC
Re: RES: [Xen-users] SAN + XEN + FC + LVM question(s)
On Monday 15 September 2008, Bruno Bertechini wrote:> Just complementing Javier''s email, I have tested the "online" resizing > inside domU''s. It does not work without reboot or (re)-mount the drive / > partition.it would be interesting to know where in the chain (xend-xenbus-pvdrivers-block device) is the resize getting stuck. ideally, partscan _should_ get the new size, and if you do use ''inner'' LVM, pvresize _should_ do the trick too. does anybody knows enough about this? who should we be nagging to fix it? -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2008-Sep-16 15:39 UTC
Re: RES: [Xen-users] SAN + XEN + FC + LVM question(s)
On Tue, Sep 16, 2008 at 07:49:16AM -0500, Javier Guerra Giraldez wrote:> On Monday 15 September 2008, Bruno Bertechini wrote: > > Just complementing Javier''s email, I have tested the "online" resizing > > inside domU''s. It does not work without reboot or (re)-mount the drive / > > partition. > > it would be interesting to know where in the chain > (xend-xenbus-pvdrivers-block device) is the resize getting stuck. ideally, > partscan _should_ get the new size, and if you do use ''inner'' LVM, pvresize > _should_ do the trick too. > > does anybody knows enough about this? who should we be nagging to fix it? >There''s no way in Xen to resize domU VBD devices on the fly.. See recent discussion about this on xen-devel: http://lists.xensource.com/archives/html/xen-devel/2008-09/msg00158.html -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen <pasik@iki.fi> writes:> On Tue, Sep 16, 2008 at 11:08:39AM +0200, Ferenc Wagner wrote: > >> "Javier Guerra" <javier@guerrag.com> writes: >> >>> On Mon, Sep 15, 2008 at 11:02 AM, Wendell Dingus <wendell@bisonline.com> wrote: >>>> Are there pitfalls or limitations I''ve not thought of here though? Is this >>>> approach a "best practices" or is some other method considered "better"? >>> >>> AFAICS, you''re right and ready to go. >>> >>> the only thing i don''t like too much is using LVM inside DomU''s. in >>> theory some scanning tools could confuse the LVM metadata inside those >>> volumes with the ''outer'' LVM metadata and wreck the whole thing. i >>> don''t think it really happens in practice... >> >> You can easily sidestep this by setting up appropriate "filter" rules >> in lvm.conf in you dom0s. That way pvscan won''t look into your LVs >> thus it won''t discover the embedded guest LVs. Still, if you shut >> down a guest and want to access its LVs from dom0, you are perfectly >> able to: just modify your filters, pvscan, and there you go. Just >> don''t forget to undo that before starting up the guest again. >> >> One thing I contemplated over was using "multipath" in the guest. >> That may give you a chance to propagate the resize into the guest >> without rebooting it or permanently introducing another PV in it. >> Like this: resize the LV in the dom0, pass it to the guest as another >> block device, replace the LVM mappings so that they point to the new >> device, then detach the original device, pvextend etc. I''m not sure >> multipath-tools would work in this setup, but dmsetup surely would. >> I never tried it, though. > > Yes, dm-multipath should allow that. > Check https://www.redhat.com/archives/dm-devel/2008-August/msg00033.htmlWell, yes... Once you are into such business, dm-multipath doesn''t buy you that much at all. One could replace the backing device in the LVM tables as well. -- Hmm, maybe using dm-multipath would remove the possibility of deadlock when pausing the root and swap devices. And is also easier to manage. But then you have to carry it all the time. Its overhead is probably negligible, though. This is interesting... -- Feri. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users