Hi everyone! I have 2 hosts + 1 ISCSI device. I want to create a shared storage repository and both hosts use together. I wont use NFS. prepared sr: xe sr-create host-uuid=xxx content-type=user name-label=NAS1 shared=true type=iscsi device-config:target=xxxx device-config:targetIQN=xxxx hosts see the iscsi device: scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access NAS iSCSI-VDISK 0 PQ: 0 ANSI: 4 sdb: unknown partition table next step: Should i make a GFS filesystem? (XCP doesn''t include the GFS and clustering services) Best Regards, Menox _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Instead of creating an SR with ''type=iscsi'', use ''type=lvmoiscsi'' instead. This will create an LVM volume group on the iSCSI LUN where guest disks (VDIs) will be LVs. The ''iscsi'' SR type is intended for ''LUN per VDI'' e.g. in a situation where you have data already on a LUN and you want to attach it directly to a guest. Normally XCP uses LVM to share one LUN amongst many guests. Consider also setting your Pool default SR to point to the new shared storage: # xe pool-param-set uuid=<uuid> default-SR=<sr-uuid> suspend-image-SR=<sr-uuid> Cheers, Dave From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of menox menox Sent: 14 January 2010 14:00 To: xen-users@lists.xensource.com Subject: [Xen-users] XCP - GFS - ISCSI Hi everyone! I have 2 hosts + 1 ISCSI device. I want to create a shared storage repository and both hosts use together. I wont use NFS. prepared sr: xe sr-create host-uuid=xxx content-type=user name-label=NAS1 shared=true type=iscsi device-config:target=xxxx device-config:targetIQN=xxxx hosts see the iscsi device: scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access NAS iSCSI-VDISK 0 PQ: 0 ANSI: 4 sdb: unknown partition table next step: Should i make a GFS filesystem? (XCP doesn''t include the GFS and clustering services) Best Regards, Menox _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jan 14, 2010 at 11:11:38PM +0000, Dave Scott wrote:> Hi, > > Instead of creating an SR with `type=iscsi'', use `type=lvmoiscsi'' instead. > This will create an LVM volume group on the iSCSI LUN where guest disks > (VDIs) will be LVs. >I haven''t looked at the XCP code yet, but are there some special patches for LVM to make it work in a shared environment on multiple hosts? I guess it''s not CLVM, since you support snapshots.. so xapi is doing some coordination of management commands and making sure only one LVM command is issued at a time? And after making changes to the LVM configuration xapi runs ''vgscan'' on every host to pick up the changes? Thanks! -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Pasi, [cc:d Julian who is responsible for storage in XCP]> I haven''t looked at the XCP code yet, but are there some special > patches for > LVM to make it work in a shared environment on multiple hosts? > > I guess it''s not CLVM, since you support snapshots.. so xapi is doing > some coordination of management commands and making sure only one LVM > command is issued at a time?Julian could describe the detail better than me but my high-level understanding is: * xapi nominates one host to be the ''SR master'': all LVM metadata-changing commands are run here * all hosts are allowed to map/unmap LVs so the LVM commands were patched to make absolutely sure they didn''t attempt to change any metadata * unless you request a special "raw" LV, vhd metadata is added to the LV: this is how we handle snapshots> And after making changes to the LVM configuration xapi runs ''vgscan'' on > every host to pick up the changes?I''ll defer to Julian for this one :-) Cheers, Dave _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jan 15, 2010 at 11:56:46AM +0000, Julian Chesterfield wrote:> Dave Scott wrote: > >Hi Pasi, > > > >[cc:d Julian who is responsible for storage in XCP] > > > > > >>I haven''t looked at the XCP code yet, but are there some special > >>patches for > >>LVM to make it work in a shared environment on multiple hosts? > >> > >>I guess it''s not CLVM, since you support snapshots.. so xapi is doing > >>some coordination of management commands and making sure only one LVM > >>command is issued at a time? > >> > > > >Julian could describe the detail better than me but my high-level > >understanding is: > > > >* xapi nominates one host to be the ''SR master'': all LVM metadata-changing > >commands are run here > > > >* all hosts are allowed to map/unmap LVs so the LVM commands were patched > >to make absolutely sure they didn''t attempt to change any metadata > > > >* unless you request a special "raw" LV, vhd metadata is added to the LV: > >this is how we handle snapshots > > > Yep, this is correct. We use XAPI as the "Cluster lock manager" > essentially. There is a strict notion of ordering of events, and XAPI > always ensures that there is a single SRMaster for any shared SR. The SR > master is the only entity that modifies LVM metadata, and it > strategically refreshes slaves as necessary. Typically slaves only > operate in an LVM Read-only mode, so the LVM metadata is refreshed when > a slave needs to access a new logical volume, and the slave is only > allowed to create device-mapper nodes, never to modify the LVM metadata. > There are patches to LVM to add an explicit ''master'' flag, this ensures > that non-masters never attempt to repair LVM metadata if ever it is read > and found to be inconsistent. In practice this would never happen due to > the way LVM updates its metadata and the fact that we do not allow > shared Volume Groups that span more than one LUN, however it''s an > important safety catch. >Thank you both for answers! This is good explanation of how it works.> Snapshot and clone support is provided via the VHD layer that resides > above raw Logical Volumes. i.e. we create VHD Copy-on-write instances in > the same way as the file-based VHD support (e.g. NFS or local Ext3 > partitions). >Oh, so XenServer/XCP doesn''t use LVM snapshots at all? That''s good to know. Is there some commandline tool to control the VHD snapshots? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
So lvmiscsi is working properly. I had a problem with ISCSI because XCP doesn''t support 4KB block size only 512B block size . It would be nice a graphical interface as XenCenter. Citrix Xenserver 5.5 and XenCenter free software without HA and workload balance. I guess XCP include this features, but without GUI at the moment. Menox 2010/1/15 Pasi Kärkkäinen <pasik@iki.fi>> On Fri, Jan 15, 2010 at 11:56:46AM +0000, Julian Chesterfield wrote: > > Dave Scott wrote: > > >Hi Pasi, > > > > > >[cc:d Julian who is responsible for storage in XCP] > > > > > > > > >>I haven''t looked at the XCP code yet, but are there some special > > >>patches for > > >>LVM to make it work in a shared environment on multiple hosts? > > >> > > >>I guess it''s not CLVM, since you support snapshots.. so xapi is doing > > >>some coordination of management commands and making sure only one LVM > > >>command is issued at a time? > > >> > > > > > >Julian could describe the detail better than me but my high-level > > >understanding is: > > > > > >* xapi nominates one host to be the ''SR master'': all LVM > metadata-changing > > >commands are run here > > > > > >* all hosts are allowed to map/unmap LVs so the LVM commands were > patched > > >to make absolutely sure they didn''t attempt to change any metadata > > > > > >* unless you request a special "raw" LV, vhd metadata is added to the > LV: > > >this is how we handle snapshots > > > > > Yep, this is correct. We use XAPI as the "Cluster lock manager" > > essentially. There is a strict notion of ordering of events, and XAPI > > always ensures that there is a single SRMaster for any shared SR. The SR > > master is the only entity that modifies LVM metadata, and it > > strategically refreshes slaves as necessary. Typically slaves only > > operate in an LVM Read-only mode, so the LVM metadata is refreshed when > > a slave needs to access a new logical volume, and the slave is only > > allowed to create device-mapper nodes, never to modify the LVM metadata. > > There are patches to LVM to add an explicit ''master'' flag, this ensures > > that non-masters never attempt to repair LVM metadata if ever it is read > > and found to be inconsistent. In practice this would never happen due to > > the way LVM updates its metadata and the fact that we do not allow > > shared Volume Groups that span more than one LUN, however it''s an > > important safety catch. > > > > Thank you both for answers! This is good explanation of how it works. > > > Snapshot and clone support is provided via the VHD layer that resides > > above raw Logical Volumes. i.e. we create VHD Copy-on-write instances in > > the same way as the file-based VHD support (e.g. NFS or local Ext3 > > partitions). > > > > Oh, so XenServer/XCP doesn''t use LVM snapshots at all? That''s good to know. > > Is there some commandline tool to control the VHD snapshots? > > -- Pasi > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jan 15, 2010 at 02:06:30PM +0200, Pasi Kärkkäinen wrote:> On Fri, Jan 15, 2010 at 11:56:46AM +0000, Julian Chesterfield wrote: > > Dave Scott wrote: > > >Hi Pasi, > > > > > >[cc:d Julian who is responsible for storage in XCP] > > > > > > > > >>I haven''t looked at the XCP code yet, but are there some special > > >>patches for > > >>LVM to make it work in a shared environment on multiple hosts? > > >> > > >>I guess it''s not CLVM, since you support snapshots.. so xapi is doing > > >>some coordination of management commands and making sure only one LVM > > >>command is issued at a time? > > >> > > > > > >Julian could describe the detail better than me but my high-level > > >understanding is: > > > > > >* xapi nominates one host to be the ''SR master'': all LVM metadata-changing > > >commands are run here > > > > > >* all hosts are allowed to map/unmap LVs so the LVM commands were patched > > >to make absolutely sure they didn''t attempt to change any metadata > > > > > >* unless you request a special "raw" LV, vhd metadata is added to the LV: > > >this is how we handle snapshots > > > > > Yep, this is correct. We use XAPI as the "Cluster lock manager" > > essentially. There is a strict notion of ordering of events, and XAPI > > always ensures that there is a single SRMaster for any shared SR. The SR > > master is the only entity that modifies LVM metadata, and it > > strategically refreshes slaves as necessary. Typically slaves only > > operate in an LVM Read-only mode, so the LVM metadata is refreshed when > > a slave needs to access a new logical volume, and the slave is only > > allowed to create device-mapper nodes, never to modify the LVM metadata. > > There are patches to LVM to add an explicit ''master'' flag, this ensures > > that non-masters never attempt to repair LVM metadata if ever it is read > > and found to be inconsistent. In practice this would never happen due to > > the way LVM updates its metadata and the fact that we do not allow > > shared Volume Groups that span more than one LUN, however it''s an > > important safety catch. > > > > Thank you both for answers! This is good explanation of how it works. > > > Snapshot and clone support is provided via the VHD layer that resides > > above raw Logical Volumes. i.e. we create VHD Copy-on-write instances in > > the same way as the file-based VHD support (e.g. NFS or local Ext3 > > partitions). > > > > Oh, so XenServer/XCP doesn''t use LVM snapshots at all? That''s good to know. >So where/how does it store the deltas between the original volume and the snapshot?> Is there some commandline tool to control the VHD snapshots? >Other than ''xe'', of course. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Jan 18, 2010 at 10:36:29AM +0000, Julian Chesterfield wrote:> Pasi Kärkkäinen wrote: > >On Fri, Jan 15, 2010 at 02:06:30PM +0200, Pasi Kärkkäinen wrote: > > > >>On Fri, Jan 15, 2010 at 11:56:46AM +0000, Julian Chesterfield wrote: > >> > >>>Dave Scott wrote: > >>> > >>>>Hi Pasi, > >>>> > >>>>[cc:d Julian who is responsible for storage in XCP] > >>>> > >>>> > >>>> > >>>>>I haven''t looked at the XCP code yet, but are there some special > >>>>>patches for > >>>>>LVM to make it work in a shared environment on multiple hosts? > >>>>> > >>>>>I guess it''s not CLVM, since you support snapshots.. so xapi is doing > >>>>>some coordination of management commands and making sure only one LVM > >>>>>command is issued at a time? > >>>>> > >>>>> > >>>>Julian could describe the detail better than me but my high-level > >>>>understanding is: > >>>> > >>>>* xapi nominates one host to be the ''SR master'': all LVM > >>>>metadata-changing commands are run here > >>>> > >>>>* all hosts are allowed to map/unmap LVs so the LVM commands were > >>>>patched to make absolutely sure they didn''t attempt to change any > >>>>metadata > >>>> > >>>>* unless you request a special "raw" LV, vhd metadata is added to the > >>>>LV: this is how we handle snapshots > >>>> > >>>> > >>>Yep, this is correct. We use XAPI as the "Cluster lock manager" > >>>essentially. There is a strict notion of ordering of events, and XAPI > >>>always ensures that there is a single SRMaster for any shared SR. The SR > >>>master is the only entity that modifies LVM metadata, and it > >>>strategically refreshes slaves as necessary. Typically slaves only > >>>operate in an LVM Read-only mode, so the LVM metadata is refreshed when > >>>a slave needs to access a new logical volume, and the slave is only > >>>allowed to create device-mapper nodes, never to modify the LVM metadata. > >>>There are patches to LVM to add an explicit ''master'' flag, this ensures > >>>that non-masters never attempt to repair LVM metadata if ever it is read > >>>and found to be inconsistent. In practice this would never happen due to > >>>the way LVM updates its metadata and the fact that we do not allow > >>>shared Volume Groups that span more than one LUN, however it''s an > >>>important safety catch. > >>> > >>> > >>Thank you both for answers! This is good explanation of how it works. > >> > >> > >>>Snapshot and clone support is provided via the VHD layer that resides > >>>above raw Logical Volumes. i.e. we create VHD Copy-on-write instances in > >>>the same way as the file-based VHD support (e.g. NFS or local Ext3 > >>>partitions). > >>> > >>> > >>Oh, so XenServer/XCP doesn''t use LVM snapshots at all? That''s good to > >>know. > >> > >> > > > >So where/how does it store the deltas between the original volume and > >the snapshot? > > > A VHD format disk is created on top of the raw Logical Volume. VHD has a > metadata format that provides sparseness and also allows chains of > dependencies to be created as differencing Copy-on-write disks via the > parent locator fields. >Ok.> > > >>Is there some commandline tool to control the VHD snapshots? > >> > >> > > > >Other than ''xe'', of course. > > > Yes. There is a vhd-util commandline tool that allows you to query VHD > headers, generate new VHD images, create CoW VHD dependency images etc... >Great. I have to try these some day. Thanks! -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi I am very much interested in getting some more information on this iscsi-LVM. I am looking at ways to get couple of these working with non-xcp vanilla xen versions. (Xen 3.1, 3.2, 3.3 one that are out there.) * Regarding LVM patches : Are these kernel patches or user space binary. * I do understand requirement that only one node should do the changes. But would you explain why is sequencing important here. * The sparseness, chaining and snapshots are very attractive features of VHD. How can one use VHD format with vanilla xen : Either file based or lvm based. To make this work we need two pieces. * utility to create and mange vhd (vhd-util ). Is this something available outside XCP? * Xen Drivers to access the disks from VM. (Any way to get this backported? I am assuming that these are part of Xen 3.4 right ? Or just part of XCP ? Thanks /Jd --- On Mon, 1/18/10, Pasi Kärkkäinen <pasik@iki.fi> wrote: From: Pasi Kärkkäinen <pasik@iki.fi> Subject: Re: [Xen-users] XCP - GFS - ISCSI To: "Julian Chesterfield" <julian.chesterfield@eu.citrix.com> Cc: "''menox menox''" <menox.menox@gmail.com>, "Dave Scott" <Dave.Scott@eu.citrix.com>, "xen-users@lists.xensource.com" <xen-users@lists.xensource.com> Date: Monday, January 18, 2010, 2:40 AM On Mon, Jan 18, 2010 at 10:36:29AM +0000, Julian Chesterfield wrote:> Pasi Kärkkäinen wrote: > >On Fri, Jan 15, 2010 at 02:06:30PM +0200, Pasi Kärkkäinen wrote: > > > >>On Fri, Jan 15, 2010 at 11:56:46AM +0000, Julian Chesterfield wrote: > >> > >>>Dave Scott wrote: > >>> > >>>>Hi Pasi, > >>>> > >>>>[cc:d Julian who is responsible for storage in XCP] > >>>> > >>>> > >>>> > >>>>>I haven''t looked at the XCP code yet, but are there some special > >>>>>patches for > >>>>>LVM to make it work in a shared environment on multiple hosts? > >>>>> > >>>>>I guess it''s not CLVM, since you support snapshots.. so xapi is doing > >>>>>some coordination of management commands and making sure only one LVM > >>>>>command is issued at a time? > >>>>> > >>>>> > >>>>Julian could describe the detail better than me but my high-level > >>>>understanding is: > >>>> > >>>>* xapi nominates one host to be the ''SR master'': all LVM > >>>>metadata-changing commands are run here > >>>> > >>>>* all hosts are allowed to map/unmap LVs so the LVM commands were > >>>>patched to make absolutely sure they didn''t attempt to change any > >>>>metadata > >>>> > >>>>* unless you request a special "raw" LV, vhd metadata is added to the > >>>>LV: this is how we handle snapshots > >>>> > >>>> > >>>Yep, this is correct. We use XAPI as the "Cluster lock manager" > >>>essentially. There is a strict notion of ordering of events, and XAPI > >>>always ensures that there is a single SRMaster for any shared SR. The SR > >>>master is the only entity that modifies LVM metadata, and it > >>>strategically refreshes slaves as necessary. Typically slaves only > >>>operate in an LVM Read-only mode, so the LVM metadata is refreshed when > >>>a slave needs to access a new logical volume, and the slave is only > >>>allowed to create device-mapper nodes, never to modify the LVM metadata. > >>>There are patches to LVM to add an explicit ''master'' flag, this ensures > >>>that non-masters never attempt to repair LVM metadata if ever it is read > >>>and found to be inconsistent. In practice this would never happen due to > >>>the way LVM updates its metadata and the fact that we do not allow > >>>shared Volume Groups that span more than one LUN, however it''s an > >>>important safety catch. > >>> > >>> > >>Thank you both for answers! This is good explanation of how it works. > >> > >> > >>>Snapshot and clone support is provided via the VHD layer that resides > >>>above raw Logical Volumes. i.e. we create VHD Copy-on-write instances in > >>>the same way as the file-based VHD support (e.g. NFS or local Ext3 > >>>partitions). > >>> > >>> > >>Oh, so XenServer/XCP doesn''t use LVM snapshots at all? That''s good to > >>know. > >> > >> > > > >So where/how does it store the deltas between the original volume and > >the snapshot? > > > A VHD format disk is created on top of the raw Logical Volume. VHD has a > metadata format that provides sparseness and also allows chains of > dependencies to be created as differencing Copy-on-write disks via the > parent locator fields. >Ok.> > > >>Is there some commandline tool to control the VHD snapshots? > >> > >> > > > >Other than ''xe'', of course. > > > Yes. There is a vhd-util commandline tool that allows you to query VHD > headers, generate new VHD images, create CoW VHD dependency images etc... >Great. I have to try these some day. Thanks! -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users