I''m currently playing with iSCSI and Xen (exporting individual DomU block devices as individual iSCSI disks) and wanted to find a safe way to use iSCSI. i.e. I want to prevent the accidental mounting of an iSCSI disk from two or more machines. Using IET as an iSCSI target I can set MaxConnections=1 in the config. This ensures that no accidental filesystem corruption happens due to two or more simultaneous mounts. However live migration is out of the question. To move a DomU from one box to another with the MaxConnections=1 safeguard I have to; Box 1: Shutdown the DomU. Perform an iSCSI logout. Box 2: Perform an iSCSI login. Start the DomU. Is there any way to perform a live migration of the above by getting Xen to run iSCSI logout/login scripts as the DomU pauses in the middle of the live migration? Are there any software hooks that can be used to do this? Or are there simpler methods to guarantee the safe/single use of iSCSI disks that still allow Xen''s live migration? Simon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I''m currently playing with iSCSI and Xen (exporting individual DomU > block devices as individual iSCSI disks) and wanted to find a safe way > to use iSCSI. i.e. I want to prevent the accidental mounting of aniSCSI> disk from two or more machines. Using IET as an iSCSI target I canset> MaxConnections=1 in the config. This ensures that no accidental > filesystem corruption happens due to two or more simultaneous mounts. > However live migration is out of the question. To move a DomU fromone> box to another with the MaxConnections=1 safeguard I have to; > > Box 1: > Shutdown the DomU. > Perform an iSCSI logout. > > Box 2: > Perform an iSCSI login. > Start the DomU. > > Is there any way to perform a live migration of the above by gettingXen> to run iSCSI logout/login scripts as the DomU pauses in the middle of > the live migration? Are there any software hooks that can be used todo> this? > > Or are there simpler methods to guarantee the safe/single use of iSCSI > disks that still allow Xen''s live migration? >It sounds like Dom0 is making the iSCSI connection and exporting the block device to DomU... is that the case? If DomU initiated the iSCSI connection then I think this would solve all of your problems, although it would create some of its own - root on iSCSI can be a bit tricky. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Harper wrote:>> I''m currently playing with iSCSI and Xen (exporting individual DomU >> block devices as individual iSCSI disks) and wanted to find a safe way >> to use iSCSI. i.e. I want to prevent the accidental mounting of an > iSCSI >> disk from two or more machines. Using IET as an iSCSI target I can > set >> MaxConnections=1 in the config. This ensures that no accidental >> filesystem corruption happens due to two or more simultaneous mounts. >> However live migration is out of the question. To move a DomU from > one >> box to another with the MaxConnections=1 safeguard I have to; >> >> Box 1: >> Shutdown the DomU. >> Perform an iSCSI logout. >> >> Box 2: >> Perform an iSCSI login. >> Start the DomU. >> >> Is there any way to perform a live migration of the above by getting > Xen >> to run iSCSI logout/login scripts as the DomU pauses in the middle of >> the live migration? Are there any software hooks that can be used to > do >> this? >> >> Or are there simpler methods to guarantee the safe/single use of iSCSI >> disks that still allow Xen''s live migration? >> > > It sounds like Dom0 is making the iSCSI connection and exporting the > block device to DomU... is that the case? > > If DomU initiated the iSCSI connection then I think this would solve all > of your problems, although it would create some of its own - root on > iSCSI can be a bit tricky. > > JamesYou''re quite correct, Dom0 is exporting block devices to the DomUs. I was trying to keep iSCSI out of the DomU''s, but maybe in the end that is the easiest way to achieve safe iSCSI with Live Migrations. A description of what may be involved with booting off iSCSI directly from within a DomU can be found here [1]. It looks like the author had to hard code the iSCSI configuration in the custom initrd.img, and change it for each DomU. The iscsi-init project [2] looks more promising, but it looks like it is no longer actively maintained. Simon [1] http://www.eludicate.com/~bolen/iscsi/#4.1_Overview [2] http://iscsi-init.sourceforge.net/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Wild wrote: ...>> > In the most recent SuSE xen implementations, there are some scripts for > block devices that will connect to an iSCSI target as it''s needed by the > domU and allows you to specify the vbd as iscsi:iqn... > > After some hacking to make them work with the latest open-iscsi, they > seem to do OK. I haven''t tried them with migration yet as I need to set > up the 2nd 16cpu box. > > MatthewInteresting, I didn''t realise Novell/SUSE had done this. In my Xen DomU configs I use disk stanzas like this: disk = [''phy:disk/by-path/ip-192.168.0.2:3260-iscsi-iqn.2007-08.com.example:storage.xen.domUname.root-lun-0,sda1,w''] It would be nice if there were some scripts that would recognise the block device as being iscsi (seems straightforward). Then the scripts could do any appropriate iSCSI logoffs/logons on the Dom0s during DomU migrations and DomU start-ups. Apart from authentication all the info needed for the logoff/logon is in the disk/by-path block device names. I''m assuming that specifying the vbd as an iscsi:iqn is a SUSE thing rather than a more distro agnostic thing, since I''ve not seen it elsewhere? Simon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Simon Capstick wrote:> Matthew Wild wrote: > ... >>> >> In the most recent SuSE xen implementations, there are some scripts for >> block devices that will connect to an iSCSI target as it''s needed by the >> domU and allows you to specify the vbd as iscsi:iqn... >> >> After some hacking to make them work with the latest open-iscsi, they >> seem to do OK. I haven''t tried them with migration yet as I need to set >> up the 2nd 16cpu box. >> >> Matthew > > Interesting, I didn''t realise Novell/SUSE had done this. In my Xen DomU > configs I use disk stanzas like this: > > disk > [''phy:disk/by-path/ip-192.168.0.2:3260-iscsi-iqn.2007-08.com.example:storage.xen.domUname.root-lun-0,sda1,w''] >That''s similar to how mine are on some older xen boxes. I had to modify a udev rule to properly manage luns and partitions from the iscsi targets.> > It would be nice if there were some scripts that would recognise the > block device as being iscsi (seems straightforward). Then the scripts > could do any appropriate iSCSI logoffs/logons on the Dom0s during DomU > migrations and DomU start-ups. >That''s certainly what these scripts are like, for creation at least. They''re just fairly simple bash scripts. Matthew -- Matthew Wild Tel.: +44 (0)1235 445173 M.Wild@rl.ac.uk URL http://www.ukssdc.ac.uk/ UK Solar System Data Centre and World Data Centre - Solar-Terrestrial Physics, Chilton Rutherford Appleton Laboratory, Chilton, Didcot, Oxon, OX11 0QX _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > You''re quite correct, Dom0 is exporting block devices to the DomUs. I was > trying to keep iSCSI out of the DomU''s, but maybe in the end that is the > easiest way to achieve safe iSCSI with Live Migrations. > > A description of what may be involved with booting off iSCSI directly from > within a DomU can be found here [1]. It looks like the author had to hard > code the iSCSI configuration in the custom initrd.img, and change it for each > DomU.There shouldn''t be a need to burn basic connection parameters into your initrd files. The init file can pick up environment variables from the kernel command line. I used this when I was testing root on nbd and gnbd configurations ... Which is to say I had the following in my domU config file (on one line) extra="NBDHOST=192.168.97.90 NBDPORT=8001 ip=192.168.97.91:1.1.1.1:192.168.97.254:255.255.255.0" I had to "break open" one of my old initrd files to extract this, but it was working :) <snip> #TAB #/bin/nbd-client host port nbd_device # The following assumes we have NBDHOST=1.2.3.4 and NBDPORT=8001 # as kernel parameters # we also need the network interface to be configured via: # ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf> /bin/nbd-client $NBDHOST $NBDPORT /dev/nbd0 -persist <snip> I''m now using aoe on a second ethernet interface (eth1) which is pretty simple, assuming udev works right ... and if it doesn''t it''s not that big a deal to build the /dev/etherd devices in your initrd. (and yes, I''m a fan of pushing the domU network disk logic into the domU, both to keep dom0 simple/stable and to allow easy migrations.) -Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users