xen discuss - Jan 2010 - best way to replace DomU''s /export/home with a virtual drive?

I have ZFS-based SAN and an xVM-based virtualization server.  I have a 
para-virtualized 2009.6 DomU that gets it''s root disk from a zvol on
the
SAN.  I want to replace this DomU''s /export/home with another virtual 
drive mounted from the SAN (so that I can easily fresh install the 
DomU''s OS).

Following some online guides, I have achieved my goal, but I wonder if 
I''m unnecessarily having two layers of ZFS - leading to inefficiencies.
What I''ve done so far is:

In Dom0:
   - zfs create -V 1T tank/lunar-export-home
   - xm block-attach lunar-1 phy:/dev/zvol/dsk/tank/lunar-export-home 1 w

In DomU:
   - touch /reconfigure
   - shutdown -i5 -g0 -y
   - echo | pfexec format  (it shows up as "c7d1")
   - zpool create data c7d1
   - zfs snapshot -r rpool/export/home@transfer
   - zfs send rpool/export/home@transfer | zfs receive data/home
   - zfs send rpool/export/home/kent@transfer | zfs receive data/home/kent
   - zfs umount -f rpool/export/home/kent
   - zfs umount -f rpool/export/home
   - zfs set mountpoint=/export/home data/home
   - zfs set mountpoint=/export/home/kent data/home/kent
   - zfs destroy -r rpool/export/home


Is there a better strategy?  Could one mount /export/home directly (i.e. 
`mount /dev/dsk/c7d1s2 /export/home`) and, if so, would it be more 
efficient?  - what ZFS-goodness would be lost?

Thanks,
Kent

You may want to consider sharing a single /export/home from an NFS server (or
from the dom0?), and NFS mounting it from each vm. I think this has a lot of
advantages, especially if they are all "your" home directories .. now
if this is a shared customer environment, it may not be as nice. I can also see
obvious advantages of keeping them separated, even if they are all yours, but
most of them are based on using a GUI desktop on more than one at a time.

COMMENTS ON THE PROCEDURE WITHIN....


On Jan 25, 2010, at 9:54 AM, Kent Watsen wrote:
> 
> I have ZFS-based SAN and an xVM-based virtualization server.  I have a
para-virtualized 2009.6 DomU that gets it''s root disk from a zvol on
the SAN.  I want to replace this DomU''s /export/home with another
virtual drive mounted from the SAN (so that I can easily fresh install the
DomU''s OS).
> 
> Following some online guides, I have achieved my goal, but I wonder if
I''m unnecessarily having two layers of ZFS - leading to inefficiencies.
What I''ve done so far is:
> 
> In Dom0:
>  - zfs create -V 1T tank/lunar-export-home
1TB PER virtual machine, dang! :)
>  - xm block-attach lunar-1 phy:/dev/zvol/dsk/tank/lunar-export-home 1 w
I think "virsh" is the "sponsored" interface to do this, but
this should work.
> 
> In DomU:
>  - touch /reconfigure
>  - shutdown -i5 -g0 -y
You should NOT need to do the above two steps. The disk shows up instantly, if
it doesn''t, try "devfsadm -c disk"


You will probably also have to "enable" the root account as a user, or
create another user account with "privileges" that does not have its
home directory on /export/home.

>  - echo | pfexec format  (it shows up as "c7d1")
>  - zpool create data c7d1
>  - zfs snapshot -r rpool/export/home@transfer
>  - zfs send rpool/export/home@transfer | zfs receive data/home
>  - zfs send rpool/export/home/kent@transfer | zfs receive data/home/kent
you might look into "zfs send -R" instead of sending each one
separately.
>  - zfs umount -f rpool/export/home/kent
>  - zfs umount -f rpool/export/home
I would suggest something like:

zfs set mountpoint=/oldhome rpool/export/home

No need to umount. I would not suggest just umounting, I prefer to
"change" the mountpoint, in case the system reboots or something
unexpectedly. Also, remember that the "kent" dataset inherits its
mountpoint from the parent, so it will automatically move underneath.

>  - zfs set mountpoint=/export/home data/home
>  - zfs set mountpoint=/export/home/kent data/home/kent
The second set mountpoint will not be necessary, "kent" will
automatically stay under data/home unless you have previously set it statically,
in which case you can do something like

zfs inherit mountpoint data/home/kent


>  - zfs destroy -r rpool/export/home
right, only after verifying everything is ok :)
> 
> 
> Is there a better strategy?  Could one mount /export/home directly (i.e.
`mount /dev/dsk/c7d1s2 /export/home`) and, if so, would it be more efficient?  -
what ZFS-goodness would be lost?
think snapshots, although you can take them from the dom0, its not as nice.
stick with zfs goodness ;)

Tommy
> 
> Thanks,
> Kent
> 
> 
> _______________________________________________
> xen-discuss mailing list
> xen-discuss@opensolaris.org

Kent Watsen wrote:
> 
> Is there a better strategy?  Could one mount /export/home directly (i.e. 
> `mount /dev/dsk/c7d1s2 /export/home`) and, if so, would it be more 
> efficient?  - what ZFS-goodness would be lost?
It really comes down to what problem your
trying to solve...

You could use NFS as Tommy suggests...  If your
trying to get the  best perf or not impact you
NIC perf, you can use a block device as your doing.

Some useful questions...
    Can you carve up your LUNs on the SAN?
    Can you snapshot/rollback your LUNs?
       If not, you probably want to use zfs/zvols on dom0.

    Do you want to be able to snapshot and rollback
    /export/home on the guest... i.e. I would imagine
    you would want to backup at the SAN or dom0..  But
    do you also want to be able to do this on the guest.
    (i.e. you don''t have to use zfs for /export/home on
    the guest).

    If you can carve up and snapshot your LUNs, you can
    just pass through the disk to domU. i.e. ideally, you
    want to manage are the SAN source, then pass through the
    SAN disks right to the guest..

    And of course, you need to think about migration, etc..




MRJ

Tommy/Mark,

Thank you so much for your thoughts, you have helped me immensely 
crystalize my own...

First off, I don''t think I can use the NFS based solution in part 
because, as Mark says, I don''t want to put the load on my NIC but, more
importantly,  because that assumes the SAN and the and the DomU are in 
the firewall zone.  In this case, my SAN/Dom0 is in my "management" 
subnet and my DomU is in my "private" network.  The machine physically
has two NICs, one which is bound to the Dom0 and the other bridged for 
the DomUs - the ethernet cables for each plug into different VLANs 
isolated by my firewall.  I know some may question if it makes sense to 
firewall off a DomU from its Dom0, as exploits in the virtualization 
layer could render useless such precautions, but it''s what I''m
doing anyway.

That leaves me with passing a block-device that I can either mount 
directly (i.e. a UFS formatted disk?) or via ZFS (i.e. a ZFS formatted 
disk).  Since neither of you identified any major concern with my 
current ZFS approach, I think I''ll stick with it.

Mark -  what did mean by "and of course, you need to think about 
migration, etc.. "? - that sound ominous...

Tommy - thanks for the awesome line by line review!

Thanks,
Kent

ZFS is the best approach, but I agree with Mark, it would be best to send the
domU a *raw* device (target) from the SAN, rather than a zvol if possible.

I have several other ideas about the "shared" home directories, but
totally understand where you are coming from with all the firewall complex
stuff, so I will let that be. When I used zones, I had one /export/home per
physical server, with xVM I have one per virtual machine in my firewall complex,
and I haven''t sorted out what I want to do there yet.

Good luck!
Tommy



On Jan 26, 2010, at 8:49 PM, Kent Watsen wrote:
> Tommy/Mark,
> 
> Thank you so much for your thoughts, you have helped me immensely
crystalize my own...
> 
> First off, I don''t think I can use the NFS based solution in part
because, as Mark says, I don''t want to put the load on my NIC but, more
importantly,  because that assumes the SAN and the and the DomU are in the
firewall zone.  In this case, my SAN/Dom0 is in my "management" subnet
and my DomU is in my "private" network.  The machine physically has
two NICs, one which is bound to the Dom0 and the other bridged for the DomUs -
the ethernet cables for each plug into different VLANs isolated by my firewall. 
I know some may question if it makes sense to firewall off a DomU from its Dom0,
as exploits in the virtualization layer could render useless such precautions,
but it''s what I''m doing anyway.
> 
> That leaves me with passing a block-device that I can either mount directly
(i.e. a UFS formatted disk?) or via ZFS (i.e. a ZFS formatted disk).  Since
neither of you identified any major concern with my current ZFS approach, I
think I''ll stick with it.
> 
> Mark -  what did mean by "and of course, you need to think about
migration, etc.. "? - that sound ominous...
> 
> Tommy - thanks for the awesome line by line review!
> 
> Thanks,
> Kent
>

Hi Tommy,

I understand that sending a single raw device from the SAN to the DomU 
would eliminate an extra layer of ZFS and hence provide a performance 
boost but, if I understand you correctly, it would also eliminate the 
redundancy my SAN delivers while also bounding my ability to grow 
/export/home over time.

Specifically, my SAN has 24 SATA drives, which are in RAIDz2 - i.e. 
4x(4+2) - a good balance between MTTDL and IOPS.  If a take out 3 drives 
to give to the DomU to use for its LUNs, with a zpool with matched 
redundancy, I think I would be left with a less performant Dom0 zpool 
(having taken out a stripe) and a difficult to grow DomU zpool (as I 
couldn''t simply allocate more from the SAN).  That doesn''t
sound like it
justifies giving /export/home a small boost in performance...

I''m still somewhat new to storage, so please tell me if I''m
missing
something

Thanks again,
Kent


On 1/26/10 11:26 PM, Tommy McNeely wrote:
> ZFS is the best approach, but I agree with Mark, it would be best to send
the domU a *raw* device (target) from the SAN, rather than a zvol if possible.
>
> I have several other ideas about the "shared" home directories,
but totally understand where you are coming from with all the firewall complex
stuff, so I will let that be. When I used zones, I had one /export/home per
physical server, with xVM I have one per virtual machine in my firewall complex,
and I haven''t sorted out what I want to do there yet.
>
> Good luck!
> Tommy
>

Kent Watsen wrote:
 > Tommy/Mark,
 >
 > Thank you so much for your thoughts, you have helped me immensely
 > crystalize my own...
 >
 > Mark -  what did mean by "and of course, you need to think about
 > migration, etc.. "? - that sound ominous...

Do you want to be able to move your guests between different
dom0s?  If so, how do you plan to do this?




Kent Watsen wrote:
> 
> Hi Tommy,
> 
> I understand that sending a single raw device from the SAN to the DomU 
> would eliminate an extra layer of ZFS and hence provide a performance 
> boost but, if I understand you correctly, it would also eliminate the 
> redundancy my SAN delivers while also bounding my ability to grow 
> /export/home over time.
> 
> Specifically, my SAN has 24 SATA drives, which are in RAIDz2 - i.e. 
> 4x(4+2) - a good balance between MTTDL and IOPS.  If a take out 3 drives 
> to give to the DomU to use for its LUNs, with a zpool with matched 
> redundancy, I think I would be left with a less performant Dom0 zpool 
> (having taken out a stripe) and a difficult to grow DomU zpool (as I 
> couldn''t simply allocate more from the SAN).  That
doesn''t sound like it
> justifies giving /export/home a small boost in performance...
Since you are exporting zvols from your san box, you
don''t need  one in dom0.  i.e. if your using comstar and FC,
you can just pass on the disks directly to the domU.

> I''m still somewhat new to storage, so please tell me if
I''m missing
> something
Have you thought about iscsi (the comstar iscsi stack is much
faster that the stock one that zfs uses)? This allows you
to be able to easily migrate your guests between dom0s..

We extended the phy:/ notation.  The following options are
supported.
   phy:iscsi:alias/<lun>/<iscsi-alias>
   phy:iscsi:static/<server IP>/<lun>/<target id>
   phy:iscsi:discover/<lun>/<alias or target id>

This way you can export zvols as iscsi LUNs directly to
the domU.  The iscsi info doesn''t have any dom0 specific
info, so when you migrate the guest, the hotplug scripts
will find the iscsi to /dev/dsk mapping for you.

virt-install -p -n nevada -l /export/snv129.iso --nographics --noautoconsole -r
1024 --mac 00:16:3e:72:9c:a8 --disk
path=/static/192.168.0.70/0/iqn.1986-03.com.sun:02:d5ab1c26-0a7a-c6b4-98f8-d6d267eb2561,driver=phy,subdriver=iscsi
virsh attach-disk nevada
/static/10.6.70.64/0/iqn.1986-03.com.sun:02:01accb27-35a3-e45f-882d-dc4e48c5685d
xvdb --driver phy --subdriver iscsi




MRJ



> Thanks again,
> Kent
> 
> 
> On 1/26/10 11:26 PM, Tommy McNeely wrote:
>> ZFS is the best approach, but I agree with Mark, it would be best to 
>> send the domU a *raw* device (target) from the SAN, rather than a zvol 
>> if possible.
>>
>> I have several other ideas about the "shared" home
directories, but
>> totally understand where you are coming from with all the firewall 
>> complex stuff, so I will let that be. When I used zones, I had one 
>> /export/home per physical server, with xVM I have one per virtual 
>> machine in my firewall complex, and I haven''t sorted out what
I want
>> to do there yet.
>>
>> Good luck!
>> Tommy
>>    
>

>
> > Mark -  what did mean by "and of course, you need to think about
> > migration, etc.. "? - that sound ominous...
>
> Do you want to be able to move your guests between different
> dom0s?  If so, how do you plan to do this?
Oh, gotcha.  No, I only have the one Dom0 - I won''t be migrating
domains

Thanks,
Kent