zfs discuss - Jul 2010 - Tips for ZFS tuning for NFS store of VM images

I am using OpenSolaris to host VM images over NFS for XenServer.  I''m
looking for tips on what parameters can be set to help optimize my ZFS pool that
holds my VM images.  I am using XenServer which is running the VMs from an NFS
storage on my OpenSolaris server.  Are there parameters that I should change to
make the read and write to the VDH files perform better since these are large
files.

Thanks,
Greg
-- 
This message posted from opensolaris.org

On Jul 19, 2010, at 5:26 PM, Gregory Gee wrote:
> I am using OpenSolaris to host VM images over NFS for XenServer. 
I''m looking for tips on what parameters can be set to help optimize my
ZFS pool that holds my VM images.  I am using XenServer which is running the VMs
from an NFS storage on my OpenSolaris server.  Are there parameters that I
should change to make the read and write to the VDH files perform better since
these are large files.

There is nothing special about tuning for VMs, the normal NFS tuning applies.
Please consult the ZFS Best Practices Guide for details
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
 -- richard

-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
Enterprise class storage for everyone
www.nexenta.com

To further this question, I have been searching for a while and can''t
find any reference to the difference and benefits between zfs sharenfs and nfs
share.  Currently I am using standard NFS I believe.

share -F nfs -o anon=0,sec=sys,rw=xenserver0:xenserver1 /files/VM

admin at nas:/files$ zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
pool2                     40.8G   142G    19K  /pool2
pool2/VM                  40.8G   142G  38.4G  /files/VM

Is one way better than the other?  I''ve also been looking for a
migration guide from nfs to zfs sharenfs.  Any pointers?

Thanks,
Greg
-- 
This message posted from opensolaris.org

On Jul 20, 2010, at 6:14 PM, Gregory Gee wrote:
> To further this question, I have been searching for a while and
can''t find any reference to the difference and benefits between zfs
sharenfs and nfs share.  Currently I am using standard NFS I believe.
> 
> share -F nfs -o anon=0,sec=sys,rw=xenserver0:xenserver1 /files/VM
> 
> admin at nas:/files$ zfs list
> NAME                       USED  AVAIL  REFER  MOUNTPOINT
> pool2                     40.8G   142G    19K  /pool2
> pool2/VM                  40.8G   142G  38.4G  /files/VM
> 
> Is one way better than the other?  I''ve also been looking for a
migration guide from nfs to zfs sharenfs.  Any pointers?
When imported, ZFS will send the contents of the sharenfs parameter
to share(1m) along with the file share type (nfs) and file system mount point 
(eg /files/VM).  Other than that, they are the same.

For a standalone system, it probably is a wash.  For clusters, it is very 
nice to be able to administer the shares once and have that administration
travel along with the file system instead of being located elsewhere, like
/etc/dfstab

-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
Enterprise class storage for everyone
www.nexenta.com

Thanks.  I guess I am in a ''If it ain''t broken, don''t
fix it'' for my NFS setup.

Thanks,
Greg
-- 
This message posted from opensolaris.org

Richard Elling wrote:
> Gregory Gee wrote:
> > I am using OpenSolaris to host VM images over NFS for XenServer. 
I''m looking
>for tips on what parameters can be set to help optimize my ZFS pool that
holds
>my VM images.
> There is nothing special about tuning for VMs, the normal NFS tuning
applies.

That''s not been my experience. Out of the box VMware server would not
work with
the VMs stored on a zfs pool via NFS. I''ve not yet found out why but
the
analytics showed millions of getattr/access/lookup compared to read/write.

A partial workaround was to turn off access time on the share and to mount with 
noatime,actimeo=60

But that''s not perfect because when left along the VM got into a
"stuck" state.
I''ve never seen that state before when the VM was hosted on a local
disk.
Hosting VMs on NFS is not working well so far...

Hmmm, that''s odd. I have a number of VMs running on NFS (hosted on ESX,
rather than Xen) with no problems at all. I did add a SLOG device to get
performance up to a reasonable level, but it''s been running flawlessly
for a few months now. Previously I was using iSCSI for most of the connections,
but with the addition of the SLOG device NFS has become feasible.

All I''m using on the OSOL side is sharenfs=anon=0 and adding the
server''s addresses to /etc/hosts to permit access. Running osol2009.06.

Cheers,

Erik

On 28 juil. 2010, at 21:11, sol wrote:
> Richard Elling wrote:
>> Gregory Gee wrote:
>>> I am using OpenSolaris to host VM images over NFS for XenServer. 
I''m looking
>> for tips on what parameters can be set to help optimize my ZFS pool
that holds
>> my VM images.
>> There is nothing special about tuning for VMs, the normal NFS tuning
applies.
> 
> 
> That''s not been my experience. Out of the box VMware server would
not work with
> the VMs stored on a zfs pool via NFS. I''ve not yet found out why
but the
> analytics showed millions of getattr/access/lookup compared to read/write.
> 
> A partial workaround was to turn off access time on the share and to mount
with
> noatime,actimeo=60
> 
> But that''s not perfect because when left along the VM got into a
"stuck" state.
> I''ve never seen that state before when the VM was hosted on a
local disk.
> Hosting VMs on NFS is not working well so far...
> 
> 
> 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> -----Original Message-----
> From: zfs-discuss-bounces at opensolaris.org 
> [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of sol
> Sent: Wednesday, July 28, 2010 3:12 PM
> To: Richard Elling; Gregory Gee
> Cc: zfs-discuss at opensolaris.org
> Subject: Re: [zfs-discuss] Tips for ZFS tuning for NFS store 
> of VM images
> 
> Richard Elling wrote:
> > Gregory Gee wrote:
> > > I am using OpenSolaris to host VM images over NFS for 
> XenServer.  I''m looking 
> >for tips on what parameters can be set to help optimize my 
> ZFS pool that holds 
> >my VM images.
> > There is nothing special about tuning for VMs, the normal 
> NFS tuning applies.
> 
> 
> That''s not been my experience. Out of the box VMware server 
> would not work with 
> the VMs stored on a zfs pool via NFS. I''ve not yet found out 
> why but the 
> analytics showed millions of getattr/access/lookup compared 
> to read/write.
> 
> A partial workaround was to turn off access time on the share 
> and to mount with 
> noatime,actimeo=60
> 
> But that''s not perfect because when left along the VM got 
> into a "stuck" state. 
> I''ve never seen that state before when the VM was hosted on a 
> local disk. 
> Hosting VMs on NFS is not working well so far...
My guess is that it''s a VMware Server + NFS client issue, not a VMs on
NFS issue. I''m using an OpenSolaris b134 system as an
''experimental'' NFS datastore for VMware vSphere/ESX and it
works great. I''ve had as many as 30 mixed-IO VMs running on the system
with no reported issues.

-Will

On Jul 28, 2010, at 12:11 PM, sol wrote:
> Richard Elling wrote:
>> Gregory Gee wrote:
>>> I am using OpenSolaris to host VM images over NFS for XenServer. 
I''m looking
>> for tips on what parameters can be set to help optimize my ZFS pool
that holds
>> my VM images.
>> There is nothing special about tuning for VMs, the normal NFS tuning
applies.
> 
> 
> That''s not been my experience. Out of the box VMware server would
not work with
> the VMs stored on a zfs pool via NFS.
I do this regularly and know many people who run this way.
> I''ve not yet found out why but the 
> analytics showed millions of getattr/access/lookup compared to read/write.
These are requests from the client being serviced by the server.
To find out why the client is sending such requests, you''ll need to 
look at the client.
> A partial workaround was to turn off access time on the share and to mount
with
> noatime,actimeo=60
Yes, these are common NFS tunables.
 -- richard
> But that''s not perfect because when left along the VM got into a
"stuck" state.
> I''ve never seen that state before when the VM was hosted on a
local disk.
> Hosting VMs on NFS is not working well so far...



-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
Enterprise class storage for everyone
www.nexenta.com

On Jul 28, 2010, at 3:11 PM, sol wrote:
> A partial workaround was to turn off access time on the share and to mount
with
> noatime,actimeo=60
> 
> But that''s not perfect because when left along the VM got into a
"stuck" state.
> I''ve never seen that state before when the VM was hosted on a
local disk.
> Hosting VMs on NFS is not working well so far...
We host a lot of VMs on NFS shares (from a 7000 series) on ESXi with no issues
other than an occasional Ubuntu machine that would do something similar to what
you describe.  For us it was this:

http://communities.vmware.com/thread/237699?tstart=30

when the timeout is set to 180 the issue has been completely eliminated.  The
current ESXi (and I think all versions of 4) VMWare tools does this properly.

Also, EMC and NetApp have this description of using NFS shares for VMWare:

http://virtualgeek.typepad.com/virtual_geek/2009/06/a-multivendor-post-to-help-our-mutual-nfs-customers-using-vmware.html

which by and large applies to any NFS server rather than just their equipment. 
We found it helpful, but nothing is that surprising in it.

Good luck,
Ware