zfs discuss - Mar 2011 - NTFS on NFS and iSCSI always generates small IO''s

Hi All,

I''ve run into a problem with my OpenSolaris system and NTFS, I
can''t seem to make sense of it.

The server running the virtual machines is Ubuntu Server 10.04 running KVM.
Storage is presented via NFS over Infiniband. ZFS is not running compression or
dedup. Zil is also currently disabled because it was causing terrible NFS
performance.
ESXi also displayed the same behaviour when running Windows VM''s. 

OpenSolaris box is running -
OS b147
Xeon 5520
24GB DDR3
24x 1.5TB 7200rpm sata drives
24 bay expander connected to an LSI raid card via 4 lane SAS cable
2x 64GB Intel X25-M L2ARC
2x 32GB Intel X25-M SLOG

Host -
4KB reads & writes are full speed. Tested using DD. Also tested using Samba
writing to the NFS share (Bare metal PC -> samba -> NFS) and it was
writing as fast as the server could dish out the data (50MB/s).

Test VM 1 -
OS is Ubuntu Server 10.10 on ext4. Reads & writes are full speed at 4kb,
tested using DD. Sequential read/write seems fine. Tested using Samba also, same
results as above (50MB/s).

Test VM 2-
OS is Windows Server 2008 on NTFS 4kb cluster size. 4KB reads & writes are
30MB/s, tested using the ATTO disk benchmark. Sequential reads and writes max
out at 30MB/s also and generate a lot of IO on the storage box.
IOStat on the storage box shows 3k+ IOPS and only 30MB/s throughput. 

Also tested a bare metal system running iSCSI over gigabit. NTFS (default 4kb
cluster size) directly on the ZFS block volume. Results are the same as Test VM
2, lots of small IO (3k+ IOPS) and small throughput 30MB/s
I also did the same test using SRP. 4kb reads and writes have the same 30MB/s.
Any IO above or equal to 128kb seems to pick up speed drastically, easily
700MB/s.

I have a feeling it''s to do with ZFS''s recordsize property but
haven''t been able to find any solid testing done with NTFS.
I''m going to do some testing using smaller record sizes tonight to see
if that helps the issue.
At the moment I''m surviving on cache and am quickly running out of
capacity.

Can anyone suggest any further tests or have any idea about what''s
going on?

Thanks
-Matt

On Thu, Mar 10, 2011 at 12:15 AM, Matthew Anderson
<matthewa at ihostsolutions.com.au> wrote:
> I have a feeling it''s to do with ZFS''s recordsize
property but haven''t been able to find any solid testing done with
NTFS. I''m going to do some testing using smaller record sizes tonight
to see if that helps the issue.
> At the moment I''m surviving on cache and am quickly running out of
capacity.
>
> Can anyone suggest any further tests or have any idea about what''s
going on?
The default blocksize for a zfs volume is 8k, so 4k writes will
probably require a read as well. You can try creating a new volume
with volblocksize set to 4k and see if that helps. The value can''t be
changed once set, so you''ll have to make a new dataset.

Make sure the "wcd" property is set to "false" for the
volume in
stmfadm in order to enable the write cache. It shouldn''t make a huge
difference with the zil disabled, but it certainly won''t hurt.

-B

-- 
Brandon High : bhigh at freaks.com

On Mar 10, 2011, at 12:15 AM, Matthew Anderson wrote:
> Hi All,
> 
> I''ve run into a problem with my OpenSolaris system and NTFS, I
can''t seem to make sense of it.
> 
> The server running the virtual machines is Ubuntu Server 10.04 running KVM.
Storage is presented via NFS over Infiniband. ZFS is not running compression or
dedup. Zil is also currently disabled because it was causing terrible NFS
performance.
Default recordsize for NFS is 128K. For the VM case, you will want to match the
block size of
the clients. However, once the file (on the NFS server) is created with 128K
records, it will remain
at 128K forever. So you will need to create a new VM store after the recordsize
is tuned.
 -- richard

On Thu, Mar 10, 2011 at 9:45 AM, Richard Elling
<richard.elling at gmail.com> wrote:
> Default recordsize for NFS is 128K. For the VM case, you will want to match
the block size of
> the clients. However, once the file (on the NFS server) is created with
128K records, it will remain
> at 128K forever. So you will need to create a new VM store after the
recordsize is tuned.
You can change the recordsize and copy the vmdk files on the nfs
server, which will re-write them with a smaller recordsize.

-B

-- 
Brandon High : bhigh at freaks.com

And make sure you align your NTFS partition regardless off the
underlying storage. Windows 2003 and before DONT do this by default, 7
and 2008 choose a default offset off 1Mb. But better check it in
advance with diskpart. Lastly format  your NTFS filesystem with an
appropriate cluster size. Which needs to be determined on a case by
case basis. Large video file are a different workload than an exchange
server.
I''ve got a set of thin provisioned vmdk''s of varying size and
cluster
size. Which I can clone for new machines. This saves me the diskpart
step, which you cannot run when installing windows.

On Thu, Mar 10, 2011 at 6:45 PM, Richard Elling
<richard.elling at gmail.com> wrote:
>
> On Mar 10, 2011, at 12:15 AM, Matthew Anderson wrote:
>
>> Hi All,
>>
>> I''ve run into a problem with my OpenSolaris system and NTFS, I
can''t seem to make sense of it.
>>
>> The server running the virtual machines is Ubuntu Server 10.04 running
KVM. Storage is presented via NFS over Infiniband. ZFS is not running
compression or dedup. Zil is also currently disabled because it was causing
terrible NFS performance.
>
> Default recordsize for NFS is 128K. For the VM case, you will want to match
the block size of
> the clients. However, once the file (on the NFS server) is created with
128K records, it will remain
> at 128K forever. So you will need to create a new VM store after the
recordsize is tuned.
> ?-- richard
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Thu, March 10, 2011 13:04, wessels wrote:
> And make sure you align your NTFS partition regardless off the
> underlying storage. Windows 2003 and before DONT do this by default, 7
> and 2008 choose a default offset off 1Mb. But better check it in
> advance with diskpart. Lastly format  your NTFS filesystem with an
> appropriate cluster size. Which needs to be determined on a case by
> case basis. Large video file are a different workload than an exchange
> server.
Or have separate VMDKs for your OS and data (either as a separate drive
letter or a volume mount point).