zfs discuss - Mar 2010 - Validating alignment of NTFS/VMDK/ZFS blocks

Good evening,
I understand that NTFS & VMDK do not relate to Solaris or ZFS, but I was
wondering if anyone has any experience of checking the alignment of data blocks
through that stack?

I have a VMware ESX 4.0 host using storage presented over NFS from ZFS
filesystems (recordsize 4KB). Within virtual machine VMDK files, I have
formatted NTFS filesystems, block size 4KB. Dedup is turned on. When I run ZDB
-DD, i see a figure of unique blocks which is higher than I expect, which makes
me wonder whether any given 4KB in the NTFS filesystem is perfectly aligned with
a 4KB block in ZFS?

e.g. consider two virtual machines sharing lots of the same blocks. Assuming
there /is/ a misalignment between NTFS & VMDK/VMDK & ZFS, if
they''re not in the same order within NTFS, they don''t align,
and will actually produce different blocks in ZFS:

VM1
NTFS    1---2---3---
        AAAABBBBCCCC

          AAAABBBBCCCC
ZFS     1---2---3---4---

ZFS blocks are "  AA", "AABB" and so on ...
Then in another virtual machine, the blocks are in a different order:

VM2
NTFS    1---2---3---
        CCCCAAAABBBB

          CCCCAAAABBBB
ZFS     1---2---3---4---
ZFS blocks for this VM would be "  CC", "CCAA",
"AABB" etc. So, no overlap between virtual machines, and no benefit
from dedup.

I may have it wrong, and there are indeed 30,785,627 unique blocks in my setup,
but if there''s a mechanism for checking alignment, I''d find
that very helpful.

Thanks,
Chris
-- 
This message posted from opensolaris.org

Please excuse my pitiful example. :-)

I meant to say "*less* overlap between virtual machines", as clearly
block "AABB" occurs in both.

-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Chris Murray
Sent: 18 March 2010 18:45
To: zfs-discuss at opensolaris.org
Subject: [zfs-discuss] Validating alignment of NTFS/VMDK/ZFS blocks

Good evening,
I understand that NTFS & VMDK do not relate to Solaris or ZFS, but I was
wondering if anyone has any experience of checking the alignment of data
blocks through that stack?

I have a VMware ESX 4.0 host using storage presented over NFS from ZFS
filesystems (recordsize 4KB). Within virtual machine VMDK files, I have
formatted NTFS filesystems, block size 4KB. Dedup is turned on. When I
run ZDB -DD, i see a figure of unique blocks which is higher than I
expect, which makes me wonder whether any given 4KB in the NTFS
filesystem is perfectly aligned with a 4KB block in ZFS? 

e.g. consider two virtual machines sharing lots of the same blocks.
Assuming there /is/ a misalignment between NTFS & VMDK/VMDK & ZFS, if
they''re not in the same order within NTFS, they don''t align,
and will
actually produce different blocks in ZFS:

VM1
NTFS    1---2---3---
        AAAABBBBCCCC

          AAAABBBBCCCC
ZFS     1---2---3---4---

ZFS blocks are "  AA", "AABB" and so on ...
Then in another virtual machine, the blocks are in a different order:

VM2
NTFS    1---2---3---
        CCCCAAAABBBB

          CCCCAAAABBBB
ZFS     1---2---3---4---
ZFS blocks for this VM would be "  CC", "CCAA",
"AABB" etc. So, no
overlap between virtual machines, and no benefit from dedup.

I may have it wrong, and there are indeed 30,785,627 unique blocks in my
setup, but if there''s a mechanism for checking alignment, I''d
find that
very helpful.

Thanks,
Chris
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Not having specific knowledge of the VMDK format, I think what you are 
seeing is that there is extra data associated with maintaining the VMDK. 
If you are seeing lower dedup ratios than you would expect, it sounds 
like some of this extra data could be added to each block.

The VMDK spec appears to be open, perhaps a read through the spec might 
help understand what VMware is doing to the NTFS data - 
http://en.wikipedia.org/wiki/VMDK

   --joe

On 3/18/2010 11:44 AM, Chris Murray wrote:
> Good evening,
> I understand that NTFS&  VMDK do not relate to Solaris or ZFS, but I
was wondering if anyone has any experience of checking the alignment of data
blocks through that stack?
>
> I have a VMware ESX 4.0 host using storage presented over NFS from ZFS
filesystems (recordsize 4KB). Within virtual machine VMDK files, I have
formatted NTFS filesystems, block size 4KB. Dedup is turned on. When I run ZDB
-DD, i see a figure of unique blocks which is higher than I expect, which makes
me wonder whether any given 4KB in the NTFS filesystem is perfectly aligned with
a 4KB block in ZFS?
>
> e.g. consider two virtual machines sharing lots of the same blocks.
Assuming there /is/ a misalignment between NTFS&  VMDK/VMDK&  ZFS, if
they''re not in the same order within NTFS, they don''t align,
and will actually produce different blocks in ZFS:
>
> VM1
> NTFS    1---2---3---
>          AAAABBBBCCCC
>
>            AAAABBBBCCCC
> ZFS     1---2---3---4---
>
> ZFS blocks are "  AA", "AABB" and so on ...
> Then in another virtual machine, the blocks are in a different order:
>
> VM2
> NTFS    1---2---3---
>          CCCCAAAABBBB
>
>            CCCCAAAABBBB
> ZFS     1---2---3---4---
> ZFS blocks for this VM would be "  CC", "CCAA",
"AABB" etc. So, no overlap between virtual machines, and no benefit
from dedup.
>
> I may have it wrong, and there are indeed 30,785,627 unique blocks in my
setup, but if there''s a mechanism for checking alignment, I''d
find that very helpful.
>
> Thanks,
> Chris
>

On Thu, Mar 18, 2010 at 2:44 PM, Chris Murray <chrismurray84 at
gmail.com>wrote:
> Good evening,
> I understand that NTFS & VMDK do not relate to Solaris or ZFS, but I
was
> wondering if anyone has any experience of checking the alignment of data
> blocks through that stack?
>
NetApp has a great little tool called mbrscan/mbralign....it''s free,
but I''m
not sure if NetApp customers are supposed to distribute it.

-marc
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100318/6ca6d26f/attachment.html>

I have only heard of alignment being discussed in reference to 
block-based storage (like DASD/iSCSI/FC). I''m not really sure how it 
would work out over NFS. I do see why you are asking though.

My understanding is that VMDK files are basically ''aligned''
but the
partitions inside of them may not be. You don''t state what OS you are 
using in your guests. Windows XP/2003 and older create mis-alligned 
partitions by default (within a VMDK). You would need to manually 
create/adjust NTFS partitions in those cases in order for them to 
properly fall on a 4k boundary. This could be a cause of the problem you 
are describing.

This doc from VMware is aimed at block-based storage but it has some 
concepts that might be helpful as well as info on aligning guest OS 
partitions:
http://www.vmware.com/pdf/esx3_partition_align.pdf

-Brian


Chris Murray wrote:
> Good evening,
> I understand that NTFS & VMDK do not relate to Solaris or ZFS, but I
was wondering if anyone has any experience of checking the alignment of data
blocks through that stack?
>
> I have a VMware ESX 4.0 host using storage presented over NFS from ZFS
filesystems (recordsize 4KB). Within virtual machine VMDK files, I have
formatted NTFS filesystems, block size 4KB. Dedup is turned on. When I run ZDB
-DD, i see a figure of unique blocks which is higher than I expect, which makes
me wonder whether any given 4KB in the NTFS filesystem is perfectly aligned with
a 4KB block in ZFS?
>
> e.g. consider two virtual machines sharing lots of the same blocks.
Assuming there /is/ a misalignment between NTFS & VMDK/VMDK & ZFS, if
they''re not in the same order within NTFS, they don''t align,
and will actually produce different blocks in ZFS:
>
> VM1
> NTFS    1---2---3---
>         AAAABBBBCCCC
>
>           AAAABBBBCCCC
> ZFS     1---2---3---4---
>
> ZFS blocks are "  AA", "AABB" and so on ...
> Then in another virtual machine, the blocks are in a different order:
>
> VM2
> NTFS    1---2---3---
>         CCCCAAAABBBB
>
>           CCCCAAAABBBB
> ZFS     1---2---3---4---
> ZFS blocks for this VM would be "  CC", "CCAA",
"AABB" etc. So, no overlap between virtual machines, and no benefit
from dedup.
>
> I may have it wrong, and there are indeed 30,785,627 unique blocks in my
setup, but if there''s a mechanism for checking alignment, I''d
find that very helpful.
>
> Thanks,
> Chris
>

On Thu, Mar 18, 2010 at 14:44, Chris Murray <chrismurray84 at gmail.com>
wrote:
> Good evening,
> I understand that NTFS & VMDK do not relate to Solaris or ZFS, but I
was wondering if anyone has any experience of checking the alignment of data
blocks through that stack?
It seems to me there''s a simple way to check.  Pick 4k of random data
(say, dd if=/dev/urandom of=newfile bs=4k count=1) and copy that onto
the VM filesystem.  Now write a little program to read the .vmdk file
and find that 4k of data.  Report the offset, and check offset % 4096
== 0.  This won''t help you fix things, but it''ll at least tell
you
that something is wrong.

Will

That''s a good idea, thanks. I get the feeling the remainder
won''t be zero, which will back up the misalignment theory. After a bit
more digging, it seems the problem is just an NTFS issue and can be addressed
irrespective of underlying storage system.

I think I''m going to try the process in the following link:
http://www.tuxyturvy.com/blog/index.php?/archives/59-Aligning-Windows-Partitions-Without-Losing-Data.html

With any luck I''ll then see a smaller dedup table, and better
performance!

Thanks to those for feedback,
Chris
-- 
This message posted from opensolaris.org