zfs discuss - Dec 2009 - iSCSI with Deduplication, is there any point?

I''ve been using OpenSolaris for my home file server for a little over a
year now.  For most of that time I have used smb to share files out to my other
systems.  I also have a Windows Server 2003 DC and all my client systems are
joined to the domain.  Most of that time was a permissions nightmare getting smb
to jive with the domain.  The last issue I ran into was the final straw.  I
backed up all my data and executed a zpool destroy.  And it felt kinda good...

I then turned to iSCSI to access my files.  I''ve put the OpenSolaris
box on a crossover cable hooked directly to my Windows server with a couple of
gig cards.  I''ve setup iSCSI, after resolving some issues with poor
performance due to sync writes and ZIL issues, I''m pretty happy with
the setup.  I''ve got a 2.4T volume in my Raidz pool my Windows box uses
for one big NTFS volume.  Windows deals with sharing the data to all the Windows
clients and everyone is happy.  No ACL issues, no user maps, it just works. 
I''m sure a ton of nix hardcores either stopped reading already or are
puking in their hat.  Sorry to disturb, I don''t like NTFS anymore than
you.  But I picked the best option I could find and it works better for me.  If
anyone is able to educate me on a better way or a way of making smb more
reliable in a Windows Domain or any other alternative that allows me to use
Windows Domain permissions I''m all ears.

I have already run into one little snag that I don''t see any way of
overcoming with my chosen method.  I''ve upgraded to snv_129 with high
hopes for getting the most out of deduplication.  But using iSCSI volumes
I''m not sure how I can gain any benefit from it.  The volumes are a set
size, Windows sees those volumes as that size despite any sort of block level
deduplication or compression taking place on the other side of the iSCSI
connection.  I can''t create volumes that add up to more than the
original pool size from what I can tell.  I can see the pool is saving space but
it doesn''t appear to become available to zfs volumes.  Dedup being
pretty new I haven''t found much on the subject online.

So my question is...  Using zfs solely for hosting iSCSI targets.  Is there any
way to use the space gained by deduplication?
-- 
This message posted from opensolaris.org

> I have already run into one little snag that I don''t see any way
of overcoming with my chosen method. ?I''ve upgraded to snv_129 with
high hopes for getting the most out of deduplication. ?But using iSCSI volumes
I''m not sure how I can gain any benefit from it. ?The volumes are a set
size, Windows sees those volumes as that size despite any sort of block level
deduplication or compression taking place on the other side of the iSCSI
connection. ?I can''t create volumes that add up to more than the
original pool size from what I can tell. ?I can see the pool is saving space but
it doesn''t appear to become available to zfs volumes. ?Dedup being
pretty new I haven''t found much on the subject online.
Create sparse volumes. -s when you create at volume or change the
reservation on your volumes. Search for sparse in the zfs man-page.
And don''t run out of space. :-)

Cool thx, sounds like exactly what I''m looking for.  

I did a bit of reading on the subject and to my understanding I should...
Create a volume of a size as large as I could possibly need.  So, siding on the
optimistic, "zfs create -s -V 4000G tank/iscsi1".  Then in Windows
initialize and quick format it and Windows will think it is 4000G.  Obviously I
would do a quick format not a full or it would write 4000G worth of zeros or die
trying.  Although with Dedup I would presume it should be able to do that.  Is
that a good procedure or is there a better way?

Anyway, my next question is what happens when it fills up?  Also what happens
when deleted files on the NTFS partition add up to consume all the available
space.

I mean if I write a file to the NTFS volume it will write all that data to the
ZFS filesystem.  Then I delete that file and all that happens is it gets marked
as deleted, the data doesn''t actually get zeroed out so as far as ZFS
is concerned the blocks still contain data and need to be stored.  As with most
NTFS partitions it will eventually use every bit of space it sees available no
matter how many active files are there.

So using this type of thin provisioning should I run scheduled cleans on the
NTFS partition from Windows to zero out the deleted data?  Also are there any
other issues I should be aware of?
-- 
This message posted from opensolaris.org

On Sun, Dec 20, 2009 at 9:24 PM, Chris Scerbo
<cscerbo at techsolutionsinc.com> wrote:
> Cool thx, sounds like exactly what I''m looking for.
>
> I did a bit of reading on the subject and to my understanding I should...
> Create a volume of a size as large as I could possibly need. ?So, siding on
the optimistic, "zfs create -s -V 4000G tank/iscsi1". ?Then in Windows
initialize and quick format it and Windows will think it is 4000G. ?Obviously I
would do a quick format not a full or it would write 4000G worth of zeros or die
trying. ?Although with Dedup I would presume it should be able to do that. ?Is
that a good procedure or is there a better way?
>
> Anyway, my next question is what happens when it fills up? ?Also what
happens when deleted files on the NTFS partition add up to consume all the
available space.
Run "sdelete -c X:"  where X: is your drive. That should take care of
you deleted, but still occupied blocks.


-- 
Regards,
        Cyril