zfs discuss - Apr 2010 - OpenSolaris snv_134 zfs pool hangs after some time with dedup=on

Sorry for the double post but I think this was better suite for zfs forum.

I am running OpenSolaris snv_134 as a file server in a test environment, testing
deduplication. I am transferring large amount of data from our production server
via using rsync.

The Data pool is on a separated raidz1-0 pool. The pool was running with the
dedup & compression turned on. The rsync would hang after a while with no
more I/O going to the raidz pool (zpool iostat) and without any error message.
the rpool is on it''s own disk is not affected. However access to the
data pool will get slower and eventually hang command such as df, du, zpool
status , zfs list. System also can not be powered off from console. It will hang
while closing zfs pool and require hard reset.

I have now turned the dedup off on the pools and the rsync seem to be going
further than before. Is this a known bug? Is there an workaround for this
without rebooting the system? I am not an Solaris expert and I haven''t
worked on Solaris since 5.8. I would be happy to provide additional information
if needed.

Thanks. 

p.s. Since I turn off the dedup, I have finished two additional volume sync
without issue.
-- 
This message posted from opensolaris.org

On Wed, Apr 28, 2010 at 1:51 PM, Jim Horng <jhorng at stretchinc.com>
wrote:
> I have now turned the dedup off on the pools and the rsync seem to be going
further than before. Is this a known bug? Is there an workaround for this
without rebooting the system? I am not an Solaris expert and I haven''t
worked on Solaris since 5.8. I would be happy to provide additional information
if needed.
Dedup can use a lot of RAM. If your host has limited memory, you will
eventually hit a brick wall on performance.

Adding more memory or a SSD L2ARC may help.

-B

-- 
Brandon High : bhigh at freaks.com

This is not a performance issue.  The rsync will hang hard and one of the child
process can not be killed (I assume it''s the one running on the zfs). 
the command gets slower I am referring to the output of the file system commands
(zpool, zfs, df, du, etc) from the different shell.  I left the system running
idle for couple of days when it first happen and none of hanging process return.
The server had 16 GB of RAM and the data pool is only 6x 1TB hard drive in one
raidz vdev. (show up as a 4.4 TB zfs pool).

What''s the size of the L2ARC needed for dedup a 4.4TB volume?
-- 
This message posted from opensolaris.org

On Wed, Apr 28, 2010 at 5:09 PM, Jim Horng <jhorng at stretchinc.com>
wrote:
> This is not a performance issue. ?The rsync will hang hard and one of the
child process can not be killed (I assume it''s
I''ve seen a similar issue on a b133 host that has a large DDT, but I
haven''t waited very long to see if it completes. You can''t
kill it
while it''s blocking on io, but once that finishes it''ll die.
> What''s the size of the L2ARC needed for dedup a 4.4TB volume?
I''m not sure if there''s a guideline other than
"big". It depends on
the size of the DDT. Lots of unique data will give you a very large
DDT, while 100,000 copies of the same file will give you a much
smaller one.

-B

-- 
Brandon High : bhigh at freaks.com