zfs discuss - Jan 2011 - ZFS/ARC consuming all memory on heavy reads (w/ dedup enabled)

Greetings Gentlemen,

I''m currently testing a new setup for a ZFS based storage system with 
dedup enabled. The system is setup on OI 148, which seems quite stable 
w/ dedup enabled (compared to the OpenSolaris snv_136 build I used 
before).

One issue I ran into, however, is quite baffling: 
With iozone set to 32 threads, ZFS''s ARC seems to consume all available
memory, making the system completely unresponsive (no SSH or console 
access).
 
This happens (repeatably) during one of the READ tests, usually in 
iozone''s random read test. The WRITE tests work perfectly fine (and 
blazingly fast).

The system has 12G RAM. It does not matter whether a reasonably fast 
L2ARC is added to the pool.

The only thing that seems to help is setting
''primarycache=metadata'' for
the ZFS in question, which does not seem like a desirable configuration.

I have also tried setting the zfs_arc_max property to 0x80000000 (2G) 
in /etc/system. This helps a little (it won''t lock up quite as
quickly),
but at some point, it will still eat up all memory it can get.

IOzone command line: 
/usr/benchmarks/iozone/iozone -R -e -s 1g -t 32 -M -r 16k -F [...]

Primarycache property:
NAME                        PROPERTY      VALUE         SOURCE
somepool                    primarycache  all           local
rpool/swap                  primarycache  metadata      local

./arcstat.pl (last few lines before the system locks up):
    Time  read  miss  miss% dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
15:56:34  121K   247      0  243    0     4    0   178    1   151M  402M
15:56:35     0     0      0    0    0     0    0     0    0   137M  402M
15:56:36   14K    25      0   25    0     0    0    24    0   265M  402M
15:56:37  381K   575      0  575    0     0    0   574    1     4G  402M
15:56:38  362K   706      0  706    0     0    0   706    1     9G  402M
15:56:39  153K   364      0  306    0    58   18   362    2    11G  402M

zpool list 3 (last output before the system locks up):
somepool  2.72T  9.53M  2.72T     0%  34466.62x  ONLINE  -
rpool      37G  8.37G  28.6G    22%  1.00x  ONLINE  -

Is this a known issue with the ARC? Is there anything else I can do 
about it?

Also, does anyone have an idea what''s actually in the ARC when the 
system locks up? 
In ''zpool list'' output, you can see that only 9.53M on the
zpool is in
use, with a dedup rate of 34466.62x. So what''s there to cache?

I realize that this test does not reflect a situation that is likely to
occur in a production environment, but I''d still be happy about any 
helpful comments.

-- Clemens
-- 
This message posted from opensolaris.org