zfs discuss - Dec 2007 - why are these three ZFS caches using so much kmem?

Got an issue which is rather annoying to me - three of my
ZFS caches are regularly using nearly 1/2 of the 1.09Gb of
allocated kmem in my system

	ultra20 m2
	1x 2.2GHz dual-core Opteron
	2Gb ram, 16Gb swap
	2 zpools, not ZFS root
	snv_77
	2 non-global zones.

This is the SUNWscat "suspect caches" report:

==== reporting kmem stats, just the suspect caches ===
suspect                      buf     buf     buf    memory    #allocations
cache name                  size   avail   total    in use    succeed fail
------------------------- ------ ------- ------- --------- ---------- ----
kmem_bufctl_cache             24  161952 1089984  26574848    7720657    0 *
kmem_alloc_8192             8192      48    5505  45096960    6685148    0
kmem_io_2G_2048             2048       6   12296  25182208      43011    0
rootnex_dmahdl              2504      83   20840  53350400   63862899    0
vn_cache                     264   75854  171780  58634240  152439475    0
anon_cache                    48   25482  487620  23777280   30578955    0
ufs_inode_cache              368   32628   58480  23953408    8207842    0
zio_buf_512                  512  114409  303440 155361280   11614738    0 *
zio_buf_16384              16384       3    7074 115900416    2109727    0 *
zio_buf_65536              65536       1     645  42270720    1218805    0 *
dmu_buf_impl_t               192  196172  392301  76517376   18875286    0
dnode_t                      656  144388  335226 228847616    7245893    0
zfs_znode_cache              200   71534  133420  27324416    8670673    0
------------------------- ------ ------- ------- --------- ---------- ----
total memory                   -       -       - 1080578048 4744404583    0

....
unix:segkmem_alloc_io_2G+0x0               24.0M
unix:segkmem_alloc_io_16M+0x0                  0
unix:bp_vmem_alloc+0x0                      768K
unix:hat_memload_alloc+0x0                 9.80M
unix:segkmem_alloc_permanent+0x0               0
unix:segkmem_alloc+0x0                     1.09G
<<<<<<<<<<<
unix:segkmem_zio_alloc+0x0                 10.0M
genunix:kmem_firewall_va_alloc+0x0         64.0M
genunix:devmap_alloc_pages+0x0              896K
genunix:vmem_alloc+0x0                     1.21G
<<<<<<<<<<<



This all has the effect of my system constantly swapping and
interactive response for my desktop is horrendous.

What can I do to find out why this is happening, and prevent
it recurring?

(If Team ZFS would like access to the core please contact me
directly for access).


thanks in advance,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

> Got an issue which is rather annoying to me - three of my
> ZFS caches are regularly using nearly 1/2 of the 1.09Gb of
> allocated kmem in my system.
I think this is just the ARC; you can limit its size using:

http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache

You can verify this using mdb as also shown at the above ...

Anton
 
 
This message posted from opensolaris.org

Anton B. Rang wrote:
>> Got an issue which is rather annoying to me - three of my
>> ZFS caches are regularly using nearly 1/2 of the 1.09Gb of
>> allocated kmem in my system.
> 
> I think this is just the ARC; you can limit its size using:
> 
>
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache
> 
> You can verify this using mdb as also shown at the above ...
That''s what I was thinking too. I''m going to set a limit
and see how it goes.


thankyou,
James
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

James C. McPherson wrote:
> Got an issue which is rather annoying to me - three of my
> ZFS caches are regularly using nearly 1/2 of the 1.09Gb of
> allocated kmem in my system
...[snip]

Following suggestions from Andre and Rich that this was
probably the ARC, I''ve implemented a 256Mb limit for my
system''s ARC, per the Solaris Internals wiki:

* http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#ARCSIZE
* set arc max to 256Mb
set zfs:zfs_arc_max=0x10000000


And my system now seems to be chugging along quite happily.


thankyou,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

> Following suggestions from Andre and Rich that this was
> probably the ARC, I''ve implemented a 256Mb limit for my
> system''s ARC, per the Solaris Internals wiki:
> 
> *
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#ARCSIZE
> * set arc max to 256Mb
> set zfs:zfs_arc_max=0x10000000
> 
> And my system now seems to be chugging along quite happily.
Is there something very special about your system that''s incompatible 
with ZFS''s default policy? I''m asking because the link above
says that
"ZFS is not designed to steal memory from applications" and yet your 
out-of-the-box experience was:

 > my system constantly swapping and
 > interactive response for my desktop is horrendous.

-Artem

Artem Kachitchkine wrote:
 >James McPherson wrote:
>> Following suggestions from Andre and Rich that this was
>> probably the ARC, I''ve implemented a 256Mb limit for my
>> system''s ARC, per the Solaris Internals wiki:
>>
>> *
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#ARCSIZE
>> * set arc max to 256Mb
>> set zfs:zfs_arc_max=0x10000000
>>
>> And my system now seems to be chugging along quite happily.
> 
> Is there something very special about your system that''s
incompatible
> with ZFS''s default policy? I''m asking because the link
above says that
> "ZFS is not designed to steal memory from applications" and yet
your
> out-of-the-box experience was:
> 
>  > my system constantly swapping and
>  > interactive response for my desktop is horrendous.
Hi Artem,
yeah, my system is a little special :-)

In my global zone I run
	gnome
	firefox
	thunderbird
	XEmacs
	XChat
	pidgin

In my webserver zone I run
	exim
	apache 2.2.3
	tomcat 5.5.17
	JRoller 3.mumble
	PostgreSQL 8.1
(all of which provide www.jmcp.homeunix.com)

In my punchin zone I run
	thunderbird
	firefox
	bugster
	cscope
	XChat



James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog