On Wed, May 11, 2011 at 12:58:50PM +0200, Robert Schulze
wrote:> We are running 8-STABLE (csupped at 20110504) on a NFS fileserver.
> It has 32 GB RAM and uses ZFS:
>
> home ONLINE 0 0 0
> raidz2 ONLINE 0 0 0
> da1 ONLINE 0 0 0
> da2 ONLINE 0 0 0
> da3 ONLINE 0 0 0
> da4 ONLINE 0 0 0
> da5 ONLINE 0 0 0
> raidz2 ONLINE 0 0 0
> da6 ONLINE 0 0 0
> da7 ONLINE 0 0 0
> da8 ONLINE 0 0 0
> da9 ONLINE 0 0 0
> da10 ONLINE 0 0 0
> logs
> mirror ONLINE 0 0 0
> da12 ONLINE 0 0 0
> da13 ONLINE 0 0 0
> cache
> ad4 ONLINE 0 0 0
> ad8 ONLINE 0 0 0
>
> Before upgrading from 8.0, the machine never used the whole system
> memory, it left about 10 GB free even after about 100 days uptime.
> Now, it eats RAM insanely (wired is between 29 GB and 30 GB), which
> is quite good I think, but after about 3 days uptime, we now have
> 106 MB swapped out. Both L2ARC SSDs are ~74 GB in size, arc_summary
> prints the following values:
>
> ARC Size:
> Current Size: 76.21% 23440.22M (arcsize)
> Target Size: (Adaptive) 76.52% 23535.40M (c)
> Min Size (Hard Limit): 12.50% 3844.77M (c_min)
> Max Size (High Water): ~8:1 30758.16M (c_max)
>
> L2 ARC Size:
> Current Size: (Adaptive) 88466.19M
> Header Size: 0.29% 259.21M
>
>
> The following sysctls were set:
>
> security.bsd.see_other_uids=0
> kern.maxvnodes=400000
> kern.ipc.somaxconn=8192
> kern.ipc.maxsockbuf=1024000
> net.inet.udp.maxdgram=57344
> vfs.ufs.dirhash_maxmem=25165824
>
> My question now: why does the machine swap, is this normal behaviour?
> Why is wired at about 30 GB if ARC=23 GB and L2ARC-header=259 MB?
I'm not really all that familiar with L2ARC at this point (conceptually
yes, real-world use no), but the delta (23GB vs. 30GB wired) is probably
explainable. The most common reason, as I understand it, is that memory
becomes fragmented in such a way that there are unoptimised/non-optimal
page layouts in memory resulting in a waste.
I tend not to use the "arcstats" script and instead look at the sysctl
data from "sysctl -a kstat.zfs.misc.arcstats". I guess I'm used
to
looking at it by now.
Anyway, for example, on my 8GB machine with vfs.zfs.arc_max="6144M"
set
in /boot/loader.conf, running 8.2-STABLE dated May 6th, "Wired" on my
machine has occasionally reached 6.8GBytes. How much of this was ZFS?
About 6.4GBytes -- the remaining 0.4GBytes? mysqld.
What you're looking for is something very low-level that gives a
complete and total kernel-level memory map to show you exactly where
everything is going. I believe "vmstat -z" provides that.
So basically what I'm trying to say here is that you're running top,
looking at Wired, and saying "all of this is ZFS" when that's
definitely
not what Wired represents exclusively.
If you're worried about swap usage, try limiting the ARC size more using
/boot/loader.conf. You do not need to adjust vm.kmem_size or
vm.kmem_size_max if the machine is running 8.2-RELEASE or newer. I only
mention this because all the online docs you'll find mention tuning one
or both of those two; that only applies to older FreeBSD releases.
--
| Jeremy Chadwick jdc@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP 4BD6C0CB |