On 2012-Feb-24 11:06:52 +0000, Luke Marsden
<luke-lists@hybrid-logic.co.uk> wrote:>We're running 8.2-RELEASE v15 in production on 24GB RAM amd64 machines
>but have been having trouble with short spikes in application memory
>usage resulting in huge amounts of swapping, bringing the whole machine
>to its knees and crashing it hard. I suspect this is because when there
>is a sudden spike in memory usage the zfs arc reclaim thread is unable
>to free system memory fast enough.
There were a large number of fairly serious ZFS bugs that have been
fixed since 8.2-RELEASE and I would suggest you look at upgrading.
That said, I haven't seen the specific problem you are reporting.
> * is this a known problem?
I'm unaware of it specifically as it relates to ZFS. You don't mention
how big the memory usage spike is but unless there is sufficient free+
cache available to cope with a usage spike then you will have problems
whether it's UFS or ZFS (though it's possibly worse with ZFS).
FreeBSD is known not to cope well with running out of memory.
> * what is the community's advice for production machines running
> ZFS on FreeBSD, is manually limiting the ARC cache (to ensure
> that there's enough actually free memory to handle a spike in
> application memory usage) the best solution to this
> spike-in-memory-means-crash problem?
Are you swapping onto a ZFS vdev? If so, change back to a raw (or
geom) device - swapping to ZFS is known to be problematic. If you
have very spiky memory requirements, increasing vm.v_cache_min and/or
vm.v_free_reserved might give you better results.
> * has FreeBSD 9.0 / ZFS v28 solved this problem?
The ZFS code is the same in 9.0 and 8.3. Since 8.3 is less of a jump,
I'd recommend that you try 8.3-prerelease in a test box and see how
it handles your load. Note that there's no need to upgrade your pools
from v15 to v28 unless you want the ZFS features - the actual ZFS
code is independent of pool version.
> * rather than setting a hard limit on the ARC cache size, is it
> possible to adjust the auto-tuning variables to leave more free
> memory for spiky memory situations? e.g. set the auto-tuning to
> make arc eat 80% of memory instead of ~95% like it is at
> present?
Memory spikes are absorbed by vm.v_cache_min and vm.v_free_reserved in
the first instance. The current vfs.zfs.arc_max default may be a bit
high for some workloads but at this point in time, you will need to
tune it manually.
--
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120227/9012f15e/attachment.pgp