freebsd stable - Nov 2012 - ZFS memory management

Hello list,

I have the following question : I have several machines with 196G of RAM that
are using
RELENG_9 with ZFS, and are running a very memory intensive java applications -
ElasticSearch
The machines are without swap configured and have "vm.swap_enabled=0"
in /etc/sysctl.conf.
The ElasticSearch processes are using mlockall(2) to pin down their memory
(configured at 40G).
And at this point I thought that there would be no problems, but from time to
time, when the machine grows it's
ARC memory and there are some other running processes like nginx with passenger
and uwsgi the ElasticSearch
process would get killed by the kernel OOM killer with reason "no swap
space available"

Of course, I've now tuned down arc_max in /boot/loader.conf, but isn't
this supposed to work automatically? Like
ZFS releasing some memory when there is a pressure, instead of the OOM killer
going postal? (at the moment when
the process was killed the ZFS ARC was 132G).

I understand that this might be problematic as AFAIK ZFS releases memory
asynchronously when the arc_reclaim_thread() is run,
which might take some time to be scheduled and complete.

Cheers,
Nikolay

Read any ZFS tuning manual on the web, including the ones direct from
SUN/Oracle, and they all list:
  - if you are running processes that need a lot of memory, then limit the
ARC to allow the apps to have access to that memory

:)


On Tue, Nov 27, 2012 at 12:10 PM, Nikolay Denev <ndenev at gmail.com>
wrote:
> Hello list,
>
> I have the following question : I have several machines with 196G of RAM
> that are using
> RELENG_9 with ZFS, and are running a very memory intensive java
> applications - ElasticSearch
> The machines are without swap configured and have
"vm.swap_enabled=0" in
> /etc/sysctl.conf.
> The ElasticSearch processes are using mlockall(2) to pin down their memory
> (configured at 40G).
> And at this point I thought that there would be no problems, but from time
> to time, when the machine grows it's
> ARC memory and there are some other running processes like nginx with
> passenger and uwsgi the ElasticSearch
> process would get killed by the kernel OOM killer with reason "no swap
> space available"
>
> Of course, I've now tuned down arc_max in /boot/loader.conf, but
isn't
> this supposed to work automatically? Like
> ZFS releasing some memory when there is a pressure, instead of the OOM
> killer going postal? (at the moment when
> the process was killed the ZFS ARC was 132G).
>
> I understand that this might be problematic as AFAIK ZFS releases memory
> asynchronously when the arc_reclaim_thread() is run,
> which might take some time to be scheduled and complete.
>
> Cheers,
> Nikolay
>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
>


-- 
Freddie Cash
fjwcash at gmail.com