freebsd stable - Apr 2007 - question: +swap_pager_getswapspace(16): failed

Hello,

I have a little understanding problem:
My box has 128MB memory, far enough for the task.
After a few days I always see some processes dying because:

+swap_pager_getswapspace(2): failed
+pid 48211 (perl5.8.8), uid 58, was killed: out of swap space

Why won't for example the 21MB Buf get freed before more swap space gets 
requested than available (swap is very low, it's FlashDisk!)?
Is there a way to find out what process is swapped?

Thanks for any hints. My only way to circumvent this problem is to reboot the 
machine daily.

-Harry

On 2007-Apr-24 09:32:06 +0200, Harald Schmalzbauer <harry@schmalzbauer.de>
wrote:
>My box has 128MB memory, far enough for the task.
If you are regularly running out of space, then maybe not - at least
without tuning some parameters.

How much swap space do you actually have and what is your box trying
to do?  My firewall also has 128MB and it's only paged out 575 pages
in the last 9.7 days.
>After a few days I always see some processes dying because:
>
>+swap_pager_getswapspace(2): failed
>+pid 48211 (perl5.8.8), uid 58, was killed: out of swap space
>
>Why won't for example the 21MB Buf get freed before more swap space gets
>requested than available (swap is very low, it's FlashDisk!)?
vfs.bufspace is inside a feedback loop that tries to keep it between
vfs.lobufspace and vfs.hibufspace - which are tuned based on memory
size by default (for 128MB RAM, hibufspace should be ~22MB).  You
could try seting kern.nbuf (in /boot/loader.conf) to reduce the buffer
space allocated (each buffer is 16KB).
>Is there a way to find out what process is swapped?
The ps output will include 'W'.  Note that 'swapped' is a
special
state and normally processes are just paged.

In top and ps, the difference between 'size' and 'res' reflects
memory
space that the process has allocated to it but is not resident.
Unfortunately, this includes both text area (which is vnode backed)
and space that has never been touched (and therefore doesn't exist
anywhere) as well as swap space.

Offhand, I don't know of any tool to report the swap utilisation by
process on FreeBSD.  (Though I have written such a tool for Tru64).

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20070425/eba30639/attachment.pgp

Harald Schmalzbauer wrote:
 > I have a little understanding problem:
 > My box has 128MB memory, far enough for the task.

Are you sure?  I see you're running perl scripts.  Those
can easily (and sometimes unexpectedly) eat a lot of
memory.

 > After a few days I always see some processes dying because:
 > 
 > +swap_pager_getswapspace(2): failed
 > +pid 48211 (perl5.8.8), uid 58, was killed: out of swap space

AFAIK, the kernel selects the largest process for killing
when it runs out of swap.  So your perl was probably quite
large.

 > Why won't for example the 21MB Buf get freed before more swap space
gets
 > requested than available (swap is very low, it's FlashDisk!)?
 > Is there a way to find out what process is swapped?

There's a difference between swapped and paged.  Swapped
means that a process has been _completely_ moved to swap
space.  Such processes had a "W" flag in the state column
of ps(1).  In contrast to that, paging happens on pages,
not on processes.  Note that pages are not uniquely
assigned to one process, but they can be shared between
processes, so it's difficult (or even impossible) to give
clear numbers on the amount of paging per process.

 > Thanks for any hints. My only way to circumvent this problem is to reboot
the
 > machine daily.

I've also set up machines with limited RAM and without
hard disk for swapping.  I never swap onto flash memory,
because flash has a limited number of write cycles (it
suffers from certain wear).  If you have another server
within short network distance, you should consider
remote swapping via NFS.  It's not as slow as it sounds
(depending on the speed of the network and the server),
and at least it's better than nothing.

Otherwise, don't swap (or page) at all.  The folowing
sysctl will optimize your system for that case:

vm.swap_enabled=0
vm.swap_idle_enabled=0
vm.defer_swapspace_pageouts=1
vm.disable_swapspace_pageouts=1
kern.ipc.shm_use_phys=1

You should also build a custom kernel that contains
"options NO_SWAPPING".  Also make sure that you remove
everything from your kernel that you don't need, so more
memory is availale for userland.  In particular, remove
"options UFS_DIRHASH", because the dirhash code can eat
quite a lot of memory.

And of course you should not run any processes that you
don't really need.  But I guess you already know that.

It might also help to set the "R" malloc flag:
# ln -s R /etc/malloc.conf
It can reduce memory fragmentation for processes that
do a lot of small (re)allocations.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch?ftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M?n-
chen, HRB 125758,  Gesch?ftsf?hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"C++ is to C as Lung Cancer is to Lung."
        -- Thomas Funke