On Mon, Mar 16, 2015 at 7:52 PM, J David <j.david.lists at gmail.com> wrote:> On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > <kostikbel at gmail.com> wrote: >> There are a lot of possibilities to create persistent anonymous shared >> memory objects. Not complete list is tmpfs mounts, swap-backed md disks, >> sysv shared memory, possibly posix shared memory (I do not remember which >> implementation is used in stable/9). > > If that's the explanation, how could it be > detected/measured/investigated/resolved/prevented? > > Under ordinary circumstances, machines will go run like this for days/weeks: > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M Free > Swap: 1024M Total, 1024M Free > > Then, when this happens, it rapidly degrades from that to so bad that > processes start getting killed for being out of swap space.These FreeBSD machines running out of swap space and dying continues to be a daily problem causing outages and unscheduled reboots. Is there really no way to even research what might be causing the problem? (Widening the cross-posting in the hopes of eliciting more help, so the brief summary of the problem orginally posted to freebsd-stable is that an unknown actor consumes all the user-space memory in the system, including swap space, to the point where processes are killed for being out of swap space, but if every process on the machine is stopped, very little of the user-space memory in use is freed. Original message with more details is here: https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html .) There are no tmpfs mounts or md disks, so it would have to be one of the other causes. How can FreeBSD's use of persistent, anonymous shared memory objects be investigated, measured, or controlled so we can get a handle on this issue? Thanks!
On Thu, Mar 26, 2015 at 12:46 PM, J David <j.david.lists at gmail.com> wrote:> On Mon, Mar 16, 2015 at 7:52 PM, J David <j.david.lists at gmail.com> wrote: > > On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > > <kostikbel at gmail.com> wrote: > >> There are a lot of possibilities to create persistent anonymous shared > >> memory objects. Not complete list is tmpfs mounts, swap-backed md > disks, > >> sysv shared memory, possibly posix shared memory (I do not remember > which > >> implementation is used in stable/9). > > > > If that's the explanation, how could it be > > detected/measured/investigated/resolved/prevented? > > > > Under ordinary circumstances, machines will go run like this for > days/weeks: > > > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M > Free > > Swap: 1024M Total, 1024M Free > > > > Then, when this happens, it rapidly degrades from that to so bad that > > processes start getting killed for being out of swap space. > > These FreeBSD machines running out of swap space and dying continues > to be a daily problem causing outages and unscheduled reboots. Is > there really no way to even research what might be causing the > problem? > > (Widening the cross-posting in the hopes of eliciting more help, so > the brief summary of the problem orginally posted to freebsd-stable is > that an unknown actor consumes all the user-space memory in the > system, including swap space, to the point where processes are killed > for being out of swap space, but if every process on the machine is > stopped, very little of the user-space memory in use is freed. > Original message with more details is here: > https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html > .) > > There are no tmpfs mounts or md disks, so it would have to be one of > the other causes. How can FreeBSD's use of persistent, anonymous > shared memory objects be investigated, measured, or controlled so we > can get a handle on this issue? >This is just a shot in the dark and not a really likely one, but I have had issues with Firefox leaking memory badly. I can free the space by killing firefox and restarting it. It seems to be linked to certain web sites, probably javascript. I have not been able to confirm which one does it. It just will start growing until the system slows to a crawl as too many things are swapped out. Normally my system does not touch swap. If it is in user space, top should show it under RES. -- Kevin Oberman, Network Engineer, Retired E-mail: rkoberman at gmail.com
On Thu, Mar 26, 2015 at 03:46:05PM -0400, J David wrote:> On Mon, Mar 16, 2015 at 7:52 PM, J David <j.david.lists at gmail.com> wrote: > > On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > > <kostikbel at gmail.com> wrote: > >> There are a lot of possibilities to create persistent anonymous shared > >> memory objects. Not complete list is tmpfs mounts, swap-backed md disks, > >> sysv shared memory, possibly posix shared memory (I do not remember which > >> implementation is used in stable/9). > > > > If that's the explanation, how could it be > > detected/measured/investigated/resolved/prevented? > > > > Under ordinary circumstances, machines will go run like this for days/weeks: > > > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M Free > > Swap: 1024M Total, 1024M Free > > > > Then, when this happens, it rapidly degrades from that to so bad that > > processes start getting killed for being out of swap space. > > These FreeBSD machines running out of swap space and dying continues > to be a daily problem causing outages and unscheduled reboots. Is > there really no way to even research what might be causing the > problem? > > (Widening the cross-posting in the hopes of eliciting more help, so > the brief summary of the problem orginally posted to freebsd-stable is > that an unknown actor consumes all the user-space memory in the > system, including swap space, to the point where processes are killed > for being out of swap space, but if every process on the machine is > stopped, very little of the user-space memory in use is freed. > Original message with more details is here: > https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html > .) > > There are no tmpfs mounts or md disks, so it would have to be one of > the other causes. How can FreeBSD's use of persistent, anonymous > shared memory objects be investigated, measured, or controlled so we > can get a handle on this issue?Start by providing useful information about your system, not a description of the information. E.g., a consistent snapshot of the following: ps auxww swapinfo mount -v mdconfig -lv vmstat -z vmstat -m vmstat -s sysctl -a ipcs -a Collect this data both during the normal run, run while the problem appear but userspace is not killed, and after you killed the processes. Just in case, show kldstat.