On Thu, Mar 26, 2015 at 12:46 PM, J David <j.david.lists at gmail.com> wrote:> On Mon, Mar 16, 2015 at 7:52 PM, J David <j.david.lists at gmail.com> wrote: > > On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > > <kostikbel at gmail.com> wrote: > >> There are a lot of possibilities to create persistent anonymous shared > >> memory objects. Not complete list is tmpfs mounts, swap-backed md > disks, > >> sysv shared memory, possibly posix shared memory (I do not remember > which > >> implementation is used in stable/9). > > > > If that's the explanation, how could it be > > detected/measured/investigated/resolved/prevented? > > > > Under ordinary circumstances, machines will go run like this for > days/weeks: > > > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M > Free > > Swap: 1024M Total, 1024M Free > > > > Then, when this happens, it rapidly degrades from that to so bad that > > processes start getting killed for being out of swap space. > > These FreeBSD machines running out of swap space and dying continues > to be a daily problem causing outages and unscheduled reboots. Is > there really no way to even research what might be causing the > problem? > > (Widening the cross-posting in the hopes of eliciting more help, so > the brief summary of the problem orginally posted to freebsd-stable is > that an unknown actor consumes all the user-space memory in the > system, including swap space, to the point where processes are killed > for being out of swap space, but if every process on the machine is > stopped, very little of the user-space memory in use is freed. > Original message with more details is here: > https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html > .) > > There are no tmpfs mounts or md disks, so it would have to be one of > the other causes. How can FreeBSD's use of persistent, anonymous > shared memory objects be investigated, measured, or controlled so we > can get a handle on this issue? >This is just a shot in the dark and not a really likely one, but I have had issues with Firefox leaking memory badly. I can free the space by killing firefox and restarting it. It seems to be linked to certain web sites, probably javascript. I have not been able to confirm which one does it. It just will start growing until the system slows to a crawl as too many things are swapped out. Normally my system does not touch swap. If it is in user space, top should show it under RES. -- Kevin Oberman, Network Engineer, Retired E-mail: rkoberman at gmail.com
In our case, On Thu, Mar 26, 2015 at 5:03 PM, Kevin Oberman <rkoberman at gmail.com> wrote:> This is just a shot in the dark and not a really likely one, but I have had > issues with Firefox leaking memory badly. I can free the space by killing > firefox and restarting it.In our case, we can log in from the console, kill every single user-mode process on the system except the init, login, and the console shell, and the memory is not recovered. Gigabytes and gigabytes user memory of it are being held by some un-findable anonymous persistent structure not linked to any process. Konstantin proposed that it was some sort of shared memory usage, but there appears to be no way to check or investigate most types of shared memory usage on FreeBSD.> If it is in user space, top should show it under RES.This is definitely *not* the case. Whatever is using the memory is not associated with any user-space process, and does not show up on top or ps. It also does not appear to be SysV shared memory, as that reports: $ ipcs -m Shared Memory: T ID KEY MODE OWNER GROUP $ Also, kern.ipc.shmmax is only 512MB whereas this problem is consuming usually 8-10GB. So I guess the remaining possibilities are anonymous mmap's that are somehow not associated with any process and Posix shared memory. Are there any ways to investigate either possibility? Thanks!
On Thu, 26 Mar 2015 14:03:45 -0700 Kevin Oberman <rkoberman at gmail.com> wrote> On Thu, Mar 26, 2015 at 12:46 PM, J David <j.david.lists at gmail.com> wrote: > > > On Mon, Mar 16, 2015 at 7:52 PM, J David <j.david.lists at gmail.com> wrote: > > > On Mon, Mar 16, 2015 at 7:24 PM, Konstantin Belousov > > > <kostikbel at gmail.com> wrote: > > >> There are a lot of possibilities to create persistent anonymous shared > > >> memory objects. Not complete list is tmpfs mounts, swap-backed md > > disks, > > >> sysv shared memory, possibly posix shared memory (I do not remember > > which > > >> implementation is used in stable/9). > > > > > > If that's the explanation, how could it be > > > detected/measured/investigated/resolved/prevented? > > > > > > Under ordinary circumstances, machines will go run like this for > > days/weeks: > > > > > > Mem: 549M Active, 3623M Inact, 567M Wired, 3484K Cache, 827M Buf, 3156M > > Free > > > Swap: 1024M Total, 1024M Free > > > > > > Then, when this happens, it rapidly degrades from that to so bad that > > > processes start getting killed for being out of swap space. > > > > These FreeBSD machines running out of swap space and dying continues > > to be a daily problem causing outages and unscheduled reboots. Is > > there really no way to even research what might be causing the > > problem? > > > > (Widening the cross-posting in the hopes of eliciting more help, so > > the brief summary of the problem orginally posted to freebsd-stable is > > that an unknown actor consumes all the user-space memory in the > > system, including swap space, to the point where processes are killed > > for being out of swap space, but if every process on the machine is > > stopped, very little of the user-space memory in use is freed. > > Original message with more details is here: > > https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081986.html > > .) > > > > There are no tmpfs mounts or md disks, so it would have to be one of > > the other causes. How can FreeBSD's use of persistent, anonymous > > shared memory objects be investigated, measured, or controlled so we > > can get a handle on this issue? > > > > This is just a shot in the dark and not a really likely one, but I have had > issues with Firefox leaking memory badly. I can free the space by killing > firefox and restarting it. > > It seems to be linked to certain web sites, probably javascript. I have not > been able to confirm which one does it. It just will start growing until > the system slows to a crawl as too many things are swapped out. Normally my > system does not touch swap.I can confirm this -- both regular, as well as ESR. Upgrading firefox [ultimately] has little-to-no effect. I have experienced this for near 2yrs. I suspect the [firefoxes] js engine. Any one of any number of sites could/would/will cause it. As Kevin already noted; stopping firefox, and starting it again, seems the only solution.> > If it is in user space, top should show it under RES. > -- > Kevin Oberman, Network Engineer, Retired > E-mail: rkoberman at gmail.com > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"--Chris