On Mon, Mar 16, 2015 at 06:59:33PM -0400, J David wrote:> Recently we have seen a large-scale memory leak on amd64 machines
> running FreeBSD 9.3-RELEASE-p10.
>
> This was first observed on 9.3p2 but has since shown up all the way through
p10.
>
> Here's what the header of top shows:
>
> last pid: 32329; load averages: 0.00, 0.01, 0.21 up 3+15:37:29
22:34:04
> 25 processes: 2 running, 22 sleeping, 1 waiting
> CPU: % user, % nice, % system, % interrupt, % idle
> Mem: 4072M Active, 895M Inact, 1284M Wired, 125M Cache, 826M Buf, 1521M
Free
> Swap: 1024M Total, 874M Used, 149M Free, 85% Inuse
>
> About 4G actively being used, another 895M inactive, and another 874M
> in swap. So it seems like this is a user-space leak, rather than a
> kernel-space leak.
>
> At the time of measurement, this machine was not doing anything and
> every possible process had been killed trying to find a culprit. The
> entire output of "ps axlww" is:
>
> UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
> 0 0 0 0 -52 0 0 224 - DLs ?? 0:00.82 [kernel]
> 0 1 0 0 20 0 6280 556 wait SLs ?? 0:00.57
/sbin/init --
> 0 2 0 0 -16 0 0 16 pftm DL ?? 0:00.85
[pfpurge]
> 0 3 0 0 -16 0 0 16 waiting_ DL ?? 0:00.00
> [sctp_iterator]
> 0 4 0 0 -16 0 0 16 - DL ?? 0:00.00
[xpt_thrd]
> 0 5 0 0 -16 0 0 16 psleep DL ?? 0:28.85
[pagedaemon]
> 0 6 0 0 -16 0 0 16 psleep DL ?? 0:45.03
[vmdaemon]
> 0 7 0 0 -16 0 0 16 pollid DL ?? 0:00.23
[idlepoll]
> 0 8 0 0 155 0 0 16 pgzero DL ?? 0:00.00
[pagezero]
> 0 9 0 0 -16 0 0 16 psleep DL ?? 0:00.83
[bufdaemon]
> 0 10 0 0 -16 0 0 16 audit_wo DL ?? 0:00.00 [audit]
> 0 11 0 0 155 0 0 32 - RL ?? 8317:13.37 [idle]
> 0 12 0 0 -76 0 0 240 - WL ?? 301:43.54 [intr]
> 0 13 0 0 -8 0 0 48 - DL ?? 0:09.89 [geom]
> 0 14 0 0 -16 0 0 16 - DL ?? 2:58.88 [yarrow]
> 0 15 0 0 -68 0 0 64 - DL ?? 0:02.32 [usb]
> 0 16 0 0 -16 0 0 16 vlruwt DL ?? 0:06.35 [vnlru]
> 0 17 0 0 16 0 0 16 syncer DL ?? 5:28.89 [syncer]
> 0 18 0 0 -16 0 0 16 sdflush DL ?? 0:10.27
> [softdepflush]
> 0 19 0 0 -16 0 0 16 - DL ?? 0:55.09 [racctd]
> 0 830 1 0 20 0 45348 2396 wait Is u0 0:00.07
> login [pam] (login)
> 500 32269 830 0 20 0 14556 2428 wait S u0 0:00.09 -sh (sh)
> 500 32340 32269 0 20 0 16296 1908 - R+ u0 0:00.00 ps axlww
>
> Since the issue doesn't seem related to kernel memory usage, vmstat -m
> and -z have been skipped, but nothing jumps out as using gigs of RAM;
> they do appear consistent with 1284M of wired memory, which is not
> unreasonable for the affected machines' tuning and workload.
>
> The only user-space processes running are login, sh, and ps. So where
> did 5.5G of userspace RAM go?
>
> The only other potentially useful information is that when this
> happens, shutting down the system will hang for about ten minutes.
>
> $ sudo halt -p
> Waiting (max 60 seconds) for system process `vnlru' to stop...done
> Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
> Waiting (max 60 seconds) for system process `syncer' to stop...
> Syncing disks, vnodes remaining...0 0 0 0 0 0 0 0 0 done
> All buffers synced. <----- 10 MINUTE HANG AFTER PRINTING THIS
> Uptime: 3d15h56m32s
> usbus0: Controller shutdown
> uhub0: at usbus0, port 1, addr 1 (disconnected)
> usbus0: controller did not stop
> usbus0: Controller shutdown complete
> acpi0: Powering system off
> Connection closed by foreign host.
>
> So it seems like somewhere after "All buffers synced" and
printing the
> uptime, it's very slowly unwinding whatever is using up all that RAM
> and swap.
>
> Does anyone have any idea what might be causing this or how to fix/prevent
it?
There are a lot of possibilities to create persistent anonymous shared
memory objects. Not complete list is tmpfs mounts, swap-backed md disks,
sysv shared memory, possibly posix shared memory (I do not remember which
implementation is used in stable/9).
I quite possible missed some object types. Also note that active/inactive
can be explained by cached file pages, and only swap usage suggests that
it might be something persisent from the list above.