> On 30 July 2015, at 01:39, Doug Hardie <bc979 at lafn.org> wrote:
>
>
>> On 29 July 2015, at 23:44, Peter Jeremy <peter at rulingia.com>
wrote:
>>
>> [reformatted]
>>
>> On 2015-Jul-29 17:41:33 -0700, Doug Hardie <bc979 at lafn.org>
wrote:
>>> I have several FreeBSD 9.3 systems that are using swap and I can?t
>>> figure out what is doing it. The key system has 6GB swap and
>>> currently it has over 2GB in use.
>>
>> Is the system currently paging (top(1) and "systat -v" will
show
>> this)? If not, this just means that at some time in the past, the
>> system was under memory pressure and paged some process memory out.
>> Since then, that memory hasn't been touched so the system
hasn't paged
>> it in.
>>
>>> ps shows only a kernel module
>>> [intr] with a W status.
>>
>> 'W' means the whole process is 'swapped' out - this
will only occur
>> under severe RAM pressure. Normally, the system will just page out
>> inactive parts of a processes address space - and none of the ps flags
>> will show this.
>>
>>> How do I figure out what that swap space is being used for?
>>
>> I don't think this can be trivially done. "procstat -v"
will show
>> the number of resident pages within each swap-backed region, any
>> pages in that region that have been touched but are not resident
>> are on the swap device but any pages that have never been touched
>> aren't counted at all.
>
> Bingo. procstat shows the problem. The process that I suspected has a
large number of entries like:
>
> 650 0x834c00000 0x835800000 rw- 0 0 1 0 ---- sw
> 650 0x835800000 0x835c00000 rw- 0 0 1 0 ---- sw
> 650 0x835c00000 0x837c00000 rw- 1 0 1 0 ---- sw
>
> I don?t know whats in those areas yet. If I were to kill the process with
SIGABRT would the core dump show those areas? I might be able to figure out
what they are from that.
>
> Thanks for the pointer.
I believe I have found a bug in FreeBSD 9.3 mmap. The following test program
(named test.c) does an mmap to itself and monitors the memory objects created
with procstat. After the mmap, munamp is called and the objects are checked
again. The objects created from the mmap do not get purged, but remain.
Eventually, after enough have been created, they start becoming type sw and
eventually the system runs out of swap.
The output:
zool# ./test
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 C--- df
31459 0x28186000 0x281ad000 rw- 13 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
The program:
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid = getpid ();
sprintf (cmd, "procstat -v %d | grep df", pid);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
fd = open ("./test.c", O_RDWR);
cp = mmap (0, 100, PROT_READ|PROT_WRITE, MAP_FILE, fd, 0);
if (cp == MAP_FAILED)
{
printf ("mmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
rc = munmap (cp, 100);
if (rc == -1)
{
printf ("munmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
close (fd);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
}