freebsd stable - Oct 2014 - Suspected kernel memory leak with mmap/sha1 ?

Hi,

We have been trying to track down an odd problem with what appears
to be a kernel memory leak, but are at a complete loss of what is
going on.  We have a cut down test program that reproduces the problem.

 FreeBSD shed62.akips.com 10.1-RC3 FreeBSD 10.1-RC3 #0 r273491:
 Thu Oct 23 05:08:22 UTC 2014
 root at shed31.akips.com:/usr/obj/usr/src/sys/GENERIC  amd64


Basically our app mmap's large data files (eg. 250M) which contain blocks
of compressed data.  Our uncompress runs the data through SHA1 before
uncompressing it.  What we see is active memory being consumed, but can't
figure out what is consuming it as our process size stays constant.
When our process exits, active memory stays high and is never freed. If
enough memory leaks, then the machine starts swapping and game over for
our application.

We've tried it on both ufs and zfs with the same results.  Interestingly
if you umount the zfs file system the active memory immediately disappears.
Without the SHA1_xxx it runs fine.

It's almost like a denial of service problem.


To show the problem, run top and watch active memory.

Compile the test program below 

 cc -o leaktest -O2 leaktest.c
 find /usr -type f | xargs ./leaktest
 
For some reason, active memory skyrockets, especially when it encounters
large files.  Running the program does not reproduce the problem 100% of
the time, but close to it.

Any suggestions ??


-----------------------------------------------

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sha.h>

#define BUFSIZE (1 * 1024 * 1024)

#define TMIN(a,b) ({            \
      typeof (a) _a = (a);      \
      typeof (b) _b = (b);      \
      _a < _b ? _a : _b;        \
})

int
main (int argc, char **argv)
{
   int     i, j,
           fd = -1,
           open_flags = O_RDONLY,
           prot_flags = PROT_READ,
           mmap_flags = 0;
   char   *filename,
          *data = NULL,
          *buf = NULL,
          *p;
   size_t  len,
           rlen;
   struct stat s;
   u_char   md[20];
   SHA_CTX  SD;


   if ((buf = malloc ((size_t) BUFSIZE)) == NULL) {
      fprintf (stderr, "malloc: %s\n", strerror (errno));
      goto END;
   }

   for (i = 1; i < argc; i++) {
      filename = argv[i];
      
      if (stat (filename, &s) != 0)
         fprintf (stderr, "stat: %s %s\n", filename, strerror
(errno));

      else if ((fd = open (filename, open_flags)) == -1)
         fprintf (stderr, "open: %s %s\n", filename, strerror
(errno));

      else if ((data = mmap (NULL, (size_t) s.st_size, prot_flags, 
                mmap_flags, fd, (off_t) 0)) == MAP_FAILED) {
         fprintf (stderr, "mmap: %s %s\n", filename, strerror
(errno));
         close (fd);
         fd = -1;
      }

      else {
         printf ("%s: %zd bytes\n", filename, s.st_size);

         p = data;
         len = s.st_size;

         while (len > 0) {
            rlen = TMIN (BUFSIZE, len);

            /* Copy BUFSIZE lumps into buf and modify it so the compiler 
             * doesn't optimise it out
             */
            memcpy (buf, p, rlen);
            for (j = 0; j < BUFSIZE; j += 1024)
               buf[j]++;

            SHA1_Init (&SD);
            SHA1_Update (&SD, p, rlen);
            SHA1_Final (md, &SD);

            p += rlen;
            len -= rlen;
         }

         if (munmap (data, (size_t) s.st_size) == -1) {
            fprintf (stderr, "munmap: %s %s\n", filename, strerror
(errno));
            goto END;
         }
         if (close (fd) == -1) {
            fprintf (stderr, "close: %s %s\n", filename, strerror
(errno));
            goto END;
         }
      }
   }

END:
   free (buf);

   exit (0);
}


-- 
Paul Koch | Founder, CEO
AKIPS Network Monitor
http://www.akips.com
Brisbane, Australia

This is normal behaviour.  Active (and inactive) memory comprises both
application memory and system disk cache.  By reading in all of those
files, you have loaded the contents of those files into the cache,
which is counted as active memory.  The kernel has no preference
between disk cache or application memory.  Its decisions as to whether
to swap application data out to disk or discard a page of disk cache
are mainly based on the VM system's estimates of how often that data
is being read.

One test that you could perform is to run your app with swap disabled.
If your application runs faster with swap disabled, that indicates
that the VM subsystem's heuristics are suboptimal for your workload
and some tuning might be necessary.  If your application runs at the
same speed or slower, then the VM subsystem is making correct (if
counter-intuitive) decisions performance-wise.  In that case, if your
performance is still unacceptable then your options are either to tune
your app's algorithm to reduce its working set (including application
and file data) to fit into memory, or install more memory in the
machine.