freebsd stable - Aug 2014 - vmdaemon CPU usage and poor performance in 10.0-RELEASE

All,

We have a handful of database servers running FreeBSD 10.0-RELEASE and
PostgreSQL 9.3.4.  The servers have 128GB or 256GB of RAM.  After a short period
of running and some "light" load (load average of 6 on a 16-core w/ HT
box), the system performance becomes terrible.  Logging in and simply attempting
to run some commands results in the command prompt hanging for several seconds
before anything is returned (even for something simple like `test`).  If we run
`top -SH` when the system performance is bad, we can see the vmdaemon
thread/kernel process is using 100% CPU.  Eventually the vmdaemon CPU usage
drops and system performance seems to return to normal, but this doesn't
last long.

Has anyone seen this behavior?  I found a bug that seems to describe the same
problem (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190300), but it's
been open for 2 months without any responses.  For what it's worth, the
problem does not seem as bad when we reduce postgresql shared buffers from 128GB
to 8GB, but it's still there.

I can't imagine what vmdaemon is spending all of its time doing - `top -SH`
output and memory usage looks like:

last pid: 97787;  load averages:  6.03,  6.17,  6.03                            
up 19+15:46:19  08:50:07
345 processes: 41 running, 220 sleeping, 78 waiting, 6 lock
CPU: 16.0% user,  0.0% nice, 10.0% system,  0.0% interrupt, 74.0% idle
Mem: 6314M Active, 234G Inact, 4078M Wired, 2720M Cache, 1704M Buf, 2719M Free
Swap: 16G Total, 329M Used, 16G Free, 2% Inuse

  PID USERNAME      PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root          155 ki31     0K   512K CPU26  26 448.8H  95.07% idle{idle:
cpu26}
   11 root          155 ki31     0K   512K CPU29  29 449.1H  92.48% idle{idle:
cpu29}
    6 root          -16    -     0K    16K CPU27  27  81.5H  89.70% vmdaemon
   11 root          155 ki31     0K   512K RUN     1 459.2H  89.60% idle{idle:
cpu1}
   11 root          155 ki31     0K   512K RUN    14 452.9H  89.26% idle{idle:
cpu14}
   11 root          155 ki31     0K   512K CPU5    5 458.7H  89.16% idle{idle:
cpu5}
   11 root          155 ki31     0K   512K CPU31  31 449.1H  88.96% idle{idle:
cpu31}
   11 root          155 ki31     0K   512K RUN     6 458.2H  88.87% idle{idle:
cpu6}
   11 root          155 ki31     0K   512K CPU11  11 453.0H  88.38% idle{idle:
cpu11}
   11 root          155 ki31     0K   512K RUN     3 457.9H  88.28% idle{idle:
cpu3}
   11 root          155 ki31     0K   512K CPU12  12 452.8H  87.89% idle{idle:
cpu12}
   11 root          155 ki31     0K   512K CPU2    2 458.6H  87.26% idle{idle:
cpu2}
   11 root          155 ki31     0K   512K CPU7    7 458.6H  87.16% idle{idle:
cpu7}
   11 root          155 ki31     0K   512K CPU10  10 458.5H  86.87% idle{idle:
cpu10}
   11 root          155 ki31     0K   512K CPU15  15 453.8H  86.47% idle{idle:
cpu15}
   11 root          155 ki31     0K   512K CPU13  13 453.6H  85.79% idle{idle:
cpu13}
   11 root          155 ki31     0K   512K CPU4    4 458.4H  85.69% idle{idle:
cpu4}
   11 root          155 ki31     0K   512K RUN     9 453.4H  85.35% idle{idle:
cpu9}
   11 root          155 ki31     0K   512K CPU0    0 460.5H  85.25% idle{idle:
cpu0}
   11 root          155 ki31     0K   512K CPU17  17 449.3H  84.08% idle{idle:
cpu17}
   11 root          155 ki31     0K   512K RUN    25 449.0H  83.40% idle{idle:
cpu25}
   11 root          155 ki31     0K   512K CPU18  18 448.9H  82.76% idle{idle:
cpu18}
   11 root          155 ki31     0K   512K CPU16  16 448.8H  82.18% idle{idle:
cpu16}
   11 root          155 ki31     0K   512K CPU23  23 448.8H  81.88% idle{idle:
cpu23}
   11 root          155 ki31     0K   512K CPU8    8 454.3H  81.30% idle{idle:
cpu8}
   11 root          155 ki31     0K   512K RUN    21 449.1H  81.05% idle{idle:
cpu21}
   11 root          155 ki31     0K   512K CPU20  20 448.6H  80.96% idle{idle:
cpu20}
   11 root          155 ki31     0K   512K RUN    22 448.7H  80.66% idle{idle:
cpu22}
   11 root          155 ki31     0K   512K RUN    30 448.4H  79.59% idle{idle:
cpu30}
   11 root          155 ki31     0K   512K CPU24  24 448.7H  79.39% idle{idle:
cpu24}
   11 root          155 ki31     0K   512K RUN    19 449.2H  78.47% idle{idle:
cpu19}
   11 root          155 ki31     0K   512K CPU28  28 448.7H  54.20% idle{idle:
cpu28}

$ uname -a
FreeBSD db00 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #2: Thu Jul 24 14:35:07 EDT
2014     root at build10:/usr/obj/usr/src/sys/GENERIC  amd64

Any help would be appreciated.  We would be more than happy to answer any other
questions.

Thanks,

Steve Polyack

Hi!
> We have a handful of database servers running FreeBSD 10.0-RELEASE
> and PostgreSQL 9.3.4.  The servers have 128GB or 256GB of RAM.
Are you aware of the recent work on that topic ?

https://www.freebsd.org/news/status/report-2014-04-2014-06.html#PostgreSQL-Performance-Improvements

Maybe kib@ knows more about this ?

-- 
pi at opsec.eu            +49 171 3101372                         6 years to go
!