Hello list,
context is (server)
freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM,
E5-2630 @2.3GHz, generic kernel.
There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also
freebsd-11-stable)
There have been no special options for zfs configuration on the server,
apart from several datasets having the compressed property set (lz4).
The server runs nothing else really apart from sshd and it uses ntpd to
sync local time.
How come such a lightly loaded server with plenty of resources is eating
up swap? If I run two bhyve instances, i.e. two of the same size as
indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace
errors in the daily logs:
+swap_pager_getswapspace(24): failed
+swap_pager_getswapspace(24): failed
+swap_pager_getswapspace(24): failed
Here's top, with one bhyve instance running:
last pid: 49494; load averages: 0.12, 0.13, 0.88
up 29+11:36:06 22:52:45
54 processes: 1 running, 53 sleeping
CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle
Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf,
4123M Free
ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other
103G Compressed, 104G Uncompressed, 1.00:1 Ratio
Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh
32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve
49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp
From the looks of it, a huge amount of ram is wired. Why is that, and
how would I debug it?
A server of similar spec which is running freebsd-current with seven
bhyve instances doesn't have this issue:
last pid: 41904; load averages: 0.26, 0.19, 0.15
up 17+01:06:11 23:14:13
27 processes: 1 running, 26 sleeping
CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle
Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free
ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other
53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead
Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse
thanks,
--
J.
On Sun, Jun 17, 2018 at 5:19 PM, tech-lists <tech-lists at zyxst.net> wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, E5-2630 > @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, 4123M > Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and how > would I debug it? >That seems to be shown in the output you provided: ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other> > A server of similar spec which is running freebsd-current with seven bhyve > instances doesn't have this issue: >Based upon the output neither ram nor swap seems like similar spec so I wonder if you could say what you mean by that. -- Adam
I'd suggest - if you haven't done so yet - you familiarize yourself with DTrace and write a probe that fires when swap_pager_getswapspace() fails, to print execname and both kernel and userland stacks (or aggregations thereof). That should give you a starting idea of what's going on. HTH Michael On Mon, Jun 18, 2018 at 12:20 AM tech-lists <tech-lists at zyxst.net> wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-questions at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe at freebsd.org" >-- Michael Schuster http://recursiveramblings.wordpress.com/ recursion, n: see 'recursion'
Hi, On Sun, 17 Jun 2018 23:19:02 +0100 tech-lists <tech-lists at zyxst.net> wrote:> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inusethis might not be related but I noticed that your swap space is small compared to RAM size. I noticed on a much smaller Raspberry Pi, that it runs into trouble when there is no swap even there is enough RAM available. Is it easily possible for you to add some GB of swap space and let the machine run then? How much swap do the other machines have? Erich
I've noticed you've got a rather large ZFS ARC. You could try limiting the ZFS max ARC size by setting the vfs.zfs.arc_max sysctl. On Sun, Jun 17, 2018, at 6:19 PM, tech-lists wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"