Hello list, context is (server) freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, E5-2630 @2.3GHz, generic kernel. There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also freebsd-11-stable) There have been no special options for zfs configuration on the server, apart from several datasets having the compressed property set (lz4). The server runs nothing else really apart from sshd and it uses ntpd to sync local time. How come such a lightly loaded server with plenty of resources is eating up swap? If I run two bhyve instances, i.e. two of the same size as indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace errors in the daily logs: +swap_pager_getswapspace(24): failed +swap_pager_getswapspace(24): failed +swap_pager_getswapspace(24): failed Here's top, with one bhyve instance running: last pid: 49494; load averages: 0.12, 0.13, 0.88 up 29+11:36:06 22:52:45 54 processes: 1 running, 53 sleeping CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, 4123M Free ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other 103G Compressed, 104G Uncompressed, 1.00:1 Ratio Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp From the looks of it, a huge amount of ram is wired. Why is that, and how would I debug it? A server of similar spec which is running freebsd-current with seven bhyve instances doesn't have this issue: last pid: 41904; load averages: 0.26, 0.19, 0.15 up 17+01:06:11 23:14:13 27 processes: 1 running, 26 sleeping CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse thanks, -- J.
On Sun, Jun 17, 2018 at 5:19 PM, tech-lists <tech-lists at zyxst.net> wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, E5-2630 > @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, 4123M > Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and how > would I debug it? >That seems to be shown in the output you provided: ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other> > A server of similar spec which is running freebsd-current with seven bhyve > instances doesn't have this issue: >Based upon the output neither ram nor swap seems like similar spec so I wonder if you could say what you mean by that. -- Adam
I'd suggest - if you haven't done so yet - you familiarize yourself with DTrace and write a probe that fires when swap_pager_getswapspace() fails, to print execname and both kernel and userland stacks (or aggregations thereof). That should give you a starting idea of what's going on. HTH Michael On Mon, Jun 18, 2018 at 12:20 AM tech-lists <tech-lists at zyxst.net> wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-questions at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe at freebsd.org" >-- Michael Schuster http://recursiveramblings.wordpress.com/ recursion, n: see 'recursion'
Hi, On Sun, 17 Jun 2018 23:19:02 +0100 tech-lists <tech-lists at zyxst.net> wrote:> freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inusethis might not be related but I noticed that your swap space is small compared to RAM size. I noticed on a much smaller Raspberry Pi, that it runs into trouble when there is no swap even there is enough RAM available. Is it easily possible for you to add some GB of swap space and let the machine run then? How much swap do the other machines have? Erich
I've noticed you've got a rather large ZFS ARC. You could try limiting the ZFS max ARC size by setting the vfs.zfs.arc_max sysctl. On Sun, Jun 17, 2018, at 6:19 PM, tech-lists wrote:> Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"