freebsd stable - Sep 2020 - How to free used Swap-Space?

On Tue, Sep 22, 2020 at 06:08:01PM +0200, Peter wrote:
> Hi all,
> 
>  my machine should use about 3-4, maybe 5 GB swapspace. Today I found
> it suddenly uses 8 GB (which is worryingly near the configured 10G).
> 
> I stopped all the big suckers - nothing found.
> I stopped all the jails - no success.
> I brought it down to singleuser: it tried to swapoff, but failed.
> 
> I unmounted all filesystems, exported all pools, detached all geli,
> and removed most of the netgraphs. Swap is still occupied.
> 
> Machine is now running only the init and a shell processes, has
> almost no filesystems mounted, has mostly native networks only, and
> this still occupies 3 GB of swap which cannot be released.
> 
> What is going on, what is doing this, and how can I get this swapspace
> released??
Do you have any shared memory segments lingering?  ipcs -a will show
SysV shared memory usage.  For POSIX shared memory, in 11.4 we do not
have any good way of listing objects, but "vmstat -m | grep shmfd"
will
at least show whether any are allocated.  If those don't turn anything
up then it's possible that there's a swap leak.  Do you use any DRM
graphics drivers on this system?
> 
> It is 11.4-RELEASE-p3 amd64.
> 
> 
> Script started on Mon Sep 21 05:43:20 2020
> root at edge# ps axlww
> UID   PID  PPID CPU PRI NI  VSZ  RSS MWCHAN   STAT TT         TIME COMMAND
>   0     0     0   0 -16  0    0  752 swapin   DLs   -    291:32.41 [kernel]
>   0     1     0   0  20  0 5416  248 wait     ILs   -      0:00.22
/sbin/init --
>   0     2     0   0 -16  0    0   16 ftcl     DL    -      0:00.00
[ftcleanup]
>   0     3     0   0 -16  0    0   16 crypto_w DL    -      0:00.00 [crypto]
>   0     4     0   0 -16  0    0   16 crypto_r DL    -      0:00.00 [crypto
returns]
>   0     5     0   0 -16  0    0   32 -        DL    -     11:41.94 [cam]
>   0     6     0   0  -8  0    0   80 t->zthr_ DL    -     13:07.13
[zfskern]
>   0     7     0   0 -16  0    0   16 waiting_ DL    -      0:00.00
[sctp_iterator]
>   0     8     0   0 -16  0    0   16 -        DL    -      2:05.20
[rand_harvestq]
>   0     9     0   0 -16  0    0   16 -        DL    -      0:00.04
[soaiod1]
>   0    10     0   0 155  0    0   64 -        RNL   -  17115:06.48 [idle]
>   0    11     0   0 -52  0    0  352 -        WL    -     49:05.30 [intr]
>   0    12     0   0 -16  0    0   64 sleep    DL    -     16:28.51
[ng_queue]
>   0    13     0   0  -8  0    0   48 -        DL    -     23:10.60 [geom]
>   0    14     0   0 -16  0    0   16 seqstate DL    -      0:00.00
[sequencer 00]
>   0    15     0   0 -68  0    0  160 -        DL    -      0:23.64 [usb]
>   0    16     0   0 -16  0    0   16 -        DL    -      0:00.04
[soaiod2]
>   0    17     0   0 -16  0    0   16 -        DL    -      0:00.04
[soaiod3]
>   0    18     0   0 -16  0    0   16 -        DL    -      0:00.04
[soaiod4]
>   0    19     0   0 -16  0    0   16 idle     DL    -      0:00.83
[enc_daemon0]
>   0    20     0   0 -16  0    0   48 psleep   DL    -     12:07.72
[pagedaemon]
>   0    21     0   0  20  0    0   16 psleep   DL    -      4:12.41
[vmdaemon]
>   0    22     0   0 155  0    0   16 pgzero   DNL   -      0:00.00
[pagezero]
>   0    23     0   0 -16  0    0   64 psleep   DL    -      0:23.50
[bufdaemon]
>   0    24     0   0  20  0    0   16 -        DL    -      0:04.21
[bufspacedaemon]
>   0    25     0   0  16  0    0   16 syncer   DL    -      0:32.48 [syncer]
>   0    26     0   0 -16  0    0   16 vlruwt   DL    -      0:02.31 [vnlru]
>   0    27     0   0 -16  0    0   16 -        DL    -      7:11.58 [racctd]
>   0   157     0   0  20  0    0   16 geli:w   DL    -      0:22.03
[g_eli[0] ada1p2]
>   0   158     0   0  20  0    0   16 geli:w   DL    -      0:22.77
[g_eli[1] ada1p2]
>   0   159     0   0  20  0    0   16 geli:w   DL    -      0:31.08
[g_eli[2] ada1p2]
>   0   160     0   0  20  0    0   16 geli:w   DL    -      0:29.41
[g_eli[3] ada1p2]
>   0 70865     1   0  20  0 7076 3104 wait     Ss   v0      0:00.21 -sh (sh)
>   0 71135 70865   0  20  0 6392 2308 select   S+   v0      0:00.00 script
>   0 71136 71135   0  23  0 7076 3068 wait     Ss    0      0:00.00 /bin/sh
-i
>   0 71142 71136   0  23  0 6928 2584 -        R+    0      0:00.00 ps axlww
> 
> root at edge# df
> Filesystem  512-blocks    Used   Avail Capacity  Mounted on
> /dev/ada3p3    1936568  860864  920784    48%    /
> devfs                2       2       0   100%    /dev
> procfs               8       8       0   100%    /proc
> /dev/ada3p4    3099192 1184896 1666368    42%    /usr
> /dev/ada3p5     580344    8112  525808     2%    /var
> 
> root at edge# pstat -s
> Device          512-blocks     Used    Avail Capacity
> /dev/ada1p2.eli   10485760  5839232  4646528    56%
> 
> root at edge# top | cat
> last pid: 71147;  load averages:  0.19,  0.08,  0.09  up 3+03:21:00   
05:44:12
> 5 processes:    1 running, 4 sleeping
> 
> Mem: 9732K Active, 10M Inact, 882M Laundry, 1920M Wired, 10M Buf, 1023M
Free
> ARC: 335K Total, 16K MFU, 304K MRU, 15K Header
>      320K Compressed, 2944K Uncompressed, 9.20:1 Ratio
> Swap: 5120M Total, 2851M Used, 2269M Free, 55% Inuse
> 
> 
>   PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU
COMMAND
> 70865 root          1  20    0  7076K  3104K wait    2   0:00   0.00% sh
> 71135 root          1  20    0  6392K  2308K select  1   0:00   0.00%
script
> 71136 root          1  20    0  7076K  3068K wait    2   0:00   0.00% sh
> 71146 root          1  20    0  7928K  2980K CPU0    0   0:00   0.00% top
> 71147 root          1  20    0  6300K  2088K piperd  1   0:00   0.00% cat
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

On Tue, Sep 22, 2020 at 12:33:19PM -0400, Mark Johnston wrote:

! On Tue, Sep 22, 2020 at 06:08:01PM +0200, Peter wrote:

! >  my machine should use about 3-4, maybe 5 GB swapspace. Today I found
! > it suddenly uses 8 GB (which is worryingly near the configured 10G).
! > 
! > I stopped all the big suckers - nothing found.
! > I stopped all the jails - no success.
! > I brought it down to singleuser: it tried to swapoff, but failed.
! > 
! > I unmounted all filesystems, exported all pools, detached all geli,
! > and removed most of the netgraphs. Swap is still occupied.
! > ! > Machine is now running only the init and a shell processes, has
! > almost no filesystems mounted, has mostly native networks only, and
! > this still occupies 3 GB of swap which cannot be released.
! > 
! > What is going on, what is doing this, and how can I get this swapspace
! > released??
! 
! Do you have any shared memory segments lingering?  ipcs -a will show
! SysV shared memory usage.

I have four small shmem segments from four postgres clusters running.
These should cleanly disappear when the clusters are stopped, and
they are very small.

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP       
NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME
m        65536      5432001 --rw------- postgres postgres postgres postgres     
7           48         4793         4793  6:09:34 18:00:31  6:09:34
m        65537            0 --rw------- postgres postgres postgres postgres     
11           48         6268         6268  6:09:42 10:48:27  6:09:42
m        65538            0 --rw------- postgres postgres postgres postgres     
5           48         6968         6968  6:09:46 18:28:36  6:09:46
m        65539            0 --rw------- postgres postgres postgres postgres     
6           48         6992         6992  6:09:47  3:38:34  6:09:47

! For POSIX shared memory, in 11.4 we do not
! have any good way of listing objects, but "vmstat -m | grep shmfd"
will
! at least show whether any are allocated.

There is something, and I don't know who owns that:
$ vmstat -m | grep shmfd
        shmfd    13    14K       -      473  64,256,1024,8192

But that doesn't look big either.

Furthermore, this machine is running for quite some time already; it
was running as i386 (with ZFS) until very recently, and I know quite
well what is using much memory: these 3 GB were illegitimate; they
came from nothing I did install. And they are new; this has not
happened before.

! If those don't turn anything
! up then it's possible that there's a swap leak.  Do you use any DRM
! graphics drivers on this system?

Probably yes. There is no graphics used at all; it just uses "device
vt" in text mode, but it uses i5-3570T CPU (IvyBridge HD2500) graphics
for that, and the driver is "drm2" and "i915drm" from
/usr/src/sys (not
those from ports).
Not sure how that would account for 3 GB, unless there is indeed some
leak.

regards,
PMc