> On Feb 11, 2018, at 9:19 PM, Eugene Grosbein <eugen at grosbein.net>
wrote:
>
> 12.02.2018 11:56, Ask Bj?rn Hansen ?????:
>> Hi,
>>
>> I have an old Soekris system with 64MB memory that I upgraded from 10.3
to 11.1 recently. Since then it?s started hanging every few days.
>>
>> Today I happened to have a ?top? instance running on the serial
console. The system is minimally responsive to the network (ICMP and CARP are
working, but no services).
>>
>>> From the top output it?s not clear what resource it?s out of.
>
> I suspect it is out of many types of kernel memory including mbuf clusters,
> hence no working TCP/UDP but ICMP works.
That makes sense. The console locks up, too (as soon as I ctrl-c?ed the running
top process the console was frozen).
>> There?s no swap configured, but that what it looks like it?s trying to
do?
>>
>> The ?pf purge? process is suspicious. There are no pf rules configured
on the system (it should be all disabled).
>>
>> Any suggestions? (Other than ?seriously ? 64MB memory?!?).
>
> Please show output of commands:
>
> grep memory /var/run/dmesg.boot
real memory = 67108864 (64 MB)
avail memory = 42098688 (40 MB)
The 24MB are for the kernel? I wonder my 11.1 kernel is less discriminating
with what I compiled in...
> top -ores -d1
Shortly after boot:
last pid: 1008; load averages: 0.57, 0.62, 0.53 up 0+00:19:31 06:24:50
8 processes: 1 running, 7 sleeping
CPU: % user, % nice, % system, % interrupt, % idle
Mem: 9084K Active, 3644K Inact, 29M Wired, 4862K Buf, 492K Free
Swap:
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
911 root 1 22 0 8816K 8844K select 0:39 4.20% ntpd
959 root 1 52 0 10756K 5196K select 0:00 0.00% sshd
709 root 1 20 0 7300K 3224K select 0:00 0.00% devd
1002 root 1 20 0 6712K 2876K pause 0:01 0.49% csh
1008 root 1 49 0 7272K 2356K RUN 0:00 0.00% top
991 root 1 26 0 6400K 2196K wait 0:01 0.00% login
985 root 1 20 0 5952K 1848K nanslp 0:00 0.00% cron
788 root 1 20 0 5920K 1776K select 0:01 0.00% syslogd
> sysctl kern.ipc.nmbclusters
kern.ipc.nmbclusters: 898
kern.ipc.nmbufs: 5745
kern.ipc.maxmbufmem: 7350272
> It would be also very useful to obtain output of "vmstat -z" in a
moment of breakage.
I made it run on the console every 30 seconds, this is the last one I got. For
comparison there?s a ?shortly after boot? version below the ===?s.
Ask
Shortly before it stopped responding:
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 256, 0, 128, 7, 128, 0, 0
UMA Zones: 408, 0, 129, 6, 129, 0, 0
UMA Slabs: 56, 0, 1655, 73, 20845, 0, 0
UMA Hash: 128, 0, 11, 20, 14, 0, 0
4 Bucket: 16, 0, 0, 504, 364, 0, 0
6 Bucket: 24, 0, 0, 336, 118, 0, 0
8 Bucket: 32, 0, 0, 378, 54150, 0, 0
12 Bucket: 48, 0, 0, 0, 0, 0, 0
16 Bucket: 64, 0, 2, 313, 1840, 8, 0
32 Bucket: 128, 0, 3, 152, 1516, 0, 0
64 Bucket: 256, 0, 1, 74, 221, 0, 0
128 Bucket: 512, 0, 4, 28, 1324, 0, 0
256 Bucket: 1024, 0, 3, 13, 3917, 4, 0
vmem btag: 28, 0, 3169, 143, 3847, 56, 0
VM OBJECT: 148, 0, 1208, 115, 38997, 0, 0
RADIX NODE: 44, 10738, 1749, 344, 88089, 0, 0
MAP: 140, 0, 3, 81, 3, 0, 0
KMAP ENTRY: 72, 0, 5, 163, 5, 0, 0
MAP ENTRY: 72, 0, 244, 148, 89326, 0, 0
VMSPACE: 232, 0, 10, 75, 2062, 0, 0
fakepg: 68, 0, 0, 0, 0, 0, 0
mt_zone: 2060, 0, 352, 0, 352, 0, 0
16: 16, 0, 1159, 353, 813531, 0, 0
32: 32, 0, 957, 303, 23866, 0, 0
64: 64, 0, 1385, 253, 29315, 0, 0
128: 128, 0, 734, 134, 14162, 0, 0
256: 256, 0, 378, 57, 5062, 0, 0
512: 512, 0, 57, 7, 4807, 0, 0
1024: 1024, 0, 31, 33, 539, 0, 0
2048: 2048, 0, 208, 2, 2908, 0, 0
4096: 4096, 0, 86, 0, 2165, 0, 0
8192: 8192, 0, 0, 0, 0, 0, 0
16384: 16384, 0, 0, 0, 0, 0, 0
32768: 32768, 0, 0, 0, 0, 0, 0
65536: 65536, 0, 0, 0, 0, 0, 0
SLEEPQUEUE: 44, 0, 85, 293, 85, 0, 0
64 pcpu: 8, 0, 2642, 174, 2714, 0, 0
ptr pcpu: 4, 0, 0, 0, 0, 0, 0
Files: 56, 0, 42, 318, 18945, 0, 0
filedesc0: 888, 0, 34, 6, 2085, 0, 0
rl_entry: 28, 0, 23, 265, 23, 0, 0
TURNSTILE: 72, 0, 85, 132, 85, 0, 0
umtx pi: 52, 0, 0, 0, 0, 0, 0
umtx_shm: 52, 0, 0, 0, 0, 0, 0
MAC labels: 20, 0, 0, 0, 0, 0, 0
PROC: 916, 0, 33, 15, 2084, 0, 0
THREAD: 904, 0, 70, 14, 71, 0, 0
cpuset: 40, 0, 49, 52, 49, 0, 0
audit_record: 1112, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 5745, 259, 0, 418909, 2, 0
mbuf: 256, 5745, 263, 273, 773249, 18, 0
mbuf_cluster: 2048, 898, 259, 131, 204839,1868, 0
mbuf_jumbo_page: 4096, 448, 0, 0, 0, 0, 0
mbuf_jumbo_9k: 9216, 132, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 74, 0, 0, 0, 0, 0
g_bio: 264, 0, 0, 0, 13391, 0, 0
ttyinq: 152, 0, 15, 37, 60, 0, 0
ttyoutq: 256, 0, 8, 22, 32, 0, 0
nvme_request: 100, 0, 0, 0, 0, 0, 0
cryptop: 60, 0, 0, 0, 0, 0, 0
cryptodesc: 56, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0
FPU_save_area: 512, 0, 0, 0, 0, 0, 0
VNODE: 284, 0, 1134, 28, 1170, 0, 0
VNODEPOLL: 64, 0, 0, 0, 0, 0, 0
BUF TRIE: 44, 0, 52, 676, 2310, 0, 0
NAMEI: 1024, 0, 0, 16, 55376, 0, 0
rentr: 12, 0, 0, 0, 0, 0, 0
S VFS Cache: 72, 0, 977, 143, 2862, 0, 0
STS VFS Cache: 92, 0, 0, 0, 0, 0, 0
L VFS Cache: 292, 0, 0, 0, 0, 0, 0
LTS VFS Cache: 312, 0, 0, 0, 0, 0, 0
NCLNODE: 360, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 0, 0, 58, 0, 0
Mountpoints: 672, 0, 4, 14, 5, 0, 0
pipe: 416, 0, 0, 0, 1030, 0, 0
procdesc: 80, 0, 0, 0, 0, 0, 0
AIO: 128, 0, 0, 0, 0, 0, 0
AIOP: 16, 0, 0, 0, 0, 0, 0
AIOCB: 428, 0, 0, 0, 0, 0, 0
AIOL: 64, 0, 0, 0, 0, 0, 0
AIOLIO: 172, 0, 0, 0, 0, 0, 0
ksiginfo: 80, 0, 166, 134, 17061, 0, 0
itimer: 232, 0, 0, 0, 0, 0, 0
bridge_rtnode: 36, 0, 0, 0, 0, 0, 0
KNOTE: 72, 0, 0, 0, 0, 0, 0
socket: 524, 1393, 20, 15, 3048, 0, 0
unpcb: 172, 1403, 6, 17, 2053, 0, 0
IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0
ipq: 32, 126, 0, 0, 0, 0, 0
udp_inpcb: 304, 1404, 11, 15, 925, 0, 0
udpcb: 20, 1414, 11, 191, 925, 0, 0
tcp_inpcb: 304, 1404, 2, 11, 61, 0, 0
tcpcb: 772, 1395, 2, 3, 61, 0, 0
tcptw: 60, 335, 0, 0, 0, 0, 0
syncache: 128, 15376, 0, 0, 54, 12, 0
hostcache: 76, 15370, 0, 0, 0, 0, 0
sackhole: 20, 0, 0, 0, 0, 0, 0
tcpreass: 20, 202, 0, 0, 0, 0, 0
sctp_ep: 1052, 1395, 0, 0, 0, 0, 0
sctp_asoc: 1656, 40000, 0, 0, 0, 0, 0
sctp_laddr: 24, 80136, 0, 0, 5, 0, 0
sctp_raddr: 540, 80003, 0, 0, 0, 0, 0
sctp_chunk: 104, 400026, 0, 0, 0, 0, 0
sctp_readq: 108, 400007, 0, 0, 0, 0, 0
sctp_stream_msg_out: 72, 400008, 0, 0, 0, 0, 0
sctp_asconf: 24, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 24, 400008, 0, 0, 0, 0, 0
udplite_inpcb: 304, 1404, 0, 0, 0, 0, 0
ripcb: 304, 1404, 0, 0, 0, 0, 0
rtentry: 112, 0, 42, 30, 50, 0, 0
pf mtags: 32, 0, 0, 0, 0, 0, 0
pf states: 208, 10013, 0, 0, 0, 0, 0
pf state keys: 64, 0, 0, 0, 0, 0, 0
pf source nodes: 116, 10030, 0, 0, 0, 0, 0
pf table entries: 92, 0, 0, 0, 0, 0, 0
pf table counters: 64, 0, 0, 0, 0, 0, 0
pf frags: 80, 0, 0, 0, 0, 0, 0
pf frag entries: 24, 5040, 0, 0, 0, 0, 0
pf state scrubs: 28, 0, 0, 0, 0, 0, 0
IPFW counters: 16, 0, 1, 63, 1, 0, 0
IPFW dynamic rule: 112, 16416, 0, 0, 0, 0, 0
divcb: 304, 1404, 0, 0, 0, 0, 0
selfd: 32, 0, 38, 340, 2106598, 0, 0
SWAPMETA: 276, 5390, 0, 0, 0, 0, 0
FFS inode: 112, 0, 1109, 151, 1144, 0, 0
FFS1 dinode: 128, 0, 632, 81, 662, 0, 0
FFS2 dinode: 256, 0, 477, 33, 482, 0, 0
md0: 512, 0, 5042, 14, 5050, 0, 0
md1: 512, 0, 4298, 6, 4299, 0, 0
==================================~25 minutes after boot:
ntp1.us.grundclock.com# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 256, 0, 128, 7, 128, 0, 0
UMA Zones: 408, 0, 129, 6, 129, 0, 0
UMA Slabs: 56, 0, 1658, 70, 2158, 0, 0
UMA Hash: 128, 0, 11, 20, 14, 0, 0
4 Bucket: 16, 0, 1, 503, 254, 0, 0
6 Bucket: 24, 0, 0, 336, 118, 0, 0
8 Bucket: 32, 0, 2, 376, 149, 0, 0
12 Bucket: 48, 0, 0, 0, 0, 0, 0
16 Bucket: 64, 0, 5, 310, 35, 8, 0
32 Bucket: 128, 0, 6, 149, 396, 0, 0
64 Bucket: 256, 0, 3, 72, 47, 0, 0
128 Bucket: 512, 0, 9, 23, 52, 0, 0
256 Bucket: 1024, 0, 15, 13, 110, 0, 0
vmem btag: 28, 0, 3101, 67, 3133, 22, 0
VM OBJECT: 148, 0, 1187, 109, 12302, 0, 0
RADIX NODE: 44, 10738, 1712, 381, 33112, 0, 0
MAP: 140, 0, 3, 81, 3, 0, 0
KMAP ENTRY: 72, 0, 5, 163, 5, 0, 0
MAP ENTRY: 72, 0, 227, 165, 27414, 0, 0
VMSPACE: 232, 0, 9, 76, 999, 0, 0
fakepg: 68, 0, 0, 0, 0, 0, 0
mt_zone: 2060, 0, 352, 0, 352, 0, 0
16: 16, 0, 1163, 349, 61329, 0, 0
32: 32, 0, 955, 305, 5553, 0, 0
64: 64, 0, 1366, 272, 20940, 0, 0
128: 128, 0, 726, 142, 13237, 0, 0
256: 256, 0, 374, 61, 1906, 0, 0
512: 512, 0, 57, 31, 1553, 0, 0
1024: 1024, 0, 31, 33, 526, 0, 0
2048: 2048, 0, 208, 6, 2703, 0, 0
4096: 4096, 0, 85, 2, 1101, 0, 0
8192: 8192, 0, 0, 0, 0, 0, 0
16384: 16384, 0, 0, 0, 0, 0, 0
32768: 32768, 0, 0, 0, 0, 0, 0
65536: 65536, 0, 0, 0, 0, 0, 0
SLEEPQUEUE: 44, 0, 77, 301, 77, 0, 0
64 pcpu: 8, 0, 2642, 174, 2714, 0, 0
ptr pcpu: 4, 0, 0, 0, 0, 0, 0
Files: 56, 0, 42, 318, 8074, 0, 0
filedesc0: 888, 0, 33, 15, 1022, 0, 0
rl_entry: 28, 0, 16, 272, 16, 0, 0
TURNSTILE: 72, 0, 77, 140, 77, 0, 0
umtx pi: 52, 0, 0, 0, 0, 0, 0
umtx_shm: 52, 0, 0, 0, 0, 0, 0
MAC labels: 20, 0, 0, 0, 0, 0, 0
PROC: 916, 0, 32, 12, 1021, 0, 0
THREAD: 904, 0, 62, 14, 63, 0, 0
cpuset: 40, 0, 49, 52, 49, 0, 0
audit_record: 1112, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 5745, 128, 253, 32698, 0, 0
mbuf: 256, 5745, 1, 263, 57270, 0, 0
mbuf_cluster: 2048, 898, 381, 3, 726, 0, 0
mbuf_jumbo_page: 4096, 448, 0, 0, 0, 0, 0
mbuf_jumbo_9k: 9216, 132, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 74, 0, 0, 0, 0, 0
g_bio: 264, 0, 0, 30, 11703, 0, 0
ttyinq: 152, 0, 15, 37, 60, 0, 0
ttyoutq: 256, 0, 8, 22, 32, 0, 0
nvme_request: 100, 0, 0, 0, 0, 0, 0
cryptop: 60, 0, 0, 0, 0, 0, 0
cryptodesc: 56, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 16, 0, 0, 0, 0, 0, 0
FPU_save_area: 512, 0, 0, 0, 0, 0, 0
VNODE: 284, 0, 1130, 32, 1165, 0, 0
VNODEPOLL: 64, 0, 0, 0, 0, 0, 0
BUF TRIE: 44, 0, 51, 677, 1736, 0, 0
NAMEI: 1024, 0, 0, 16, 21104, 0, 0
rentr: 12, 0, 0, 0, 0, 0, 0
S VFS Cache: 72, 0, 972, 148, 2134, 0, 0
STS VFS Cache: 92, 0, 0, 0, 0, 0, 0
L VFS Cache: 292, 0, 0, 0, 0, 0, 0
LTS VFS Cache: 312, 0, 0, 0, 0, 0, 0
NCLNODE: 360, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 9, 11, 27, 0, 0
Mountpoints: 672, 0, 4, 14, 5, 0, 0
pipe: 416, 0, 0, 9, 534, 0, 0
procdesc: 80, 0, 0, 0, 0, 0, 0
AIO: 128, 0, 0, 0, 0, 0, 0
AIOP: 16, 0, 0, 0, 0, 0, 0
AIOCB: 428, 0, 0, 0, 0, 0, 0
AIOL: 64, 0, 0, 0, 0, 0, 0
AIOLIO: 172, 0, 0, 0, 0, 0, 0
ksiginfo: 80, 0, 30, 170, 1209, 0, 0
itimer: 232, 0, 0, 0, 0, 0, 0
bridge_rtnode: 36, 0, 0, 0, 0, 0, 0
KNOTE: 72, 0, 0, 0, 0, 0, 0
socket: 524, 1393, 20, 8, 2030, 0, 0
unpcb: 172, 1403, 6, 17, 1748, 0, 0
IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0
ipq: 32, 126, 0, 0, 0, 0, 0
udp_inpcb: 304, 1404, 11, 15, 262, 0, 0
udpcb: 20, 1414, 11, 191, 262, 0, 0
tcp_inpcb: 304, 1404, 2, 11, 11, 0, 0
tcpcb: 772, 1395, 2, 3, 11, 0, 0
tcptw: 60, 335, 0, 0, 0, 0, 0
syncache: 128, 15376, 0, 0, 4, 0, 0
hostcache: 76, 15370, 0, 0, 0, 0, 0
sackhole: 20, 0, 0, 0, 0, 0, 0
tcpreass: 20, 202, 0, 0, 0, 0, 0
sctp_ep: 1052, 1395, 0, 0, 0, 0, 0
sctp_asoc: 1656, 40000, 0, 0, 0, 0, 0
sctp_laddr: 24, 80136, 0, 0, 5, 0, 0
sctp_raddr: 540, 80003, 0, 0, 0, 0, 0
sctp_chunk: 104, 400026, 0, 0, 0, 0, 0
sctp_readq: 108, 400007, 0, 0, 0, 0, 0
sctp_stream_msg_out: 72, 400008, 0, 0, 0, 0, 0
sctp_asconf: 24, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 24, 400008, 0, 0, 0, 0, 0
udplite_inpcb: 304, 1404, 0, 0, 0, 0, 0
ripcb: 304, 1404, 0, 0, 0, 0, 0
rtentry: 112, 0, 42, 30, 50, 0, 0
pf mtags: 32, 0, 0, 0, 0, 0, 0
pf states: 208, 10013, 0, 0, 0, 0, 0
pf state keys: 64, 0, 0, 0, 0, 0, 0
pf source nodes: 116, 10030, 0, 0, 0, 0, 0
pf table entries: 92, 0, 0, 0, 0, 0, 0
pf table counters: 64, 0, 0, 0, 0, 0, 0
pf frags: 80, 0, 0, 0, 0, 0, 0
pf frag entries: 24, 5040, 0, 0, 0, 0, 0
pf state scrubs: 28, 0, 0, 0, 0, 0, 0
IPFW counters: 16, 0, 1, 63, 1, 0, 0
IPFW dynamic rule: 112, 16416, 0, 0, 0, 0, 0
divcb: 304, 1404, 0, 0, 0, 0, 0
selfd: 32, 0, 32, 220, 150507, 0, 0
SWAPMETA: 276, 5390, 0, 0, 0, 0, 0
FFS inode: 112, 0, 1105, 155, 1139, 0, 0
FFS1 dinode: 128, 0, 628, 147, 658, 0, 0
FFS2 dinode: 256, 0, 477, 18, 481, 0, 0
md0: 512, 0, 5041, 7, 5049, 0, 0
md1: 512, 0, 4266, 6, 4267, 0, 0