freebsd stable - Feb 2016 - 10-STABLE hangups frequently

On 2016-Feb-02 16:55:46 +0900, Hajimu UMEMOTO <ume at mahoroba.org>
wrote:
>I'm disturbed by a frequent hangup of my 10-STABLE boxes since this
>year.  It seems occur during running the periodic daily scripts.
>I've narrowed which commit causes this problem.  It seems r292895
>causes it.  I see many `Resource temporarily unavailable' message just
>before hangup occurs.
>Any idea?
As others have said, you need to provide lots more detail on your
configuration.

That said, I'm seeing something potentially similar on a Google
Compute Engine f1-micro instance (1 vCPU, 0.6GB RAM) that is running
FreeBSD 10-stable/amd64 with ZFS but basically idle.  (Yes, I realize
that's very little RAM for ZFS but I previously had no problems with
things like buildworld).

There were no problems at r290231 but after I upgraded to r295005, I
started seeing "out of swap" errors and hangs during the periodic
daily runs.  I'm not seeing this on 1GB instances - though they are
all running UFS.

Some experimentation suggested that just "find /" was enough to wedge
my system.  I did some experimenting and found that the following
loader config was enough to prevent it hanging:
vfs.zfs.arc_max="128M"
vfs.zfs.arc_meta_limit="50M"
vfs.zfs.arc_min="25M"
(previously, I had no ZFS tuning at all).

One odditity was that I would semi-regularly see:
 kernel: pid 67431 (ntpd), uid 0, was killed: out of swap space
I haven't worked out why the OOM killer preferred ntpd to anything else -
it didn't seem to be bigger.  And I didn't see any signs that swap space
was being consumed (though I haven't done a scientific examination).
(Note that swap is on a raw partition).

The behaviour is definitely a regression and my initial suspicion is ZFS,
though I haven't identified any smoking gun.

Unfortunately, GCE only offers read access to the console, so I can't
use DDB to poke around after it wedges.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160203/42b16daa/attachment.sig>

Hi,
>>>>> On Wed, 3 Feb 2016 07:07:38 +1100
>>>>> Peter Jeremy <peter at rulingia.com> said:
peter> As others have said, you need to provide lots more detail on your
peter> configuration.

CPU: AMD Athlon(tm) 64 Processor 3500+
Memory: 4GB
HDD: 3TB

I'm using ZFS only setup.

peter> There were no problems at r290231 but after I upgraded to r295005, I
peter> started seeing "out of swap" errors and hangs during the
periodic
peter> daily runs.  I'm not seeing this on 1GB instances - though they
are
peter> all running UFS.

r292875 runs well:

FreeBSD asuka.mahoroba.org 10.2-STABLEFreeBSD 10.2-STABLE #5 r292875: Tue Feb  2
07:08:29 JST 2016 root at asuka.mahoroba.org:/usr/obj/usr/src/sys/ASUKA  amd6

r292895 hangs:

FreeBSD asuka.mahoroba.org 10.2-STABLE FreeBSD 10.2-STABLE #6 r292895: Tue Feb 
2 10:17:28 JST 2016 root at asuka.mahoroba.org:/usr/obj/usr/src/sys/ASUKA  amd64

I tried latest stable (r295137) with the sys/kern/vfs_subr.c part of
r292895 reverted, and it seems running well, here:

FreeBSD asuka.mahoroba.org 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #0 r295137M:
Tue Feb  2 20:39:11 JST 2016     root at
asuka.mahoroba.org:/usr/obj/usr/src/sys/ASUKA  amd64

peter> Some experimentation suggested that just "find /" was enough
to wedge
peter> my system.  I did some experimenting and found that the following
peter> loader config was enough to prevent it hanging:
peter> vfs.zfs.arc_max="128M"
peter> vfs.zfs.arc_meta_limit="50M"
peter> vfs.zfs.arc_min="25M"
peter> (previously, I had no ZFS tuning at all).

I had ZFS tuning before.  However, after this problem was occur, I
removed all of ZFS tuning.
The FS related setting is only kern.maxvnodes=400000, now.

Sincerely,

--
Hajimu UMEMOTO
ume at mahoroba.org  ume at FreeBSD.org
http://www.mahoroba.org/~ume/