freebsd stable - Apr 2015 - 10-STABLE live locks, looks like VM-related

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512


 I got several live locks of my server in a row (3 in one week).
 It is amd64 10-STABLE r277307.
 Looks like live locks is VM related but manifest themselves under
multi-threaded mixed CPU + I/O load (CrashPlan backup + torrents +
openjdk8 rebuild, for example).

 This system doesn't have ZFS, but have several UFS2 SU+J with
different block sizes (16Kb and 32Kb)! As far as I remeber several
versions ago "16Kb + 64Kb" mix was a killer (known bug). but "16
+ 32"
works well for several years.

 I have INVARIANTS and WITNESS in the kernel, but it doesn't help:
only report is "bufwait/dirhash" right after booting.

 When system hangs, I could ping it, but NFS, SMB, ssh, local console,
everything else userland-related stop to answer.

 I could break into kernel debugger on console and "panic" manually,
so I have two "crash dumps" of system in this state.

 Many processes are in "vmwait" or "pfault" state according
to DDB's
"ps" output.

 here are logs from two latest crashes:

 http://lev.serebryakov.spb.ru/freebsd/ll/

 I've tested memory with memtest86 for 12 hours without errors.

- -- 
// Lev Serebryakov AKA Black Lion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQJ8BAEBCgBmBQJVIknjXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF
QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP5ZkP+QGJje5ejYvKrDS28/Dqrew8
7H07Oqh6XXHG/CPqsQts0MCSfJTjZ/XnaYM5hFARJQYBVj+nZfMyIgaUTgFp6hOd
UnQ+qGYgEuTYm3uxewyPL4C6NtLTJq79Au28oz6ONIbLsl5VQIkTxJMn/40RxtQr
EjqXSDc2al9s36bOidbD/ZdvuK91NaYcNH2tCvv2uNKR8SwA4LXPU+P/K3MtMDsT
K5c+yI9Fb0OWmYAF7wTUgrUrJeXxWcVyTsirtBB4C4sKXFKz1RDieGvUWnVjikbP
27c+cOpGPFzf3EgqFDwER2tLXdoB/YJ4BEY87alNI+mogjNrQa+2ck4wetfUKnSS
QrYKUEoQCJuskbgc2J4LKxIPGOJfYGLBIKv5QzlPcdu5hVOiwNmxn5zpaHbI8nRR
ikXhHmUl5dBOJ1u+6p6fPFJ38l8Ig+vZKRPhSsft6450yftLPliPiOH43mTaAoyd
dhCO0xHBbTNmt0QJEyniyBKxIEs3qYBFYFSQmZjiIXY70lP16Tea0m5r5zt9JYr6
j38IBgFL+TbK0BhwSU0479d29J7loOuIutoVbzKwpgjgu8eGZRcILjyAFonUX4EA
r90Lp4WUjR7qsjyCqFLG+g8kt4Hr6oTKAbMTtscWWlst5PCUOo07j6NAOhTjj5bV
XfC55FMCCuSa95UEC/5M
=a3a7
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 06.04.2015 11:55, Lev Serebryakov wrote:
> I have INVARIANTS and WITNESS in the kernel, but it doesn't help: 
> only report is "bufwait/dirhash" right after booting.
 One more LOR:

lock order reversal:
 1st 0xfffff8011b9b82d8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2157
 2nd 0xfffffe0170ffaec8 bufwait (bufwait) @
/usr/src/sys/ufs/ffs/ffs_vnops.c:262
 3rd 0xfffff800b6587a28 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2157


- -- 
// Lev Serebryakov AKA Black Lion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQJ8BAEBCgBmBQJVIlzTXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF
QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP8RYP/jJjw6sru0HIqHlPKAAbIFoT
f2NFqSI7aBlfYghC9mQxIpSGOtT5trrG5ZN2hg1jONbk/c2gmSm4otknp4lPDU00
jEghidQ5XQEpSsyr23fMvou3NYLn1iGKGYNuIJE1VZshQUdtxnBUbMokPZb5aVyu
nn3jUsxW3bcHxRR1ygHjU1XC/nv0AKw1xhy6aHodJci9TJnUxOFPEiYWXe3E66YC
83miSicOeJZTPaji2IH6T1pO2aqOaVZhqioAku5eqt6R3NEb41zNyeID6+4swgXb
bDdo7mE9i9L/QTOoAoVn1eKgLLJialhOpDxj3wHOiM3PeziTwS6Hw40F+7oCKcqi
gor5anq9apu9x8npElUGJOgSBZDxEuFzZBWx5ApJj7Iq3DqRF8iGY5/OJ++XP86O
PsKX9AhtQxQojTBqk5FXjSp5LRQLGSAN6m2uB4A7cB17r/MgJo0ZXer1ryZTwNXp
7zVL1xaZNcZHfZusrDs6m69Czpm1PE5Z2e33qa7YNFLihRpBbYazfVhDFORhVS5W
FUeQWJzIdYe7AA7hl/7hpCL2G8dj1+RH5oyQX1kEF76bOSx5LEgPr4vE3WnoTQQr
Go4uFUBJ9zjjAnKdKhUXfJG3i7rBNGwTuAxkU+o+jLX+Ry00iP0GC9xLJBJGtQZo
Kwnd2fW/6hLfNvTzivbS
=o9pc
-----END PGP SIGNATURE-----

Hello Lev,

Monday, April 6, 2015, 11:55:09 AM, you wrote:
>  I got several live locks of my server in a row (3 in one week).
>  It is amd64 10-STABLE r277307.
  And again, rebuilded to r281159
>  http://lev.serebryakov.spb.ru/freebsd/ll/
>  I've tested memory with memtest86 for 12 hours without errors.
  Added as "core.txt.6"



-- 
Best regards,
 Lev                            mailto:lev at serebryakov.spb.ru
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 960 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20150409/15aac3d4/attachment.sig>