Juan Ramón Molina Menor
2014-Dec-10 14:28 UTC
System hang on shutdown when running freebsd-update
El 10/12/2014 14:36, andrew clarke escribi?:> On Wed 2014-12-10 14:13:28 UTC+0100, Juan Ram?n Molina Menor (listjm at club-internet.fr) wrote: > >>>> / Another update. I narrowed down the issue due to /sbin/init being >>>> replaced. This might be the magic that freebsd-update does to make the >>>> crash happen. (Although there might be more situations that cause a >>>> hang.) >>> >>> />/ I can completely reliably trigger the hang on a default >>> />/ 10.1-RELEASE install on UFS2 in VMware Fusion with the following >>>>> / procedure: >>> />/ >>> />/ # chflags noschg /sbin/init >>> />/ # cp -Rp /sbin/init /sbin/init2 >>> />/ # rm -f /sbin/init >>> />/ # mv /sbin/init2 /sbin/init >>> />/ # chflags schg /sbin/init >>> />/ # reboot >>> />/ => Hang after "All buffers synced." >>> /These commands do not trigger the issue here on a 10.1-RELEASE on VMware >>> Player, but do it in a real system with the same release, both i386. >>> When I last freebsd-updated this latter machine from 10.0 to >>> 10.1-RELEASE, I also remember seeing the hang on shutdown. >>> >>> Hope it helps, >>> Juan >> >> The hang occurs again when running freebsd-update to install the >> security advisories of today on a 10.1-RELEASE i386. No problem under >> VMWare, same version. >> >> Best regards, >> Juan > > A FreeBSD 10.1-REL amd64 VM here also hangs after /sbin/reboot is run, > after installing updates with freebsd-update. The VM runs under > VirtualBox on an Ubuntu 14.04 host. > > On hard-resetting the VM, it appears the filesystem (UFS) wasn't flushed: > > Dec 11 00:29:52 vbox-freebsd kernel: WARNING: / was not properly dismountedYes, I forgot that, same here on bare metal.> > Regards > Andrew >
> On 10 Dec 2014, at 15:28, Juan Ram?n Molina Menor <listjm at club-internet.fr> wrote: > >> A FreeBSD 10.1-REL amd64 VM here also hangs after /sbin/reboot is run, >> after installing updates with freebsd-update. The VM runs under >> VirtualBox on an Ubuntu 14.04 host. >> >> On hard-resetting the VM, it appears the filesystem (UFS) wasn't flushed: >> >> Dec 11 00:29:52 vbox-freebsd kernel: WARNING: / was not properly dismounted > > Yes, I forgot that, same here on bare metal.I?d expect it to happen again, since the update touched /sbin/init, and that seems to be a way to get reboot/unmount hanging with UFS2+softupdates. (See my minimal example in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458>) A workaround to prevent the hang/fsck is to first disable softupdates on / before doing the freebsd-update. We updated around 40 boxes with this one weird trick, and we did not have the problem. I?ve tried if the hang is apparent in a 11-CURRENT snapshot too, and it is. (Tried the iso snapshot available today, which is r273635). WITNESS indicates a lock order reversal, which looks related to me: root at current:~ # chflags noschg /sbin/init root at current:~ # cp -Rp /sbin/init /sbin/init2 lock order reversal: 1st 0xfffffe007b842fa0 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3093 2nd 0xfffff80002b9ea00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe000025c270 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe000025c320 witness_checkorder() at witness_checkorder+0xdad/frame 0xfffffe000025c3b0 _sx_xlock() at _sx_xlock+0x75/frame 0xfffffe000025c3f0 ufsdirhash_add() at ufsdirhash_add+0x3a/frame 0xfffffe000025c430 ufs_direnter() at ufs_direnter+0x6a0/frame 0xfffffe000025c4f0 ufs_makeinode() at ufs_makeinode+0x560/frame 0xfffffe000025c6a0 VOP_CREATE_APV() at VOP_CREATE_APV+0xf1/frame 0xfffffe000025c6d0 vn_open_cred() at vn_open_cred+0x29d/frame 0xfffffe000025c820 kern_openat() at kern_openat+0x26f/frame 0xfffffe000025c9a0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe000025cab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe000025cab0 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x80094f01a, rsp = 0x7fffffffe958, rbp = 0x7fffffffe9c0 --- Screenshot is here: http://lf.ms/current-r273635-hang-1.png <http://lf.ms/current-r273635-hang-1.png> Finally, when rebooting, another lock order reversal appears and the system hangs. I don?t have a text log of this, so I?ll copy the first few lines: Syncing disks, vnodes remaining?1 0 0 done All buffers synced. lock order reversal: 1st 0xfffff80002e65d50 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1223 2nd 0xfffff80002e665f0 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2144 Screenshot is here: http://lf.ms/current-r273635-hang-2.png <http://lf.ms/current-r273635-hang-2.png> I don?t have kernel hacking experience, but these source files look awfully related to the parts that we are having problems with. I would really love some research into this and possibly an errata for 10.1. What can we do to make this actionable? WH -- Walter Hop | PGP key: https://lifeforms.nl/pgp