On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote:> Joel wrote: > > Hi, > > > > Just about every machine I have seems to hang after running freebsd-update and doing a reboot. The last message on the screen is "All buffers synced? and it just freezes. > > > > This happens when doing a freebsd-update and going from 10.0 to 10.1, but also when doing a fresh 10.1 install and using freebsd-update to get the latest -pX security patches. As soon as I reboot the machine, it hangs. > > > > I?ve tried it on several different HP ProLiant models, on Intel NUCs and on VMware virtual machines. Same phenomenon everywhere. It?s really easy to trigger: just install 10.1, use default settings everywhere, freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think I?ve seen it on > > > > > > > > > > 30 servers or so now. > > > > Everything works like it should after the initial hang tough - no matter how many times I reboot it completes the reboot cycle just fine. > > > > I?ve seen several people (mostly on IRC) mention this problem, but no solution. > > > > Is anyone working on fixing this? > > I ran into this problem in spades when upgrading a set of servers from > FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, > but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung > at the same place, after printing the "All buffers synced" message. > > I ultimately determined that if I did the following, rather than > just a "reboot" or "shutdown -r now 'FreeBSD 9.1-RELEASE upgrade'", > it would consistently AVOID the hang: > > sync ; sync ; sync ; shutdown -o -n -r now "FreeBSD 9.1 install" > > Your mileage may vary, but you don't have a lot to lose by trying it. > > -Kurt >That is just bad advice. sync(1) does not g'tee that all data has been written, no matter how many times you type it. shutdown -n tells the system to abandon unwritten data. All in all, this is a recipe for silent filesystem corruption. Using it after an update is just asking to have a mix of old and new files on the system after the reboot. A more robust workaround would be to "mount -r" on all filesystems before invoking the shutdown (even a shutdown -n should be safe after everything has been remounted readonly). If the mount -r hangs on one of the filesystems, then you've probably got a clue as to where a normal shutdown is hanging. -- Ian
On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore <ian at freebsd.org> wrote:> On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: > > Joel wrote: > > > Hi, > > > > > > Just about every machine I have seems to hang after running > freebsd-update and doing a reboot. The last message on the screen is "All > buffers synced? and it just freezes. > > > > > > This happens when doing a freebsd-update and going from 10.0 to 10.1, > but also when doing a fresh 10.1 install and using freebsd-update to get > the latest -pX security patches. As soon as I reboot the machine, it hangs. > > > > > > I?ve tried it on several different HP ProLiant models, on Intel NUCs > and on VMware virtual machines. Same phenomenon everywhere. It?s really > easy to trigger: just install 10.1, use default settings everywhere, > freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think > I?ve seen it on > > > > > > > > > > > > > > > 30 servers or so now. > > > > > > Everything works like it should after the initial hang tough - no > matter how many times I reboot it completes the reboot cycle just fine. > > > > > > I?ve seen several people (mostly on IRC) mention this problem, but no > solution. > > > > > > Is anyone working on fixing this? > > > > I ran into this problem in spades when upgrading a set of servers from > > FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, > > but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung > > at the same place, after printing the "All buffers synced" message. > > > > I ultimately determined that if I did the following, rather than > > just a "reboot" or "shutdown -r now 'FreeBSD 9.1-RELEASE upgrade'", > > it would consistently AVOID the hang: > > > > sync ; sync ; sync ; shutdown -o -n -r now "FreeBSD 9.1 install" > > > > Your mileage may vary, but you don't have a lot to lose by trying it. > > > > -Kurt > > > > That is just bad advice. sync(1) does not g'tee that all data has been > written, no matter how many times you type it. shutdown -n tells the > system to abandon unwritten data. All in all, this is a recipe for > silent filesystem corruption. Using it after an update is just asking > to have a mix of old and new files on the system after the reboot. > > A more robust workaround would be to "mount -r" on all filesystems > before invoking the shutdown (even a shutdown -n should be safe after > everything has been remounted readonly). If the mount -r hangs on one > of the filesystems, then you've probably got a clue as to where a normal > shutdown is hanging. > > -- IanI agree, Ian. This is especially true for any system running an active database. By the way, the purpose of typing "sync" three times was to spend the time typing it. Putting it on a single line completely eliminates the advantage of three "sync"s. I have had success with a different method: # shutdown now Wait 30 seconds # reboot This has always worked for my to this point, but I can't promise anything. I have only one system running a RELEASE these days. I'd be far more certain if I still had the 20-30 systems I had when I retired. -- Kevin Oberman, Network Engineer, Retired E-mail: rkoberman at gmail.com
Ian Lepore wrote:>> I ran into this problem in spades when upgrading a set of servers from >> FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, >> but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung >> at the same place, after printing the "All buffers synced" message. >> >> I ultimately determined that if I did the following, rather than >> just a "reboot" or "shutdown -r now 'FreeBSD 9.1-RELEASE upgrade'", >> it would consistently AVOID the hang: >> >> sync ; sync ; sync ; shutdown -o -n -r now "FreeBSD 9.1 install" >> >> Your mileage may vary, but you don't have a lot to lose by trying it. >> >> -Kurt >> > > That is just bad advice. sync(1) does not g'tee that all data has been > written, no matter how many times you type it. shutdown -n tells the > system to abandon unwritten data. All in all, this is a recipe for > silent filesystem corruption. Using it after an update is just asking > to have a mix of old and new files on the system after the reboot.I didn't specify that I was using ZFS, but I was. And this comment in the ZFS code (zfs_vfsops.c, line 170): /* * Sync all ZFS filesystems. This is what happens when you * run sync(1M). Unlike other filesystems, ZFS honors the * request by waiting for all pools to commit all dirty data. */ spa_sync_allpools(); Says that sync() won't return until the dirty data is flushed. For my use case, I believe it was perfectly safe. -Kurt
On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore <ian at freebsd.org> wrote:> On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: > > Joel wrote: > > > Hi, > > > > > > Just about every machine I have seems to hang after running > freebsd-update and doing a reboot. The last message on the screen is "All > buffers synced? and it just freezes. > > > > > > This happens when doing a freebsd-update and going from 10.0 to 10.1, > but also when doing a fresh 10.1 install and using freebsd-update to get > the latest -pX security patches. As soon as I reboot the machine, it hangs. > > > > > > I?ve tried it on several different HP ProLiant models, on Intel NUCs > and on VMware virtual machines. Same phenomenon everywhere. It?s really > easy to trigger: just install 10.1, use default settings everywhere, > freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think > I?ve seen it on > > > > > > > > > > > > > > > 30 servers or so now. > > > > > > Everything works like it should after the initial hang tough - no > matter how many times I reboot it completes the reboot cycle just fine. > > > > > > I?ve seen several people (mostly on IRC) mention this problem, but no > solution. > > > > > > Is anyone working on fixing this? > > > > I ran into this problem in spades when upgrading a set of servers from > > FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, > > but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung > > at the same place, after printing the "All buffers synced" message. > > > > I ultimately determined that if I did the following, rather than > > just a "reboot" or "shutdown -r now 'FreeBSD 9.1-RELEASE upgrade'", > > it would consistently AVOID the hang: > > > > sync ; sync ; sync ; shutdown -o -n -r now "FreeBSD 9.1 install" > > > > Your mileage may vary, but you don't have a lot to lose by trying it. > > > > -Kurt > > > > That is just bad advice. sync(1) does not g'tee that all data has been > written, no matter how many times you type it. shutdown -n tells the > system to abandon unwritten data. All in all, this is a recipe for > silent filesystem corruption. Using it after an update is just asking > to have a mix of old and new files on the system after the reboot. > > A more robust workaround would be to "mount -r" on all filesystems > before invoking the shutdown (even a shutdown -n should be safe after > everything has been remounted readonly). If the mount -r hangs on one > of the filesystems, then you've probably got a clue as to where a normal > shutdown is hanging. >FWIW mount -r on the root filesystem hangs for me. If I disable softupdates-journaling on the root filesystem before the upgrade process, the system no longer hangs on the last reboot after userland upgrade. However, the root filesystem still comes up dirty with an incorrect free block count during fsck.> -- Ian > > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >