Ján Lalinský
2019-Nov-26 23:50 UTC
[CentOS] systemd: Failed unmounting /var on reboot, should I worry about fs corruption?
Hi all, I have Centos 8 installed on a physical machine (www6) with separate LVM volumes for /, /var, /var/lib/mysql etc. System boot proceeds without a hiccup, in terminal systemctl status says everything is OK and running, journalctl says so as well - systemd mounts everything stated in fstab. However, on reboot systemd echoes problems with filesystem on /var : ...// unmounting all volumes Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. Nov 26 23:51:30 www6 systemd[1]: Deactivating swap /dev/disk/by-label/lv_swap... Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, code=exited status=32 Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var. Then proceeds and reboots the machine. This occurs on every reboot. Did anybody here encounter similar problems on reboot? I found this bug https://github.com/systemd/systemd/issues/867 which was closed without a definitive solution, the developer says the issue is hard to solve and just cosmetic since /var gets unmounted in the end anyway, which is strange because there is a log line about unmounting just about every other mountpoint and in the end of the journal, there are about 7 log lines about unmounting swap volume, but zilch about /var. If the system is getting down without proper unmount, data corruption can happen. I would feel much better if there were no such errors... My /etc/fstab: /dev/mapper/vg_www6-lv_root??????????????? /?????????????? xfs?? defaults,noatime 0? 1 /dev/mapper/vg_www6-lv_var???????????????? /var??????????? xfs?? defaults,noatime 0? 0 /dev/mapper/vg_www6-lv_swap??????????????? swap??????????? swap? defaults???????? 0? 0 /dev/mapper/vg_www6-lv_tmp???????????????? /tmp??????????? xfs?? defaults,noatime 0? 0 /dev/mapper/vg_www6-lv_usr_local?????????? /usr/local????? xfs?? defaults,noatime 0? 0 UUID=XXX?????????????????????????????????? /boot?????????? xfs?? defaults???????? 0? 0 #/dev/mapper/vg_www6-lv_tmp_mysql????????? /tmp_mysql????? xfs?? defaults,noatime 0? 0 tmpfs????????????????????????????????????? /tmp_mysql????? tmpfs defaults,noatime,nosuid,nodev,noexec,mode=0755,size=5G 0? 0 /dev/mapper/vg_www6-lv_var_lib_mysql?????? /var/lib/mysql? xfs?? defaults,noatime,uquota,gquota 0? 0 /dev/mapper/vg_www6-lv_data??????????????? /data?????????? xfs?? defaults,noatime,uquota,gquota 0? 0 -- S pozdravem J?n Lalinsk? Webhosting C4 ?ESK? WEBHOSTING s.r.o. Tel: +420 234 139 876 E-mail: info at c4.cz http://www.c4.cz
Tony Mountifield
2019-Nov-27 10:35 UTC
[CentOS] systemd: Failed unmounting /var on reboot, should I worry about fs corruption?
In article <db155fbf-7c80-f89f-d401-fcc1097dfe76 at c4.cz>, J??n Lalinsk?? <lalinsky at c4.cz> wrote:> Hi all, > > I have Centos 8 installed on a physical machine (www6) with separate LVM > volumes for /, /var, /var/lib/mysql etc. > > System boot proceeds without a hiccup, in terminal systemctl status says > everything is OK and running, journalctl says so as well - systemd > mounts everything stated in fstab. > > However, on reboot systemd echoes problems with filesystem on /var : > > ...// unmounting all volumes > > Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... > Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. > Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. > Nov 26 23:51:30 www6 systemd[1]: Deactivating swap > /dev/disk/by-label/lv_swap... > Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, > code=exited status=32 > Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var. > > Then proceeds and reboots the machine. This occurs on every reboot. > > Did anybody here encounter similar problems on reboot? > > I found this bug > > https://github.com/systemd/systemd/issues/867 > > which was closed without a definitive solution, the developer says the > issue is hard to solve and just cosmetic since /var gets unmounted in > the end anyway, which is strange because there is a log line about > unmounting just about every other mountpoint and in the end of the > journal, there are about 7 log lines about unmounting swap volume, but > zilch about /var. If the system is getting down without proper unmount, > data corruption can happen. I would feel much better if there were no > such errors...You need to un-hide and read the hidden comments on that bug. They give a few different workarounds, but also indicate that the issue has indeed been solved in a later version of systemd. But that was only in April of this year; I don't know which version of upstream systemd was used in RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of the workarounds such as lazy unmount for /var ought to work in the meantime. Cheers Tony -- Tony Mountifield Work: tony at softins.co.uk - http://www.softins.co.uk Play: tony at mountifield.org - http://tony.mountifield.org
Ján Lalinský
2019-Nov-30 00:25 UTC
[CentOS] systemd: Failed unmounting /var on reboot, should I worry about fs corruption?
Thanks, after reading some more complaints online and suggestions on this, I think setting up a lazy umount of /var is the best option until the bug is resolved in Centos 8. The lazy umount prevents systemd from registering and reporting the error that ordinary umount /var would generate (such ordinary umount of /var by systemd is a misconceived action, as /var is still in use, at least by journald). For others struggling with this mess, here is the procedure that worked for me: 1. Comment out mountpoint /var in /etc/fstab 2. Create a file /etc/systemd/system/var.mount with these lines: [Unit] Description=Lazy mount file for /var to work around systemd bug 867 DefaultDependencies=no Conflicts=umount.target Requires=system.slice -.mount Before=local-fs.target umount.target After=swap.target [Mount] What=/dev/disk/by-uuid/<var-filesystem-uuid> Where=/var Type=xfs Options=defaults,noatime LazyUnmount=true [Install] WantedBy=multi-user.target 3. Then run systemctl daemon-reload and that should be it, subsequent reboots should go nice and clean, hopefully without this bug rearing its head again. Best regards J?n Lalinsk? Webhosting C4 ?ESK? WEBHOSTING s.r.o. Tel: +420 234 139 876 E-mail: info at c4.cz http://www.c4.cz On 27/11/2019 11:35, Tony Mountifield wrote:> In article <db155fbf-7c80-f89f-d401-fcc1097dfe76 at c4.cz>, > J?n Lalinsk? <lalinsky at c4.cz> wrote: >> Hi all, >> >> I have Centos 8 installed on a physical machine (www6) with separate LVM >> volumes for /, /var, /var/lib/mysql etc. >> >> System boot proceeds without a hiccup, in terminal systemctl status says >> everything is OK and running, journalctl says so as well - systemd >> mounts everything stated in fstab. >> >> However, on reboot systemd echoes problems with filesystem on /var : >> >> ...// unmounting all volumes >> >> Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... >> Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. >> Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. >> Nov 26 23:51:30 www6 systemd[1]: Deactivating swap >> /dev/disk/by-label/lv_swap... >> Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, >> code=exited status=32 >> Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var. >> >> Then proceeds and reboots the machine. This occurs on every reboot. >> >> Did anybody here encounter similar problems on reboot? >> >> I found this bug >> >> https://github.com/systemd/systemd/issues/867 >> >> which was closed without a definitive solution, the developer says the >> issue is hard to solve and just cosmetic since /var gets unmounted in >> the end anyway, which is strange because there is a log line about >> unmounting just about every other mountpoint and in the end of the >> journal, there are about 7 log lines about unmounting swap volume, but >> zilch about /var. If the system is getting down without proper unmount, >> data corruption can happen. I would feel much better if there were no >> such errors... > You need to un-hide and read the hidden comments on that bug. They give > a few different workarounds, but also indicate that the issue has indeed > been solved in a later version of systemd. But that was only in April of > this year; I don't know which version of upstream systemd was used in > RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of > the workarounds such as lazy unmount for /var ought to work in the > meantime. > > Cheers > Tony > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos