CentOS - Nov 2019 - systemd: Failed unmounting /var on reboot, should I worry about fs corruption?

Hi all,

I have Centos 8 installed on a physical machine (www6) with separate LVM
volumes for /, /var, /var/lib/mysql etc.

System boot proceeds without a hiccup, in terminal systemctl status says
everything is OK and running, journalctl says so as well - systemd
mounts everything stated in fstab.

However, on reboot systemd echoes problems with filesystem on /var :

...// unmounting all volumes

Nov 26 23:51:30 www6 systemd[1]: Unmounting /var...
Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy.
Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap.
Nov 26 23:51:30 www6 systemd[1]: Deactivating swap
/dev/disk/by-label/lv_swap...
Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited,
code=exited status=32
Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.

Then proceeds and reboots the machine. This occurs on every reboot.

Did anybody here encounter similar problems on reboot?

I found this bug

https://github.com/systemd/systemd/issues/867

which was closed without a definitive solution, the developer says the
issue is hard to solve and just cosmetic since /var gets unmounted in
the end anyway, which is strange because there is a log line about
unmounting just about every other mountpoint and in the end of the
journal, there are about 7 log lines about unmounting swap volume, but
zilch about /var. If the system is getting down without proper unmount,
data corruption can happen. I would feel much better if there were no
such errors...


My /etc/fstab:

/dev/mapper/vg_www6-lv_root??????????????? /?????????????? xfs??
defaults,noatime 0? 1
/dev/mapper/vg_www6-lv_var???????????????? /var??????????? xfs??
defaults,noatime 0? 0
/dev/mapper/vg_www6-lv_swap??????????????? swap??????????? swap?
defaults???????? 0? 0
/dev/mapper/vg_www6-lv_tmp???????????????? /tmp??????????? xfs??
defaults,noatime 0? 0
/dev/mapper/vg_www6-lv_usr_local?????????? /usr/local????? xfs??
defaults,noatime 0? 0
UUID=XXX?????????????????????????????????? /boot?????????? xfs??
defaults???????? 0? 0
#/dev/mapper/vg_www6-lv_tmp_mysql????????? /tmp_mysql????? xfs??
defaults,noatime 0? 0
tmpfs????????????????????????????????????? /tmp_mysql????? tmpfs
defaults,noatime,nosuid,nodev,noexec,mode=0755,size=5G 0? 0
/dev/mapper/vg_www6-lv_var_lib_mysql?????? /var/lib/mysql? xfs??
defaults,noatime,uquota,gquota 0? 0
/dev/mapper/vg_www6-lv_data??????????????? /data?????????? xfs??
defaults,noatime,uquota,gquota 0? 0

-- 
S pozdravem
J?n Lalinsk?

Webhosting C4
?ESK? WEBHOSTING s.r.o.
Tel: +420 234 139 876
E-mail: info at c4.cz
http://www.c4.cz

In article <db155fbf-7c80-f89f-d401-fcc1097dfe76 at c4.cz>,
J??n Lalinsk?? <lalinsky at c4.cz> wrote:
> Hi all,
> 
> I have Centos 8 installed on a physical machine (www6) with separate LVM
> volumes for /, /var, /var/lib/mysql etc.
> 
> System boot proceeds without a hiccup, in terminal systemctl status says
> everything is OK and running, journalctl says so as well - systemd
> mounts everything stated in fstab.
> 
> However, on reboot systemd echoes problems with filesystem on /var :
> 
> ...// unmounting all volumes
> 
> Nov 26 23:51:30 www6 systemd[1]: Unmounting /var...
> Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy.
> Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap.
> Nov 26 23:51:30 www6 systemd[1]: Deactivating swap
> /dev/disk/by-label/lv_swap...
> Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited,
> code=exited status=32
> Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.
> 
> Then proceeds and reboots the machine. This occurs on every reboot.
> 
> Did anybody here encounter similar problems on reboot?
> 
> I found this bug
> 
> https://github.com/systemd/systemd/issues/867
> 
> which was closed without a definitive solution, the developer says the
> issue is hard to solve and just cosmetic since /var gets unmounted in
> the end anyway, which is strange because there is a log line about
> unmounting just about every other mountpoint and in the end of the
> journal, there are about 7 log lines about unmounting swap volume, but
> zilch about /var. If the system is getting down without proper unmount,
> data corruption can happen. I would feel much better if there were no
> such errors...
You need to un-hide and read the hidden comments on that bug. They give
a few different workarounds, but also indicate that the issue has indeed
been solved in a later version of systemd. But that was only in April of
this year; I don't know which version of upstream systemd was used in
RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of
the workarounds such as lazy unmount for /var ought to work in the
meantime.

Cheers
Tony

-- 
Tony Mountifield
Work: tony at softins.co.uk - http://www.softins.co.uk
Play: tony at mountifield.org - http://tony.mountifield.org

Thanks, after reading some more complaints online and suggestions on
this, I think setting up a lazy umount of /var is the best option until
the bug is resolved in Centos 8. The lazy umount prevents systemd from
registering and reporting the error that ordinary umount /var would
generate (such ordinary umount of /var by systemd is a misconceived
action, as /var is still in use, at least by journald).

For others struggling with this mess, here is the procedure that worked
for me:

1. Comment out mountpoint /var in /etc/fstab

2. Create a file /etc/systemd/system/var.mount with these lines:

[Unit]
Description=Lazy mount file for /var to work around systemd bug 867
DefaultDependencies=no
Conflicts=umount.target
Requires=system.slice -.mount
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=/dev/disk/by-uuid/<var-filesystem-uuid>
Where=/var
Type=xfs
Options=defaults,noatime
LazyUnmount=true

[Install]
WantedBy=multi-user.target

3. Then run

systemctl daemon-reload

and that should be it, subsequent reboots should go nice and clean,
hopefully without this bug rearing its head again.


Best regards
J?n Lalinsk?

Webhosting C4
?ESK? WEBHOSTING s.r.o.
Tel: +420 234 139 876
E-mail: info at c4.cz
http://www.c4.cz

On 27/11/2019 11:35, Tony Mountifield wrote:
> In article <db155fbf-7c80-f89f-d401-fcc1097dfe76 at c4.cz>,
> J?n Lalinsk? <lalinsky at c4.cz> wrote:
>> Hi all,
>>
>> I have Centos 8 installed on a physical machine (www6) with separate
LVM
>> volumes for /, /var, /var/lib/mysql etc.
>>
>> System boot proceeds without a hiccup, in terminal systemctl status
says
>> everything is OK and running, journalctl says so as well - systemd
>> mounts everything stated in fstab.
>>
>> However, on reboot systemd echoes problems with filesystem on /var :
>>
>> ...// unmounting all volumes
>>
>> Nov 26 23:51:30 www6 systemd[1]: Unmounting /var...
>> Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy.
>> Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap.
>> Nov 26 23:51:30 www6 systemd[1]: Deactivating swap
>> /dev/disk/by-label/lv_swap...
>> Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited,
>> code=exited status=32
>> Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.
>>
>> Then proceeds and reboots the machine. This occurs on every reboot.
>>
>> Did anybody here encounter similar problems on reboot?
>>
>> I found this bug
>>
>> https://github.com/systemd/systemd/issues/867
>>
>> which was closed without a definitive solution, the developer says the
>> issue is hard to solve and just cosmetic since /var gets unmounted in
>> the end anyway, which is strange because there is a log line about
>> unmounting just about every other mountpoint and in the end of the
>> journal, there are about 7 log lines about unmounting swap volume, but
>> zilch about /var. If the system is getting down without proper unmount,
>> data corruption can happen. I would feel much better if there were no
>> such errors...
> You need to un-hide and read the hidden comments on that bug. They give
> a few different workarounds, but also indicate that the issue has indeed
> been solved in a later version of systemd. But that was only in April of
> this year; I don't know which version of upstream systemd was used in
> RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of
> the workarounds such as lazy unmount for /var ought to work in the
> meantime.
>
> Cheers
> Tony
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos