On Thu, May 13, 2021 at 02:15:15PM +0000, James Pearson
wrote:>
> I have a CentOS 7 system where I needed to restart chronyd - but the
> systemctl restart failed with the error:
>
> systemd[1]: Starting NTP client/server...
> systemd[43578]: Failed at step NAMESPACE spawning /usr/sbin/chronyd: Stale
file handle
> systemd[1]: chronyd.service: control process exited, code=exited
status=226
>
> Turns out there are a couple of Stale NFS file handles from fuse
> mounts (related to gvfsd) of sub directories under an NFS mounted
> home directory server - but the home directory for the user in this
> case, no longer exist (user has left)
>
> However, I have no idea why these 'Stale file handles' prevent a
> service being started by systemd ?
>
> In this case, chronyd has nothing to do with NFS mounted user home
> directories - so shouldn't really care ?
>
> I have tried everything I can think of to clear these stale mounts,
> but with no luck
>
> Does anyone know why systemd complains about unconnected 'Stale file
> handles' - and is there any way I can tell systemctl to start a
> service regardless of these 'errors' ?
>
> Rebooting the host will be a last resort (the system is used by many
> users) - but in the meantime, I've manually started the
> /usr/sbin/chronyd binary directly, which runs fine
So, the chronyd systemd unit looks like this:
# /usr/lib/systemd/system/chronyd.service
[Unit]
Description=NTP client/server
Documentation=man:chronyd(8) man:chrony.conf(5)
After=ntpdate.service sntp.service ntpd.service
Conflicts=ntpd.service systemd-timesyncd.service
ConditionCapability=CAP_SYS_TIME
[Service]
Type=forking
PIDFile=/var/run/chrony/chronyd.pid
EnvironmentFile=-/etc/sysconfig/chronyd
ExecStart=/usr/sbin/chronyd $OPTIONS
ExecStartPost=/usr/libexec/chrony-helper update-daemon
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
[Install]
WantedBy=multi-user.target
So, you'll notice there are "ProtectHome=yes" and
"ProtectSystem=yes"
settings in the Service section. This sets up a private namespace for
the systemd unit so /home, /root and /run/user are made inaccessible
and empty (ProtectHome), and /usr, /boot and /etc are read-only
(ProtectSystem). It does this to reduce the ability of a malicious
NTP server attacking the system through bogus NTP traffic (which is a
real thing that can happen). Many systemd services limit their
processes this way.
I suspect that is why you're seeing stale file handle errors, the
kernel can't set up the namespace for directories that are now stale
on the system.
You can probably just do a lazy unmount (umount -l) to make them go
away until you reboot. You can also disable the namespaced
directories by doing a 'systemctl edit chronyd.service' and setting
the options to 'off', but you'll be reducing the security of your
system.
We've seen some weird stuff in the past related to this feature. For
example, I couldn't unmount /home because a service with
ProtectHome=read-only was running (cups), and 'fuser' and 'lsof'
didn't show anything was using it. It's because the kernel namespace
stuff operates as a mountpoint, so it's all kernel. Another fun issue
I discovered is that we had some locally-developed services that used
files in /tmp as a communication channel, and with PrivateTmp=yes set,
they no longer could communicate. So it forced us to actually do the
right thing and use more appropriate methods.
It is kinda confusing but I do appreciate that I now have a lot of
ways I can now lock down services beyond simple UNIX
permissions. systemd is a rather neat init system. My complaints with
it usually are with the parts that reach outside of being an init
system (I'm looking at you, systemd-logind and systemd-resolved).
--
Jonathan Billings <billings at negate.org>