openssh unix dev - Oct 2021 - Howto log multiple sftpd instances with their chroot shared via NFS

Hello all, first let me thank you all for your input and suggestions.

I think I have found now a workaround for the problem.
> Since we have 800 users, it would be impractical unrobust to use
user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple,
clear and coherent, all user's homes must be on the same one singular
NFS-Share.
It seems, that I cannot uphold this requirement of not having user-specific
bind-over-mounts. (while the requirement that there is only exactly one nfs
mount is still true).

My workaround is the following:

sudo mkdir /var/data/dev    # Create a directroy under which user subdirectories
are created

For very username <username> and the user's primary group
<groupname> do the following:

sudo mkdir             /var/data/dev/<username>
sudo chmod 550         /var/data/dev/<username>       # This restrictive
permission is a requirement I think
sudo chgrp <groupname> /var/data/dev/<username>       # so the user
can read the directory

So the new directory is exactly the same as the existing
/var/data/chroot/<username>/dev directory (which is on the nfs mount
/var/data/chroot/).

Then do
mount --bind /var/data/dev/<username>    
/var/data/chroot/<username>/dev

so /var/data/chroot/<username>/dev is now effectively local on the sftp
server, not anymore on nfs mount.

Then change the syslog-ng config

from
source s_chroot_<username>     {
unix-stream("/var/data/chroot/<username>/dev/log" optional(yes)
); };

to
source s_chroot_<username>     {
unix-stream("/var/data/dev/<username>/log" optional(yes) ); };

(this is not strictly needed, but I think it's nice having syslog-ng
definitely now only reading from local file, guaranteed not from nfs mount
anymore)

I have tested this successfully with one user so far - whether the user logs in
on the one sftp server or the other, syslog-ng can now log the sftp session on
the affected sftp server.

While it is still a mess to have 800 (or in the future maybe 2000 or 3000) bind
mounts which I very much would like to avoid, I think this workaround is
acceptable because the impact is limited to the logging functionality and not
the sftp service itself. Speaking, if there are problems with the bind mounts,
the impact is only that there is no sftp session logging.
Also, this is very simple, uniform and clear.
And one has 100% reliable sftp session logging.
No need to change anything else of the well established production system.

I think I will create systemd unit files for every bind mount (I experienced
systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18 boot before
anyway, so systemd units are needed anyway).
Maybe one can define a systemd dependency so that when you unmount the nfs mount
/var/data/chroot/, all the bind mounts under it are automatically unmounted
before (sftpd needs to be stopped first of course, otherwise mount would be
busy), and also the bind-mounts automatically be mounted after /var/dat/chroot
is mounted again.

So there should not be much hassle with all those simple uniform bind mounts.

I think I will implement a nagios monitoring check that checks if for every
username /var/data/chroot/<username>/dev is a mountpoint, so logging is
assured.

According to
https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860
the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be
a problem.

--
Possible alternative
I guess (not tested) an alternative could be to hardlink (so needs to be on the
same (nfs) file system) every /var/data/chroot/<username>/dev/log to e.g.
/var/data/chroot/general_dev_log  (which would need access permission for all
users of course)
and do
mount --bind /var/data/general_dev_log /var/data/chroot/general_dev_log

so every sftp session would log to the one and only log device (unix socket),
again using the local bind mount.
So you could write all the sftp session logs of all users into one log file with
syslog-ng.
Advantage:
- only one bind mount needed
- only one syslog-ng unix-stream configuration needed

Disadvantage:
You need to parse and filter the big one session log to create user-specific log
files which only conatin the sessions for that user.
I have no experience with such tools and I think this filtering can not be 100%
reliable, there will always be some cases where the filter will fail and log
entries will be missing.

Best regards

Hi Hildegard,

On 1/10/21 2:44 pm, Hildegard Meier wrote:
> mount
--bind/var/data/dev/<username>/var/data/chroot/<username>/dev
>
> so/var/data/chroot/<username>/dev is now effectively local on the
sftp server, not anymore on nfs mount.
That's not right.? This doesn't magically turn an NFS mount into a local
mount.? If /var/data/chroot/<username>/dev is NFS mounted, 
/var/data/dev/<username> is also NFS mounted.

I believe I explained what you need to do but for some reason you didn't 
like it, and when I asked why, you never replied.

To repeat myself:

1. Create /var/data/chroot/dev, /var/data/chroot/home and 
/var/data/chroot/lib on both machines.

2. Configure your syslog daemon to read from /var/data/chroot/dev/log.

3. Configure SSH to chroot for the SFTP users to /var/data/chroot.

4. Mount your current NFS share which contains all 800 user directories 
over /var/data/chroot/home.

For extra points, instead of step 4, although I think is the bit you 
didn't like, while at the same time I think is the bit that you said you 
really want, so there's cognitive dissonance for you:

4. Mount a NFS share which contains empty directories for all 800 users 
over /var/data/chroot/home.

5. When a user logs in use automount to NSF mount their home directory 
over /var/data/chroot/home/<username>.

This is not hard.

Regards,

David

Can confirm this is working in production successfully now.
> Gesendet: Freitag, 01. Oktober 2021 um 06:14 Uhr
> Von: "Hildegard Meier" <daku8938 at gmx.de>
> Cc: openssh-unix-dev at mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared
via NFS
>
> My workaround is the following:
>
> sudo mkdir /var/data/dev    # Create a directroy under which user
subdirectories are created
>
> For very username <username> and the user's primary group
<groupname> do the following:
>
> sudo mkdir             /var/data/dev/<username>
> sudo chmod 550         /var/data/dev/<username>       # This
restrictive permission is a requirement I think
> sudo chgrp <groupname> /var/data/dev/<username>       # so the
user can read the directory
>
> So the new directory is exactly the same as the existing
/var/data/chroot/<username>/dev directory (which is on the nfs mount
/var/data/chroot/).
>
> Then do
> mount --bind /var/data/dev/<username>    
/var/data/chroot/<username>/dev
>
> so /var/data/chroot/<username>/dev is now effectively local on the
sftp server, not anymore on nfs mount.
>
> Then change the syslog-ng config
>
> from
> source s_chroot_<username>     {
unix-stream("/var/data/chroot/<username>/dev/log" optional(yes)
); };
>
> to
> source s_chroot_<username>     {
unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
>
> (this is not strictly needed, but I think it's nice having syslog-ng
definitely now only reading from local file, guaranteed not from nfs mount
anymore)
>
> I have tested this successfully with one user so far - whether the user
logs in on the one sftp server or the other, syslog-ng can now log the sftp
session on the affected sftp server.
>
> While it is still a mess to have 800 (or in the future maybe 2000 or 3000)
bind mounts which I very much would like to avoid, I think this workaround is
acceptable because the impact is limited to the logging functionality and not
the sftp service itself. Speaking, if there are problems with the bind mounts,
the impact is only that there is no sftp session logging.
> Also, this is very simple, uniform and clear.
> And one has 100% reliable sftp session logging.
> No need to change anything else of the well established production system.
>
> I think I will create systemd unit files for every bind mount (I
experienced systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18
boot before anyway, so systemd units are needed anyway).
> Maybe one can define a systemd dependency so that when you unmount the nfs
mount /var/data/chroot/, all the bind mounts under it are automatically
unmounted before (sftpd needs to be stopped first of course, otherwise mount
would be busy), and also the bind-mounts automatically be mounted after
/var/dat/chroot is mounted again.
>
> So there should not be much hassle with all those simple uniform bind
mounts.
>
> I think I will implement a nagios monitoring check that checks if for every
username /var/data/chroot/<username>/dev is a mountpoint, so logging is
assured.
>
> According to
https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860
the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be
a problem.