openssh unix dev - Sep 2021 - Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS

Jochen Bern wrote:
> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I 
> have no idea how portable it is to other distribs, so just give it a try:
> 
> > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > Subsystem       sftp    internal-sftp
> > Match group mandanten
> >         ForceCommand internal-sftp -l INFO -u 0077
> 
> 
> > # ls -al ~lvinq4/dev
> > insgesamt 0
> > drwx--x---. 2 root mandanten  6 20. Mai 17:25 .
> > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
> 
> As you can see, the chroots have an *empty* /dev subdir, but logging 
> *still works*, apparently because the chrooted process just keeps using 
> the system-central /dev/log it opened *before* chroot()ing.
I think that works specifically because *no* new process is created
when using internal-sftp as opposed to executing the sftp-server binary.

If syslog-ng can be made to reliably route internal-sftp messages to
user-specific log files then this approach would work well in Hildegard's
setup and would be a lot more pleasant than messing with LD_PRELOAD.


//Peter

On 9/26/2021 3:20 PM, Peter Stuge wrote:
> Jochen Bern wrote:
>> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
>> have no idea how portable it is to other distribs, so just give it a
try:
>>
>>> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
>>> Subsystem       sftp    internal-sftp
>>> Match group mandanten
>>>          ForceCommand internal-sftp -l INFO -u 0077
>>
>>
>>> # ls -al ~lvinq4/dev
>>> insgesamt 0
>>> drwx--x---. 2 root mandanten  6 20. Mai 17:25 .
>>> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>>
>> As you can see, the chroots have an *empty* /dev subdir, but logging
>> *still works*, apparently because the chrooted process just keeps using
>> the system-central /dev/log it opened *before* chroot()ing.
> 
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
> 
It is not clear from Hildegard's email what is in the user's home
directory
and if the sftp server can run in interactive mode. i.e. if a shell is started,
and if it can run any commands that may require logging.

OpenSSH sshd_config man page discusses ChrootDirectory says: "For an
interactive session this requires at least a shell"
and also says: "sessions which use logging may require /dev/log inside the
chroot directory on some operating systems."

So I would say the /dev/log or logs are required.

Based on the problem in original note:
> So, if a user logs in on the first server, where syslog-ng was started
least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not
logged, neither on the second nor on the first server.
> 
> If the syslog-ng is then restarted on the second server, the sftp
user's activity is exclusively logged only on the second server and only for
logins on the second server.
Logging works on the server that started syslog-ng. It sounds like syslog-ng
opens a "unix-stream" socket.
She also says: "for every user has a log device
/var/data/chroot/<username>/dev/log"
(not sure what it changes, it may save the socket handle/index (or whatever it
is) to be used in that server's kernel.)

Each server is overwriting the /dev/log in the chroot directories.

So if interactive is not required, then no chroot /dev/log would be needed and
sftp-initeral could work.
But it sounds like there is a need to share these NFS volumes

So we are back to using bind-mounting as it is done in the kernel and NFS
version of /dev/log is not touched.

OR

Each server has its own /dev/log_<hostname> and the syslog-ng files on
each server do the same
and use LD_PRELOAD= as proposed by Peter Stuge. I have used the LD_PRELOAD
before when running
valgrind and/or gdb to not unload modules so as the debug tables stay available,

> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in
Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD. 
As noted above, depends if any other process is stared that does logging is
started after the
change to chroot.
> 
> 
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> 
-- 

  Douglas E. Engert  <DEEngert at gmail.com>

Jochen, are you sure that you see the real sftp user session detailed activity
log, e.g.

internal-sftp[27918]: session opened for local user <username> from
[1.2.3.4]
internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE
mode 0666

etc. and not just the sshd auth log, e.g.

sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504
ssh2

? With Ubuntu 18.04.1 LTS and it's shipped OpenSSH 7.6p1-4ubuntu0.5 I can
not get any sftp session logging when having only an empty /dev directory under
the user's chroot directory, without it's "log" file.

I wonder if it would be a bug or a feature if you can manage to get sftp session
logging without /dev/log in the sftp user's chroot dir?
What CentOS and OpenSSH version do you have exactly?
What could be the difference of the CentOS OpenSSH? Do you have special starting
options?

Peter,
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
For every sftp subsystem login (here with user "sftp_nagios"), I see a
new sftpd process created:

First login:

ps auxww |grep sftp
root      4192  0.0  0.1  72304  6512 ?        Ss   11:01   0:00 /usr/sbin/sftpd
-D -f /etc/sftpd/sftpd_config
root      4590  0.2  0.1  74736  6632 ?        Ss   11:05   0:00 sftpd:
sftp_nagios [priv]
sftp_na+  4592  0.0  0.0  74736  3432 ?        S    11:05   0:00 sftpd:
sftp_nagios at notty
sftp_na+  4593  0.0  0.0  74736  3108 ?        Ss   11:05   0:00 sftpd:
sftp_nagios at internal-sftp

Then logout and login again, second login:

ps auxww |grep sftp
root      4192  0.0  0.1  72304  6512 ?        Ss   11:01   0:00 /usr/sbin/sftpd
-D -f /etc/sftpd/sftpd_config
root      4630  0.5  0.1  74736  6596 ?        Ss   11:05   0:00 sftpd:
sftp_nagios [priv]
sftp_na+  4632  0.0  0.0  74736  3552 ?        S    11:05   0:00 sftpd:
sftp_nagios at notty
sftp_na+  4633  0.0  0.0  74736  3164 ?        Ss   11:05   0:00 sftpd:
sftp_nagios at internal-sftp

> Gesendet: Sonntag, 26. September 2021 um 22:20 Uhr
> Von: "Peter Stuge" <peter at stuge.se>
> An: openssh-unix-dev at mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared
via NFS
>
> Jochen Bern wrote:
> > OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> > have no idea how portable it is to other distribs, so just give it a
try:
> >
> > > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > > Subsystem       sftp    internal-sftp
> > > Match group mandanten
> > >         ForceCommand internal-sftp -l INFO -u 0077
> >
> >
> > > # ls -al ~lvinq4/dev
> > > insgesamt 0
> > > drwx--x---. 2 root mandanten  6 20. Mai 17:25 .
> > > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
> >
> > As you can see, the chroots have an *empty* /dev subdir, but logging
> > *still works*, apparently because the chrooted process just keeps
using
> > the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>
> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in
Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>