openssh unix dev - Jun 2023 - Subsystem sftp invoked even though forced command created

Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically
Linux) may be at odds with the actual behavior I have seen in my experiments.
Here is the background:

I manage an application which collects data from a client script (Korn shell)
which runs on Unix and Linux servers across the entire enterprise. The client
communicates with a Linux server (currently running RHEL 7.9, reporting
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017) using a technique we call SSH
Pass-Through and described in an article in ;login: (Secure Automated File
Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the
original author of the client). The technique is predicated on using a forced
command, which we specify per key with the "command=" directive in the
authorized_keys file. (The user ID we use performs several functions, so we
cannot use ForceCommand in configuration.)

In theory, any SSH connection made to the account with the corresponding private
key should be forced to run the command provided, which accepts a
fully-qualified domain name, environment, and stream of XML data on standard
input and stores the data in a file with a name reflecting the FQDN and
environment along with a date stamp. In all documentation I have to hand, the
forced command is applied to all sessions including interactive,
non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key
embedded in the application can be extracted and used to launch a WinSCP session
against the user ID which the client uses to send the data to the server. This
session has the same access to all files on the server which the original user
ID does, including the ability to traverse directories. This should not happen
if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log
data to indicate whether the forced command is even executed, and if so, what
command line it was passed from the client (or WinSCP). The added code is as
follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
    print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
    print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and
received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no
further data in the log. This indicates to me that the sftp subsystem does not
even execute the forced command.

The subsystem configuration in sshd_config is 
# override default of no subsystems
Subsystem sftp  /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem,
but not with an external executable. Can anyone shed some light on this
situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus at att.com

On Thu, 29 Jun 2023, MCMANUS, MICHAEL P wrote:
> Folks,
> 
> I'm curious if the documented behavior of portable OpenSSH
(specifically Linux) may be at odds with the actual behavior I have seen in my
experiments. Here is the background:
[snip]

It's very hard to figure out what is happening here without a debug log.

You can get one by stopping the listening sshd and running it manually
in debug mode, e.g. "/usr/sbin/sshd -ddd"

Separate traces from expected vs problematic behaviour would be most
helpful.

-d

This is a new branch of an old thread, made necessary because the email system
here purges sent messages after a period of time so I can't reply to the
last message in the thread. The operative portion of that last message
(retrieved from the archives and dated July 3, 2023) follows:

/*****/
So I set up a fresh key to use for this test, and gave it similar parameters.

I wasn't aware of the "restrict" argument but the original
authorized_keys entries had a long list of restrictions:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/.../receive.ksh"
ssh-rsa ...

I did the same thing as I had done with the live key and got the same results
(as expected).

It then occurred to me that the issue is probably on the server side, which did
not change. So I set up the user and forced command on 3 lab servers:
OS		OpenSSH version
Ubuntu 18.04	OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n  7 Dec 2017
Solaris 11	OpenSSH_8.4p1, OpenSSL 1.0.2zf  21 Jun 2022
HP-UX 11.31	OpenSSH_8.1p1+sftpfilecontrol-v1.3-hpn14v20, OpenSSL 1.1.1d  10 Sep
2019

I got the following results:
OS		ssh					sftp					SSH_ORIGINAL_COMMAND
Ubuntu 18.04	Hung waiting for input, pressed ^C	Obtained sftp prompt. 			Not
logged
Solaris 11	Hung waiting for input, pressed ^C	Hung waiting for input, pressed ^C
Command: internal-sftp
HP-UX 11.31	Hung waiting for input, pressed ^C	Hung waiting for input, pressed
^C	Command: /opt/ssh/libexec/sftp-server

So the issue looks like it happens across Linux distributions, but not on
non-Linux. Curiouser and curiouser.

I then changed the forced command to /usr/bin/logger and got similar results. I
think the default priority for logs on our servers goes nowhere; next round
I'll have to specify a priority that is actually logged.
/*****/

The conclusion I came to was that the Linux distributors are working with a
modified version of the Berkeley sources (they have to, in order to back-port
security fixes like they do). Therefore I downloaded OpenSSH 9.4p1 sources from
the Berkeley depot and compiled them on a Rocky Linux 8 server.

Sadly, the problem remains. I noticed in the logs there were references to Red
Hat specific configuration, but that was on the client side and not the server
side. I'm beginning to think that there is something in the configuration of
OpenSSH on Linux servers which prioritizes the sftp subsystem over the forced
command, whether the binaries are downloaded from the OS distributor or built
from sources.

Unless I can get some expert advice, I believe this kills off the use of SSH as
a quick and secure transport for unattended file transfer and ingestion by
applications, at least on Linux. It will have to be replaced by TLS or the
Splunk Universal Agent. Pity, that.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus at att.com  

-----Original Message-----
From: openssh-unix-dev <openssh-unix-dev-bounces+mm1072=att.com at
mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Thursday, June 29, 2023 3:06 PM
To: openssh-unix-dev at mindrot.org
Subject: Subsystem sftp invoked even though forced command created

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically
Linux) may be at odds with the actual behavior I have seen in my experiments.
Here is the background:

I manage an application which collects data from a client script (Korn shell)
which runs on Unix and Linux servers across the entire enterprise. The client
communicates with a Linux server (currently running RHEL 7.9, reporting
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017) using a technique we call SSH
Pass-Through and described in an article in ;login: (Secure Automated File
Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the
original author of the client). The technique is predicated on using a forced
command, which we specify per key with the "command=" directive in the
authorized_keys file. (The user ID we use performs several functions, so we
cannot use ForceCommand in configuration.)

In theory, any SSH connection made to the account with the corresponding private
key should be forced to run the command provided, which accepts a
fully-qualified domain name, environment, and stream of XML data on standard
input and stores the data in a file with a name reflecting the FQDN and
environment along with a date stamp. In all documentation I have to hand, the
forced command is applied to all sessions including interactive,
non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key
embedded in the application can be extracted and used to launch a WinSCP session
against the user ID which the client uses to send the data to the server. This
session has the same access to all files on the server which the original user
ID does, including the ability to traverse directories. This should not happen
if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log
data to indicate whether the forced command is even executed, and if so, what
command line it was passed from the client (or WinSCP). The added code is as
follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
    print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
    print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and
received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no
further data in the log. This indicates to me that the sftp subsystem does not
even execute the forced command.

The subsystem configuration in sshd_config is 
# override default of no subsystems
Subsystem sftp  /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem,
but not with an external executable. Can anyone shed some light on this
situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus at att.com  

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev at mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!gAfqyQFKV_nTGhaz-PwBqHZat18A67O51MZnz1_jFaordOKkpGy9dSyhdHYW0dqNiT5SUd2Mh6M$