openssh unix dev - Jul 2002 - scp not tolerant of extraneous shell messages

I spent some time debugging a failing Scp that turned out to be caused
by the remote shell producing messages on Standard Output that aren't
part of the SCP protocol. 

Scp from a remote system works by running another Scp on the remote
system.  The remote Scp writes SCP protocol messages (and file data)
to its Standard Output, which the local Scp sees as Standard Input.

But it's important to note that the remote Scp runs in a shell, and
not just any shell -- an interactive shell.  It's an interactive shell
because its Standard Input and Standard Output are terminals, and the
shell knows it and makes some decisions based on the assumption that
there's a human being there.  This makes Scp analogous to the file
download function of an old-fashioned terminal emulator -- it's making
automated use of a data stream intended for an intelligent human.

So lets say the remote interactive shell puts an informational message
on Standard Output as it starts up (mine does).  The local SCP
responds by issuing the received line verbatim, with no additional
context, as an error message, and terminates.  It apparently assumes
that anything the shell says before remote Scp starts running must be
a message explaining why it cannot run Scp.  And that the message
contains enough context that the Scp user will be able to tell what to
fix and where.

But the message may just be informational, in which case Scp terminates
needlessly, and with no apparent indication of why.


I propose that local Scp just ignore all messages on Standard Input until
it sees something that looks like an SCP control record.  If it never sees
an SCP control record, it should issue an error message saying so, and
include the last thing it _did_ see in that message.

If there isn't something fundamentally wrong with this approach, I'll
submit a patch.


Ideally, I'd like to see a way to run remote Scp not in a shell at
all.  It would be far more robust and faster.  Does the SSH protocol
allow for such a thing?

-- 
Bryan Henderson                                    Phone 408-621-2000
San Jose, California

You are really looking for sftp which has no shell dependancies.

But in general. No.. Fix your startup scripts.  scp is rcp in a ssh
wrapper.  and rcp will also fail.

- Ben

On Mon, 1 Jul 2002, Bryan Henderson wrote:
> I spent some time debugging a failing Scp that turned out to be caused
> by the remote shell producing messages on Standard Output that aren't
> part of the SCP protocol.
>
> Scp from a remote system works by running another Scp on the remote
> system.  The remote Scp writes SCP protocol messages (and file data)
> to its Standard Output, which the local Scp sees as Standard Input.
>
> But it's important to note that the remote Scp runs in a shell, and
> not just any shell -- an interactive shell.  It's an interactive shell
> because its Standard Input and Standard Output are terminals, and the
> shell knows it and makes some decisions based on the assumption that
> there's a human being there.  This makes Scp analogous to the file
> download function of an old-fashioned terminal emulator -- it's making
> automated use of a data stream intended for an intelligent human.
>
> So lets say the remote interactive shell puts an informational message
> on Standard Output as it starts up (mine does).  The local SCP
> responds by issuing the received line verbatim, with no additional
> context, as an error message, and terminates.  It apparently assumes
> that anything the shell says before remote Scp starts running must be
> a message explaining why it cannot run Scp.  And that the message
> contains enough context that the Scp user will be able to tell what to
> fix and where.
>
> But the message may just be informational, in which case Scp terminates
> needlessly, and with no apparent indication of why.
>
>
> I propose that local Scp just ignore all messages on Standard Input until
> it sees something that looks like an SCP control record.  If it never sees
> an SCP control record, it should issue an error message saying so, and
> include the last thing it _did_ see in that message.
>
> If there isn't something fundamentally wrong with this approach,
I'll
> submit a patch.
>
>
> Ideally, I'd like to see a way to run remote Scp not in a shell at
> all.  It would be far more robust and faster.  Does the SSH protocol
> allow for such a thing?
>
> --
> Bryan Henderson                                    Phone 408-621-2000
> San Jose, California
> _______________________________________________
> openssh-unix-dev at mindrot.org mailing list
> http://www.mindrot.org/mailman/listinfo/openssh-unix-dev
>

At 12:22 PM -0500 7/1/02, Ben Lindstrom wrote:
>You are really looking for sftp which has no shell dependancies.
>
>But in general. No.. Fix your startup scripts.  scp is rcp in a
>ssh wrapper.  and rcp will also fail.
For what it is worth, I fixed this in my own shell processing by
moving any message-producing commands inside of:

     if [ "$TERM" != "dumb" ] ; then
        echo "Welcome!"
     fi

-- 
Garance Alistair Drosehn            =   gad at gilead.netel.rpi.edu
Senior Systems Programmer           or  gad at freebsd.org
Rensselaer Polytechnic Institute    or  drosih at rpi.edu

On Mon, Jul 01, 2002 at 12:22:33PM -0500, Ben Lindstrom
wrote:
> You are really looking for sftp which has no shell dependancies.
no, sftp has, but a lot of things have.

of course you can hack magic things into sftp/sftp-server
and discard the output from .profile.

-m

On Tue, 2 Jul 2002, Markus Friedl wrote:
> On Tue, Jul 02, 2002 at 08:48:57AM -0700, Dan Kaminsky wrote:
> >    Why should sftp-server have any shell dependancy at all?  I was
> > under the impression the whole point of the subsystem interface was to
> > finally gain independance from the vagaries of the shell.
>
> more people think the shell should always be involved.
>
> otherwise you can still sftp if the shell is set
> to /bin/false for example.
>
>
Only way around it would be trusting /etc/shells

- Ben

On Tue, 2 Jul 2002, Ben Lindstrom wrote:
> On Tue, 2 Jul 2002, Markus Friedl wrote:
> > more people think the shell should always be involved.
> >
> > otherwise you can still sftp if the shell is set
> > to /bin/false for example.
> 
> Only way around it would be trusting /etc/shells
You still need to execute the shell because of environmental
issues, some of which can't be handled through mechanisms
provided by OpenSSH (e.g., umask). The umask could be made
settable through auth_keys options or other ~/.ssh mechanisms,
but even so, that would mean that users would have to setup
such things in multiple locations and, in any case, it really
would be surprising if shell initialization were not done at
all (it's certainly a very useful thing).

Why can't users learn? This is really a FAQ and you could
probably automate the closing of bug reports filed about
it :) :)

Year ago I must have made the same mistake - heck, I remember
making the mistake of writing test.c, compiling, running
"test" and nothing happening. Yeah, let's rename /bin/test,
yeah - oh and change all the shells too (after they stop
working :) - NOT.

This is just another Unix-ism - we learn to live with and to
love it because there is expressive power implied within that
we don't want to give up. So please keep using the shell to
invoke sub-system servers.

> - Ben

Cheers,

Nico
-- 

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

I spoke of sub-systems generally - sftp may or may not care
about umask settings - other sub-systems might in fact care.

Nico
-- 

On Tue, 2 Jul 2002, Ben Lindstrom wrote:
> umask can and is ignored by some sftp clients.  Mainly ones that are
> windows based.  Mainly SSH Corp derived last I checked (a year ago).
> 
> So you are trusting that the software on the other side 
> honors umask which
> is a false thing to do since some sftp clients propogate their beliefs
> (real or fictional) of what the permissions should be.
> 
> - Ben
> 
> 
Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

On Tue, 2 Jul 2002, Ben Lindstrom wrote:
> On Tue, 2 Jul 2002 Nicolas.Williams at ubsw.com wrote:
> > I spoke of sub-systems generally - sftp may or may not care
> > about umask settings - other sub-systems might in fact care.
> 
> Which would be a flaw in the subsystem application that was written.
> Unless you know for a fact that subsystem will *NEVER* be 
> implemented on a
> non-UNIX based platform.
By using the shell to launch the sub-system you win
platform independence with respect to setting of umask.
Q.E.D.

As for other sub-systems and their use of permissions: there is
nothing wrong with assuming that ACLs/umasks/whatever are set
correctly and that new files created by a sub-system will
therefore inherit the correct permissions. To do otherwise would
defeat useful mechanisms for setting default permissions
(possibly in context-sensitive ways, as with ACLs, which can be
inherited from a file's containing directory)
> In reality, file permissions should be set by the application 
> or by the
> end-user that is connecting.  To do so otherwise is non-portable.
The setting of file permissions itself varies from platform to
platform.

Some apps should indeed set permissions directly (e.g., package
managers) and some should not (because having file perms set by
directory ACLs or process umasks can be a very useful thing)
and some should give the user the option (I believe this ought
to include file transfer applications [and protocols]).
> After saying that.  I still agree that the user's shell needs 
> to be ran.
> =) Because it's the correct way for UNIX to handle things.
Good. :)
 
Nico
-- 

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

On Tue, July 02 2002, Dan Kaminsky wrote:
> OK, so I modify my .profile to execute arbitrary commands.  Look mah, 
> every time I access a file on a machine, it's "implied" that 
> I have the 
> right to execute stuff on it too.
> 
> You *do* realize this is the kind of logic that's cursed us 
> with macro 
> virii, right?
No. I don't see the connection to macro virii.
> *Sigh* SFTP executes from lower security (command execution) 
> to higher 
> security (file exchange).  It's actually less secure than FTP.
A lot of things work this way. As long as the shell command
line is built securely and the shell setup (including
/etc/profile and ~/.profile) is secure there is nothing wrong.

Yes, users can shoot their feet and legs off if you let them.

Use a restricted shell if you must.

Perhaps SSHD should be configurable with respect to how sub-
systems are started - in fact, I don't why why it shouldn't be
so (though someone is bound to complain about code and config
option bloat...).
> The fix involves detecting an SFTP client in the connection headers, 
> launching an SSHD that refuses to do anything *but* run 
> sftp-server, and 
> removing all exec style functionality from it.  That gives us a clean 
> file transfer environment w/ SSH-class comm security.
You can't detect the client's intention to run SFTP until
after the SSHv2 connection setup, which kinda means you must
run SSHD first.
> --Dan

Nico
-- 

Visit our website at http://www.ubswarburg.com

This message contains confidential information and is intended only 
for the individual named.  If you are not the named addressee you 
should not disseminate, distribute or copy this e-mail.  Please 
notify the sender immediately by e-mail if you have received this 
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free 
as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses.  The sender therefore 
does not accept liability for any errors or omissions in the contents 
of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.  This 
message is provided for informational purposes and should not be 
construed as a solicitation or offer to buy or sell any securities or 
related financial instruments.

On Wed, Jul 03, 2002 at 04:09:44AM -0700, raam raam
wrote:
> Hi
> 
> Is there any test tool available for testing SSH
Test ssh with -v -v -v 
or use the testscript from the snailbook:

http://www.snailbook.com/tools.html

/magnus

-- 
http://x42.com/